Did a yum update a few hours ago without looking what are to be updated. Turns out this is a big yum update, upgrading CentOS from 4.4 to 4.5.
Went out, came back, yum update completed. Shutted down server.
Just booted up the server to find that normal bootup now fails...
System message:
Checking root filesystem [[FAILED]]
/dev/VolGroup00/LogVol00 is mounted. e2fsck : cannot continue, aborting.
*** An error occurred during the file system check *** Dropping you to a shell; the system will reboot *** when you leave the shell
Give root password for maintenance or type Ctrl-D to continue:
Luckily, this is a test server (while I'm waiting for VMpro support for CentOS 5.0).
Not so luckily, there are development scripts I am working on on the server. And being the genius that I am, I didn't back up! :-(
Joe and Jamie.... HELP!
ON bootup click the menu and bootup on the older grub, unless it's been replaced. I still sometimes have occasion where the new kernal isn't placed right.
THanks for the update, I'll back off and hold my yum ;-)
I cannot believe this has not been fixed! This has been a problem in CentOS (maybe RHEL, too, though I don't think I've seen it on Virtualmin.com which runs RHEL) for over two years (some of the earliest discussions on Virtualmin.com were about this very issue). It was hitting Fedora during FC3 and FC4, as well, but has long since been fixed. I'm stunned that CentOS is still running into it.
Something is broken in the mkinitrd on CentOS that fails to set the root partition read only while booting, and thus fsck refuses to run.
I'd suggest filing a bug with the CentOS folks first (or searching their forum and bug tracker for previous occurrences of this). I'm having a hard time googling the solution to this.
It'll involve booting from a rescue CD (insert the first CentOS CD, type "linux rescue", hit enter, allow it to mount your existing system).
Then you'll need to use cpio to unarchive the initrd in /boot, and edit...hmmm...I can't recall what you edit within the initrd. Basically, the root filesystem needs to have "ro" added to the mount command.
I'll keep digging.
--
Check out the forum guidelines!
Does anyone know how to do what Joe is talking about?
As I understand it, I need to:
1. Boot rescue CD (linux rescue)
2. Enter server files
3. Unzip and unarchive initrd image
4. Change something to include 'ro' in mount command
5. Archive and zip back initrd image
6. Reboot
I am NOT a tech guy, but a tack-in-the-head guy (ouch!). So detailed instructions will be appreciated very muchly.
Does anyone know how to do what Joe is talking about?
As I understand it, I need to:
1. Boot rescue CD (linux rescue)
2. Enter server files
3. Unzip and unarchive initrd image
4. Change something to include 'ro' in mount command
5. Archive and zip back initrd image
6. Reboot
I am NOT a tech guy, but a tack-in-the-head guy (ouch!). So detailed instructions will be appreciated very muchly.
I haven't tried the update yet. What I'd meant earlier is the CentOS bootup countdown screen. Usually if you catch it (hit enter) you should see two bootups, choose the older kernal and it should boot you up yet.
Thanks Dan. I loaded the older kernel and got my server back. Guess who is busy backing up his butt. ;-)
Is there a way to yum uninstall the previous update? I'd like to go back to the good old days...
Cheers bro.
I've kept my head in the sand for a while now, still using the CentOS 2.6.9-34 kernel (CentOS 4.3), though decided to just nut this sucker out last night.
There is a script called 'init' in the initrd image that contains the relevant mount commands. I compared the script between the 2.6.9-34 and 2.6.9-55 initrd images and found a difference between the mount commands. The 2.6.9-34 kernel used:
mount -o defaults --ro -t ext3 /dev/root /sysroot
While the 2.6.9-55 kernel used:
mount -o grpquota,usrquota,rw --ro -t ext3 /dev/root /sysroot
This is the general procedure I used to edit my initrd image.
1. mkdir /tmp/initrd
2. cd /tmp/initrd
2. gzip -dc /boot/initrd-2.6.9-55.EL.img | cpio -id
3. vi init
4. Change 'grpquota,usrquota,rw' to 'defaults' as mentioned above
5. mv /boot/initrd-2.6.9-55.EL.img /boot/initrd-2.6.9-55.EL.img.orig
6. find ./ cpio -H newc -o] /boot/initrd-new.img
7. gzip /boot/initrd-new.img
8. mv /boot/initrd-new.img.gz /boot/initrd-2.6.9-55.EL.img
The server booted fine after this change.
I've kept my head in the sand for a while now, still using the CentOS 2.6.9-34 kernel (CentOS 4.3), though decided to just nut this sucker out last night.
There is a script called 'init' in the initrd image that contains the relevant mount commands. I compared the script between the 2.6.9-34 and 2.6.9-55 initrd images and found a difference between the mount commands. The 2.6.9-34 kernel used:
mount -o defaults --ro -t ext3 /dev/root /sysroot
While the 2.6.9-55 kernel used:
mount -o grpquota,usrquota,rw --ro -t ext3 /dev/root /sysroot
This is the general procedure I used to edit my initrd image.
1. mkdir /tmp/initrd
2. cd /tmp/initrd
2. gzip -dc /boot/initrd-2.6.9-55.EL.img | cpio -id
3. vi init
4. Change 'grpquota,usrquota,rw' to 'defaults' as mentioned above
5. mv /boot/initrd-2.6.9-55.EL.img /boot/initrd-2.6.9-55.EL.img.orig
6. find ./ cpio -H newc -o] /boot/initrd-new.img
7. gzip /boot/initrd-new.img
8. mv /boot/initrd-new.img.gz /boot/initrd-2.6.9-55.EL.img
The server booted fine after this change.
Given that, then you should be able to go into the GRUB bootloader module and make the corrections ;-)
-
I've kept my head in the sand for a while now, still using the CentOS 2.6.9-34 kernel (CentOS 4.3), though decided to just nut this sucker out last night.
There is a script called 'init' in the initrd image that contains the relevant mount commands. I compared the script between the 2.6.9-34 and 2.6.9-55 initrd images and found a difference between the mount commands. The 2.6.9-34 kernel used:
mount -o defaults --ro -t ext3 /dev/root /sysroot
While the 2.6.9-55 kernel used:
mount -o grpquota,usrquota,rw --ro -t ext3 /dev/root /sysroot
This is the general procedure I used to edit my initrd image.
1. mkdir /tmp/initrd
2. cd /tmp/initrd
2. gzip -dc /boot/initrd-2.6.9-55.EL.img | cpio -id
3. vi init
4. Change 'grpquota,usrquota,rw' to 'defaults' as mentioned above
5. mv /boot/initrd-2.6.9-55.EL.img /boot/initrd-2.6.9-55.EL.img.orig
6. find ./ cpio -H newc -o] /boot/initrd-new.img
7. gzip /boot/initrd-new.img
8. mv /boot/initrd-new.img.gz /boot/initrd-2.6.9-55.EL.img
The server booted fine after this change.
Given that, then you should be able to go into the GRUB bootloader module and make the corrections ;-)
-
Hey Dan,
Actually the initrd is outside of the control of grub, which is why this is such an annoying problem. Just editing the grub options won't correct the problem.
--
Check out the forum guidelines!
Ah!!! That explains why it didn't work for me.
Another case of do first, speak second :-(
Thanks Simon, for the detailed instructions.
I must have made a typo along the way, or was it because I was messing around with another set of similar (piecemeal) instructions that I google-found http://www.hostingforum.ca/228302-how-rebuilding-install-initrd-sles-10-... ... now my bootup is royally screwed. Booting up an older kernel now generates many errors too.
Screw it. I will install CentOS 5.0 and wait for J&J's release of a new VMpro installer with CentOS 5.0 support... hopefully before my beard turns white and touches the floor (wait a minute, I don't have a beard, yet). Hehehe... ;-)
I've kept my head in the sand for a while now, still using the CentOS 2.6.9-34 kernel (CentOS 4.3), though decided to just nut this sucker out last night.
There is a script called 'init' in the initrd image that contains the relevant mount commands. I compared the script between the 2.6.9-34 and 2.6.9-55 initrd images and found a difference between the mount commands. The 2.6.9-34 kernel used:
mount -o defaults --ro -t ext3 /dev/root /sysroot
While the 2.6.9-55 kernel used:
mount -o grpquota,usrquota,rw --ro -t ext3 /dev/root /sysroot
This is the general procedure I used to edit my initrd image.
1. mkdir /tmp/initrd
2. cd /tmp/initrd
2. gzip -dc /boot/initrd-2.6.9-55.EL.img | cpio -id
3. vi init
4. Change 'grpquota,usrquota,rw' to 'defaults' as mentioned above
5. mv /boot/initrd-2.6.9-55.EL.img /boot/initrd-2.6.9-55.EL.img.orig
6. find ./ cpio -H newc -o] /boot/initrd-new.img
7. gzip /boot/initrd-new.img
8. mv /boot/initrd-new.img.gz /boot/initrd-2.6.9-55.EL.img
The server booted fine after this change.
Simon,
Is a pipe missing in Step 6?
Shouldn't it be:
# find ./ | cpio -H newc -o] /boot/initrd-new.img
Anyway, I have amended a few things in your instruction to suit my needs (e.g. I don't like to use vi; prefer nano instead). Also added -v verbose to the mv command so that I am informed if something goes wrong (e.g. typo). So here is a slightly different version.
1. Load an older kernel at bootup (if you have physical access to the server) by pressing ENTER just before the kernel is loaded.
2. Log in as root
3. mkdir /tmp/initrd
4. cd /tmp/initrd
5. gzip -dc /boot/initrd-2.6.9-55.EL.img | cpio -id
6. nano init
7. Ctrl-W to find "grpquota,usrquota,rw" (without quotation marks) in init file
8. Change 'grpquota,usrquota,rw' to 'defaults' as mentioned above
9. mv -v /boot/initrd-2.6.9-55.EL.img /boot/initrd-2.6.9-55.EL.img.orig
10. find ./ | cpio -H newc -o] /boot/initrd-new.img
11. gzip /boot/initrd-new.img
12. mv -v /boot/initrd-new.img.gz /boot/initrd-2.6.9-55.EL.img
13. Reboot server.
Yep you're right. I did forget the pipe. Thanks for picking that up.
ON bootup click the menu and bootup on the older grub, unless it's been replaced. I still sometimes have occasion where the new kernal isn't placed right.
THanks for the update, I'll back off and hold my yum ;-)
I went through this same procedure again for the 2.6.9-55.0.6 kernel and referenced my previous post to make the job easier. I noticed that I had more than one typo. For everyone else's benefit, as well as my own, here is the revised procedure, with Step 6 properly corrected:
1. mkdir /tmp/initrd
2. cd /tmp/initrd
2. gzip -dc /boot/initrd-2.6.9-55.EL.img | cpio -id
3. vi init
4. Change 'grpquota,usrquota,rw' to 'defaults' as mentioned above
5. mv /boot/initrd-2.6.9-55.EL.img /boot/initrd-2.6.9-55.EL.img.orig
6. find ./ | cpio -H newc -o > /boot/initrd-new.img
7. gzip /boot/initrd-new.img
8. mv /boot/initrd-new.img.gz /boot/initrd-2.6.9-55.EL.img
Cheers,
Simon
Thank you for that last post correction.
I remember when I started college, on Engineering orientation day we all had to take this test. It was 50 questions and we only had 2 hours to complete it. Well I had at it and I'm grinding away on question 2 about 5 minutes in when I noticed some people getting up and turning their answer books in. I'm beginning to think, sheeesh, I'm in real deep with these braniacs. Finally after a frustrating 45 minutes and the lecture hall half empty I started cruising the exam to answer the easiest questions.......till I got to question # 50 when I let out an inappropriately loud "sh.."
Question # 50 said "please ignore the preceeding 49 questions"
It was supposed to get the point across to read the entire exam before you start, because the process in question 3 might help you solve question 1.
.....
After a couple hours of trying AH's steps and trying to figure out what to do about that square bracket ( eliminate it, maybe he meant a pipe, maybe no space between....) I got frustrated and closed some windows covering your last correction post. Jeeez Lueeez, then It was a breeze ;-)
Dan