mdadm: Cannot open /dev/hde1: Device or resource busyI got similar output trying to add hde3 back to /dev/md3. Those partitions are only used for raid so it was really bothering me that it said they were in use. I googled around a bit and found a reference to the device mapper (which starts up on bootup for me because I use LVM) creating some devices based on a motherboard raid controller. You can get a listing of what the device mapper has created using dmsetup ls. I ran the command and sure enough there were 4 devices listed starting with nvidia_. I was able to remove 3 of the 4 using dmsetup -C
adj - subject to whims, crankiness, or ill temper
I use Gentoo Linux both at home and at work. Every so often I hit some snag and I'd like to detail the fix here both for my benefit and to possibly help anyone else having a similar issue.
Sunday, February 17, 2008
Saturday, February 16, 2008
Covering my ass...
Since this is a potentially deadly operation (the general consensus in #gentoo-hardened was that some people have done it and it's probably ok, BUT Bad Things© could happen) so they don't really recommend doing so. Because of this, I'm doing the following to help mitigate data loss.
I shut down most of my services (switching to single user mode would be better, but I was too lazy to hook up monitor/kb/mouse to server...) and ran a backup to get a snapshot of the system. My /boot and / partitions are mirrored using kernel raid and I told mdadm to kick the second drive out of each of the arrays:
# mdadm /dev/md1 -f /dev/hde1So now all of my changes for hardened will be occurring on one drive, If I completely screw up the system and can't fix, I can just switch to the "faulty" drive and rebuild the array. If all goes well, I just re-add the partitions to the 2 raids and they'll resync and all will be rosy.
# mdadm /dev/md3 -f /dev/hde3
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 hdg1
40064 blocks [2/1] [U_]
bitmap: 2/5 pages [8KB], 4KB chunk
md0 : active raid5 sdd1 sdc1 sdb1 sda1
937705728 blocks level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/150 pages [0KB], 1024KB chunk
md3 : active raid1 hdg3
155244032 blocks [2/1] [U_]
bitmap: 57/149 pages [228KB], 512KB chunk
Getting to the task at hand...
First, switch to the new profile. You can do this with eselect. Let's see what's avaialable on the system:
# eselect profile listChange the profile:
Available profile symlink targets:
 default-linux/amd64/2007.0 *
# eselect profile set 10Next, you need to build the hardened toolchain:
# emerge -av --oneshot binutils gcc virtual/libcTell the system to use the new (older) hardened gcc profile:
# gcc-config -lSlight change to the /etc/make.conf CFLAGS (adding -fforce-addr, I don't know what it does but if you download a hardened stage tarball, it's set in the make.conf by default so I'm adding it here) Substitute my march for yours, of course:
 x86_64-pc-linux-gnu-4.1.2 *
# gcc-config x86_64-pc-linux-gnu-3.4.6
* Switching native-compiler to x86_64-pc-linux-gnu-3.4.6 ...
>>> Regenerating /etc/ld.so.cache... [ ok ]
* If you intend to use the gcc from the new profile in an already
* running shell, please remember to do:
* # source /etc/profile
# source /etc/profile
CFLAGS="-march=k8 -pipe -O2 -fforce-addr"Next, I do a test emerge command and look for green (use flags that are changing state). The reason you need to do this is each profile has a set of profile defined USE defaults. The new hardened profile added a couple and removed a few in my case. So basically, do an emerge -ave world and look for green and * which signifies a change in the use flag since the last time you merged a package. Add or remove corresponding use flags to /etc/make.conf (or use app-portage/ufed as I do). Keep running the emerge -ave world and saying n until you are happy with the output and then hit y to actually start merging.
# emerge -ave worldIf you run into any snags (a package fails to build), just note the package that failed and restart the emerge with "emerge -ave world --resume --skipfirst". Obviously things can get a little tricky if the package with the problem is a core system library or something, but if you don't use --resume, it's going to start rebuilding the WHOLE system again. In the past I've found it's relatively safe to "fix" the problem in another shell while continuing to build in the primary shell.
So about 9 hours and 312 packages later it's done. I restarted most of my network services just to make sure they wouldn't blow up right off the bat and everything seemed alright so far. I emerged hardened-sources while the world was rebuilding so I kicked off genkernel to configure (according to the various hardened guides), build and install the new kernel with hardened sources. After that I rebooted and everything still came up OK.
After testing things out a bit, I re-added the second drive to the mirrors and let the arrays resync:
# mdadm /dev/md1 -a /dev/hde1So those are the basic steps to switch over to hardened. Remember, always have backups ready before you do something like this.
# mdadm /dev/md3 -a /dev/hde3