Migrating Linux software raid from 4 device raid5 to 6 device raid6

In a previous post, I discussed migrating from 2xIDE device mirror to a 2xSATA device mirror.  Since the old arrays were using 160GB and I bought 500GB drives I figured I'd use the space left over to add a couple more devices to my storage array.

Here's what I'm starting with:

# cat /proc/mdstat

md0 : active raid5 sdd1[1] sdb1[3] sdc1[2] sda1[0]

      937705728 blocks level 5, 128k chunk, algorithm 2 [4/4] [UUUU]

      bitmap: 0/150 pages [0KB], 1024KB chunk

It's a 4x320GB raid5 array.  I'm going to expand the array to include 2 more devices (sde4 and sdf4) and reshape it to a raid6 array at the same time.  I will end up gaining 1 device's worth of space (320GB) and 1 more drive of redundancy.  With raid6, the array will be able to survive 2 failures and still function instead of the 1 failure a raid5 array can survive.

Unfortunately, the current, stable hardened kernel is 2.6.28-r9 and to reshape a raid5 to raid6 requires at least a 2.6.31 kernel.  Additionally mdadm >=3.1.0 is required and 3.0 is currently stable.  The second is reasonably easy to fix:

# echo "=sys-fs/mdadm-3.1.1-r1" >> /etc/portage/package.keywords 

# emerge -av mdadm

For the kernel, I installed layman and added the hardened-development overlay (not covered here) and unmasked the minimum required kernel:

# echo "=sys-kernel/hardened-sources-2.6.31-r11" >> /etc/portage/package.keywords

# emerge -av hardened-sources

I'm also not going to cover configuring/building/installing/booting to the new kernel.  If you're using Gentoo, you should know what you're doing already in that respect.

After all the prerequisites are taken care of (I created the partitions I'm using here during the previous array muddling in the previous blog post) we can move forward.

Add the 2 new devices to the array.

# mdadm /dev/md0 --add /dev/sde4 /dev/sdf4

At this point the new devices will be acting as "spares" as shown below (the (S) next to the device):

# cat /proc/mdstat

md0 : active raid5 sdf4[4](S) sde4[5](S) sdd1[1] sdb1[3] sdc1[2] sda1[0]

      937705728 blocks level 5, 128k chunk, algorithm 2 [4/4] [UUUU]

      bitmap: 1/150 pages [4KB], 1024KB chunk

Turn off the write-intent bitmap on the array temporarily.  This is necessary for the reshape to occur.  I originally was getting an error and Neil Brown (mdadm author http://neil.brown.name) told me I needed to remove the bitmap while reshaping:

# mdadm --grow /dev/md0 --bitmap none

To speed up the sync process we're about to cause, issue the following:

# echo 200000 > /proc/sys/dev/raid/speed_limit_max
# echo 200000 > /proc/sys/dev/raid/speed_limit_min

Start the reshape:

# mdadm --grow /dev/md0 --level=6 --raid-devices=6 --backup-file=/root/raid-backup 

mdadm level of /dev/md0 changed to raid6 

mdadm: Need to backup 1536K of critical section..

Watch the *extremely* slow reshape (you can literally watch it with watch -n 1 cat /proc/mdstat):

# cat /proc/mdstat

md0 : active raid6 sda1[4] sdf4[0] sde4[5] sdb1[3] sdd1[1] sdc1[2]
      937705728 blocks super 0.91 level 6, 128k chunk, algorithm 18 [6/7] [UUUUUU]
      [====>................]  reshape = 22.6% (70662528/312568576) finish=286.1min speed=14088K/sec 

At this point, mdadm --detail output still shows my array as being the old size:

Array Size : 937705728 (894.27 GiB 960.21 GB)
Used Dev Size : 312568576 (298.09 GiB 320.07 GB) 

I was curious about this as I should have gained 320GB.  My device size is 320GB, raid5 capacity is n-1 devices: 320x3 = 960GB.  After the reshape it will be n-2 devices: 320x4=1280GB. So I ran a test with some loopback devices and the size of the array will be correct when the reshape is completed.

After the reshape, turn the write intent bitmap back on:

# mdadm --grow /dev/md0 --bitmap internal

As you can see, the array now has the proper 320x4 size (and the superblock version went back 0.90):

# mdadm -D /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Thu Sep  7 18:41:05 2006
     Raid Level : raid6
     Array Size : 1250274304 (1192.35 GiB 1280.28 GB)
  Used Dev Size : 312568576 (298.09 GiB 320.07 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sun Mar 14 11:40:59 2010
          State : active
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

           UUID : fdc29307:ba90c91c:d9adde8d:723321bc
         Events : 0.692377

    Number   Major   Minor   RaidDevice State
       0       8       84        0      active sync   /dev/sdf4
       1       8       49        1      active sync   /dev/sdd1
       2       8       33        2      active sync   /dev/sdc1
       3       8       17        3      active sync   /dev/sdb1
       4       8        1        4      active sync   /dev/sda1
       5       8       68        5      active sync   /dev/sde4 

Since I use LVM to chop up this array, I just need to grow my pv to make LVM aware of the new, larger, size of the underlying raid array:

# pvresize /dev/md0

pvdisplay now shows the full size:

  PV Size               1.16 TiB / not usable 2.81 MiB

Similarly, vgdisplay shows the extra space available for allocation:

  Free  PE / Size       86234 / 336.85 GiB

And that's about it.  Big thanks to Neil for the tip on the write-intent bitmap.  The combination of Linux kernel raid and mdadm let's you do some pretty amazing things.  I was able to do both the raid1 migrations and this raid5 -> raid6 extend/reshape while the system was up and running with live filesystems.  That's pretty impressive.

Migrate an existing Gentoo system to hardened profile

This post is about migrating a system running a current amd64 profile to a hardened profile and all the things entailed in setting up a reasonably "hardened" Gentoo system. I've been wanting to use hardened but in the past when I have looked into it, the process of switching would have required a downgrade of libc that portage doesn't want to allow. Currently the hardened profile uses the same libc that I already have so this presents the opportunity to do the switch.

Covering my ass...

Since this is a potentially deadly operation (the general consensus in #gentoo-hardened was that some people have done it and it's probably ok, BUT Bad Things© could happen) so they don't really recommend doing so. Because of this, I'm doing the following to help mitigate data loss.

I shut down most of my services (switching to single user mode would be better, but I was too lazy to hook up monitor/kb/mouse to server...) and ran a backup to get a snapshot of the system. My /boot and / partitions are mirrored using kernel raid and I told mdadm to kick the second drive out of each of the arrays:

# mdadm /dev/md1 -f /dev/hde1
# mdadm /dev/md3 -f /dev/hde3
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 hdg1[0]
      40064 blocks [2/1] [U_]
      bitmap: 2/5 pages [8KB], 4KB chunk

md0 : active raid5 sdd1[1] sdc1[2] sdb1[0] sda1[3]
      937705728 blocks level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/150 pages [0KB], 1024KB chunk

md3 : active raid1 hdg3[0]
      155244032 blocks [2/1] [U_]
      bitmap: 57/149 pages [228KB], 512KB chunk

unused devices: 

So now all of my changes for hardened will be occurring on one drive, If I completely screw up the system and can't fix, I can just switch to the "faulty" drive and rebuild the array. If all goes well, I just re-add the partitions to the 2 raids and they'll resync and all will be rosy.

Getting to the task at hand...

First, switch to the new profile. You can do this with eselect. Let's see what's avaialable on the system:

# eselect profile list
Available profile symlink targets:
[1]   default-linux/amd64/2006.1
[2]   default-linux/amd64/2006.1/desktop
[3]   default-linux/amd64/2006.0/no-symlinks
[4]   default-linux/amd64/2006.1/no-multilib
[5]   default-linux/amd64/2007.0 *
[6]   default-linux/amd64/2007.0/desktop
[7]   default-linux/amd64/2007.0/no-multilib
[8]   default-linux/amd64/2007.0/server
[9]   hardened/amd64
[10]  hardened/amd64/multilib
[11]  selinux/2007.0/amd64
[12]  selinux/2007.0/amd64/hardened

Change the profile:

# eselect profile set 10

Next, you need to build the hardened toolchain:

# emerge -av --oneshot binutils gcc virtual/libc

Tell the system to use the new (older) hardened gcc profile:

# gcc-config -l
[1] x86_64-pc-linux-gnu-3.4.6
[2] x86_64-pc-linux-gnu-3.4.6-hardenednopie
[3] x86_64-pc-linux-gnu-3.4.6-hardenednopiessp
[4] x86_64-pc-linux-gnu-3.4.6-hardenednossp
[5] x86_64-pc-linux-gnu-3.4.6-vanilla
[6] x86_64-pc-linux-gnu-4.1.2 *
# gcc-config x86_64-pc-linux-gnu-3.4.6
* Switching native-compiler to x86_64-pc-linux-gnu-3.4.6 ...
>>> Regenerating /etc/ld.so.cache...                                                                                                  [ ok ]

* If you intend to use the gcc from the new profile in an already
* running shell, please remember to do:

*   # source /etc/profile

# source /etc/profile

Slight change to the /etc/make.conf CFLAGS (adding -fforce-addr, I don't know what it does but if you download a hardened stage tarball, it's set in the make.conf by default so I'm adding it here) Substitute my march for yours, of course:

CFLAGS="-march=k8 -pipe -O2 -fforce-addr"

Next, I do a test emerge command and look for green (use flags that are changing state). The reason you need to do this is each profile has a set of profile defined USE defaults. The new hardened profile added a couple and removed a few in my case. So basically, do an emerge -ave world and look for green and * which signifies a change in the use flag since the last time you merged a package. Add or remove corresponding use flags to /etc/make.conf (or use app-portage/ufed as I do). Keep running the emerge -ave world and saying n until you are happy with the output and then hit y to actually start merging.

# emerge -ave world

If you run into any snags (a package fails to build), just note the package that failed and restart the emerge with "emerge -ave world --resume --skipfirst". Obviously things can get a little tricky if the package with the problem is a core system library or something, but if you don't use --resume, it's going to start rebuilding the WHOLE system again. In the past I've found it's relatively safe to "fix" the problem in another shell while continuing to build in the primary shell.

So about 9 hours and 312 packages later it's done. I restarted most of my network services just to make sure they wouldn't blow up right off the bat and everything seemed alright so far. I emerged hardened-sources while the world was rebuilding so I kicked off genkernel to configure (according to the various hardened guides), build and install the new kernel with hardened sources. After that I rebooted and everything still came up OK.

After testing things out a bit, I re-added the second drive to the mirrors and let the arrays resync:

# mdadm /dev/md1 -a /dev/hde1
# mdadm /dev/md3 -a /dev/hde3

So those are the basic steps to switch over to hardened. Remember, always have backups ready before you do something like this.