Migrating a software raid1 array to 2 new harddrives

My server currently has 2 IDE drives mirrored for the boot/root and swap partitions.  They are 160 and 250GB drives.  The 250 is a refurb sent to me by Seagate after the original matching 160 died.  So the mirror has already saved me once.  I'm going to migrate to newer/faster/larger 500GB SATA discs.  Rather than use the whole 500 for the root filesystem which I don't need.  I decided to keep the partition sizes all the same and use the leftover space on the drive to add capacity and redundancy to my storage array which is the subject for another blog post.

So here's what I'm starting with partition wise:

/dev/hda1          0+      4       5-     40131   fd  Linux raid autodetect
/dev/hda2          5     129     125    1004062+  82  Linux swap / Solaris
/dev/hda3        130   19456   19327  155244127+  fd  Linux raid autodetect

hdb obviously has the exact same partition layout.  

/dev/hda1 and hdb1 are raid1 /dev/md1 for /boot
/dev/hda2 and hdb2 are swap

/dev/hda3 and hdb3 are raid1 /dev/md3 for /

I originally had the swap partitions mirrored as well but after reading up a bit I decided I didn't need THAT level of protection so just added the 2 partitions individually.  (They used to be /dev/md2)

The first step (after adding the new drives to the system) is to partition new disks exactly like the old ones:

sfdisk -d /dev/hda | sfdisk /dev/sde  

sfdisk -d /dev/hda | sfdisk /dev/sdf

At this point I added a 4th partition to sde and sdf taking up the rest of the drives.  That's in preparation for the other migration.

set up swap:

# mkswap /dev/sde2
Setting up swapspace version 1, size = 1004056 KiB
no label, UUID=d9a6fd39-e768-4334-8496-2b0b5ab44bdf
# mkswap /dev/sdf2
Setting up swapspace version 1, size = 1004056 KiB
no label, UUID=529c0773-9a3e-434d-b6e4-16cb0e8f24a2
 
Turn on the new swaps (I mount stuff almost exclusively with UUIDs):
# swapon UUID=d9a6fd39-e768-4334-8496-2b0b5ab44bdf
# swapon UUID=529c0773-9a3e-434d-b6e4-16cb0e8f24a2

Add swaps to fstab (I removed the old ones at this point before I forgot):

UUID=d9a6fd39-e768-4334-8496-2b0b5ab44bdf  none  swap  sw,pri=1 0 0
UUID=529c0773-9a3e-434d-b6e4-16cb0e8f24a2  none  swap  sw,pri=1 0 0

Add 2 new devices to md1 (/boot):

mdadm /dev/md1 --add /dev/sde1 --add /dev/sdf1

Add 2 new devices to md3 (/):

mdadm /dev/md3 --add /dev/sde3 --add /dev/sdf3

Snipped mdadm detail output shows the new devices as spares:

# mdadm --detail /dev/md1

    Number   Major   Minor   RaidDevice State
       0       3        1        0      active sync   /dev/hda1
       1       3       65        1      active sync   /dev/hdb1

       2       8       81        -      spare   /dev/sdf1
       3       8       65        -      spare   /dev/sde1

# mdadm --detail /dev/md3
    Number   Major   Minor   RaidDevice State
       0       3        3        0      active sync   /dev/hda3
       1       3       67        1      active sync   /dev/hdb3

       2       8       83        -      spare   /dev/sdf3
       3       8       67        -      spare   /dev/sde3

To speed up the sync process we're about to cause, issue the following:

# echo 200000 > /proc/sys/dev/raid/speed_limit_max
# echo 200000 > /proc/sys/dev/raid/speed_limit_min

Before I continue.  The next couple steps are where the system doesn't have the full redundancy.  I'm going to mark one of the 2 ide devices in the array as faulty.  The kernel will automatically grab a spare and start rebuilding the array.  Personally, I'm not worrying about this because, if something does happen, I can always re-add the drive I "failed".  Just making the point that when you manually degrade the array, you're mirror isn't redundant until the rebuild/resync is complete.  Continuing on...

Mark one of the old devices in a raid as faulty.  It's very important that you only mark one device faulty!  This will cause the array to grab a spare and start syncing the remaining good device, hdb1 in this case, to the new device .  You can watch the progress of the resync via cat /proc/mdstat.

# mdadm /dev/md1 -f /dev/hda1
mdadm: set /dev/hda1 faulty in /dev/md1 

Since this raid volume is all of 40MB, it resyncs before I can even look at the mdstat output.  Still, I check and make sure it's all synced up and fault the other old partition:
# mdadm /dev/md1 -f /dev/hdb1
mdadm: set /dev/hdb1 faulty in /dev/md1

Again, check mdstat output and make sure it finishes.  It should look similar to this:
md1 : active raid1 sdf1[2] sde1[3] hdb1[1](F) hda3[1](F)

      40064 blocks [2/2] [UU]
      bitmap: 0/5 pages [0KB], 4KB chunk

Now all the data has been copied over to the new drives and we just need to remove the old ones from the array:

# mdadm /dev/md1 --remove /dev/hda1 --remove /dev/hdb1
mdadm: hot removed /dev/hda1

mdadm: hot removed /dev/hdb1

mdstat now says:
md1 : active raid1 sdf1[0] sde1[1]
      40064 blocks [2/2] [UU]
      bitmap: 0/5 pages [0KB], 4KB chunk

Now do the same steps for hda3 and hdb3 for the md3 array.

Fail one of the devices:
# mdadm /dev/md3 -f /dev/hda3
mdadm: set /dev/hda3 faulty in /dev/md3

This array is ~155GB so it takes a little longer to resync, here's the mdstat output:

md3 : active raid1 sdf3[2] sde3[3](S) hdb3[1] hda3[4](F)
      155244032 blocks [2/1] [_U]
      [=====>...............]  recovery = 29.2% (45338816/155244032) finish=38.1min speed=48002K/sec
      bitmap: 25/149 pages [100KB], 512KB chunk

Incidentally, here's some iostat output showing why I want to get rid of the old IDE harddrives...hdb is reading 46MB/sec and it as 99.44% utilization.  Meanwhile sde is writing at 46MB/sec and is only at 38% utilization:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
hdb             649.20     0.00   93.60    1.40    46.34     0.01   999.07     3.45   36.33  10.47  99.44
sdf               0.00   646.80    0.00   96.20     0.00    46.44   988.74     0.41    4.24   3.98  38.32

When the array has resynced after failing the first partition, go ahead and mark the second original device as faulted:

# mdadm /dev/md3 -f /dev/hdb3
mdadm: set /dev/hdb3 faulty in /dev/md3

The new drives are a little over twice as fast as the old.  It's reading off of sde to fill sdf:

md3 : active raid1 sdf3[0] sde3[2] hdb3[3](F) hda3[4](F)
      155244032 blocks [2/1] [U_]
      [=====>...............]  recovery = 28.9% (44921984/155244032) finish=18.7min speed=97804K/sec
      bitmap: 39/149 pages [156KB], 512KB chunk 

Once the array is finished up with the second sync, time to remove the 2, now faulty, device from the array:

#mdadm /dev/md3 --remove /dev/hda3 --remove /dev/hdb3
mdadm: hot removed /dev/hda3
mdadm: hot removed /dev/hdb3

At this point I can turn off the swaps on the old drives:

swapoff /dev/hda2 /dev/hdb2

And the system is no longer "using" the old hard drives at all. 

I shutdown the system and remove the old IDE hard drives.  Then I boot to system rescue cd and chroot into the system (following same procedure as initially entering the chroot of your system from Gentoo Handbook) to run grub and install it into the MBR of both new drives.  Grub names drives differently than the kernel.  It uses BIOS numbering as well.  Meaning grub sees the 2 new SATA drives (on the sil3132 controller) as the 5th and 6th drives.  They are named hd4 and hd5 respectively.  The kernel, on the other hand, sees them as sde and sdf because it enumerates the 4 drives plugged into the motherboard first.

First start grub

# grub 

Install to first drive's MBR (boot partition is first partition on 5th drive)
grub> root (hd4,0)
 Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd4) 

Install to second drive's MBR (boot partition is first partition on 6th drive)

grub> root (hd5,0)
 Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd5)

Now I reboot and check the BIOS settings to tell the computer to boot off of one of the 2 new SATA drives and let 'er go.

The arrays and swap are now running exclusively on the new SATA drives:

# cat /proc/mdstat
md1 : active raid1 sdf1[0] sde1[1]
      40064 blocks [2/2] [UU]
      bitmap: 0/5 pages [0KB], 4KB chunk

md3 : active raid1 sdf3[0] sde3[1]
      155244032 blocks [2/2] [UU]
      bitmap: 24/149 pages [96KB], 512KB chunk

# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/sde2                               partition       1004052 0       1
/dev/sdf2                               partition       1004052 0       1