Howto mirror a NetBSD-1.5 / 1.6 installation with RAIDFrame This document describes the steps needed, to implement mirroring on a NetBSD 1.5 or 1.6 server, which has a second, identical Disk available Use at your own Risk! Version 1.3 28. October 2002 Prerequisites _______________________________________________________________________ I have been setting up my i386 Server with two identical disks. So, for this tutorial, the prerequisites are: - i386 platform - 2 identical disks - NetBSD >= 1.5.1 installed on 1st disk - A current backup of your data! Of course you can setup RAIDFrame on other platforms than i386 and with non-identical disks. If you have another NetBSD port or non-identical disks, you can still setup mirroring, but you can't just follow the steps presented below. Read the raidctl(8) and raid(4) man pages! These steps also work on NetBSD 1.6. The differences are noted through­ out the text. Overview _______________________________________________________________________ The following steps will be taken: - Enable RAIDFrame code in kernel - Setup 2nd Disk as 2nd half of the mirror - boot off 2nd disk with RAIDFrame active (The mirror is damaged, of course, but it works nonetheless) - Repartition/relabel 1st disk the same way as the 2nd and resync it Step 1: Enable RAIDFrame code in kernel _______________________________________________________________________ - Reconfigure your kernel to include pseudo-device raid 8 # RAIDFrame disk driver options RAID_AUTOCONFIG # auto-configuration of RAID components - Hardcode the disk controller(s) and disks (according to your dmesg output): pciide0 at pci? dev ? function ? flags 0x0000 wd0 at pciide0 channel 0 drive 0 flags 0x0000 wd1 at pciide0 channel 1 drive 0 flags 0x0000 If you have SCSI disks, you would do something like this: scsibus0 at ahc? sd0 at scsibus0 target 0 lun 0 # SCSI disk drives sd1 at scsibus0 target 2 lun 0 # SCSI disk drives Step 2: Setup 2nd Disk as 2nd half of the mirror _______________________________________________________________________ - Prepare the wd1 DOS-Partition table fdisk -u wd1 - Activate the DOS partition you want to use (in my case, partition 0): fdisk -a -0 wd1 My DOS partitioning looks like this: +----------------------------------------------------------------------+ | | |NetBSD disklabel disk geometry: | |cylinders: 16383 heads: 16 sectors/track: 63 (1008 sectors/cylinder) | | | |BIOS disk geometry: | |cylinders: 1024 heads: 255 sectors/track: 63 (16065 sectors/cylinder) | | | |Partition table: | |0: sysid 169 (NetBSD) | | start 63, size 120103137 (58644 MB), flag 0x80 | | beg: cylinder 0, head 1, sector 1 | | end: cylinder 1023, head 254, sector 63 | |1: | |2: | |3: | | | +----------------------------------------------------------------------+ - Plan your disk layout Take a minute to plan, how you want your RAID devices structured. In addition to the normal, primary disklabel, every RAID set needs its own disklabel. My own overall disk layout looks like this: +---------------------------------------------------------------------+ | | |wd0 | | DOS Partition 0 (active) | | Slice a: 768MB (fstype=RAID) will be the RAID set raid0 | | raid0a: will be the root Filesystem | | Slice b: 2048MB (fstype=RAID) will be the RAID set raid1 | | raid1a: will be the swap partition | | Slice c: unused | | Slice d: unused | | Slice e: 55827MB (fstype=RAID) will be the RAID set raid2 | | raid2a: will be the /usr Filesystem | | raid2b: will be the /var Filesystem | | raid2e: will be the /home Filesystem | | Slice f: 1MB (fstype=4.2BSD) will contain secondary bootblocks | | DOS Partition 1 | | unused | | DOS Partition 2 | | unused | | DOS Partition 3 | | unused | | | +---------------------------------------------------------------------+ - Label the disk wd1 according to your plan cd /root disklabel wd1 >disklabel.wd1 vi disklabel.wd1: - disk: mydisk1 - create your disklabel with a partition for each RAID device you want to use - make sure that the partitions for the RAID devices have enough room for the filesystems you want to create in them later - make sure that the partitions for the RAID devices are fstype RAID Example: +-------------------------------------------------------------------------------+ | | |# /dev/rwd1d: | |type: unknown | |disk: mydisk1 | |label: | |flags: | |bytes/sector: 512 | |sectors/track: 63 | |tracks/cylinder: 16 | |sectors/cylinder: 1008 | |cylinders: 16383 | |total sectors: 120103200 | |rpm: 7200 | |interleave: 1 | |trackskew: 0 | |cylinderskew: 0 | |headswitch: 0 # microseconds | |track-to-track seek: 0 # microseconds | |drivedata: 0 | | | |6 partitions: | | # size offset fstype [fsize bsize cpg/sgs] | | a: 1572864 2111 RAID | | b: 4194304 1574975 RAID | | c: 120103137 63 unused 0 0 # (Cyl. 0*- 119149) | | d: 120103200 0 unused 0 0 # (Cyl. 0 - 119149) | | e: 114333921 5769279 RAID | | f: 2048 63 4.2BSD 1024 8192 32 | | | +-------------------------------------------------------------------------------+ In the above example, I still have a normal FFS Partition (wd1f) at the *beginning* of the disk. It is needed, because installboot (at the time of NetBSD 1.5.2) can't correctly write the first- and second stage bootloader for booting off a RAID device. - Copy disklabel.wd1 to disklabel.wd0 and change the disk: identifier to mydisk0. You will need this file later (wd0 will look exactly the same, as it will be the mirror of wd1). - Label wd1 and make it bootable: disklabel -R -r wd1 disklabel.wd1 newfs /dev/wd1f /usr/mdec/installboot /usr/mdec/biosboot.sym /dev/rwd1f - Prepare /etc/raid0.conf (raid1.conf, raid2.conf) with a fake, NONEX­ ISTING first component) Example +-------------+ | | |START array | |1 2 0 | | | |START disks | |/dev/wd2a | |/dev/wd1a | | | |START layout | |128 1 1 1 | | | |START queue | |fifo 100 | | | +-------------+ In the above example, /dev/wd2a doesn't exist. It will later be re­ placed by /dev/wd0a. The second component, /dev/wd1a, is the slice which will be used for the second half of the mirror, which we are setting up right now (see raidctl(8) for a description of what be­ longs in raid?.conf files). - Configure the RAID Devices, but don't initialize them: raidctl -C /etc/raid0.conf raid0 raidctl -I 2001100 raid0 raidctl -C /etc/raid1.conf raid1 raidctl -I 2001101 raid1 raidctl -C /etc/raid2.conf raid2 raidctl -I 2001102 raid2 - Disklabel the RAID Devices according to your plan disklabel raid0 >disklabel.raid0 vi disklabel.raid0 disklabel -R -r raid0 disklabel.raid0 disklabel raid1 >disklabel.raid1 vi disklabel.raid1 disklabel -R -r raid1 disklabel.raid1 disklabel raid2 >disklabel.raid2 vi disklabel.raid2 disklabel -R -r raid2 disklabel.raid2 Example disklabels for raid0 and raid1 +---------------------------------------------------------------------------------+ | | |# /dev/rraid0d: | |type: RAID | |disk: raid | |label: default label | |flags: | |bytes/sector: 512 | |sectors/track: 128 | |tracks/cylinder: 8 | |sectors/cylinder: 1024 | |cylinders: 1535 | |total sectors: 1572736 | |rpm: 7200 | |interleave: 1 | |trackskew: 0 | |cylinderskew: 0 | |headswitch: 0 # microseconds | |track-to-track seek: 0 # microseconds | |drivedata: 0 | | | |4 partitions: | |# size offset fstype [fsize bsize cpg/sgs] | | a: 1572736 0 4.2BSD 1024 8192 64 # (Cyl. 0 - 1535*) | | d: 1572736 0 unused 0 0 | | | +---------------------------------------------------------------------------------+ +----------------------------------------------------------+ | | |# /dev/rraid1d: | |type: RAID | |disk: raid | |label: default label | |flags: | |bytes/sector: 512 | |sectors/track: 128 | |tracks/cylinder: 8 | |sectors/cylinder: 1024 | |cylinders: 4095 | |total sectors: 4194176 | |rpm: 7200 | |interleave: 1 | |trackskew: 0 | |cylinderskew: 0 | |headswitch: 0 # microseconds | |track-to-track seek: 0 # microseconds | |drivedata: 0 | | | |4 partitions: | |# size offset fstype [fsize bsize cpg/sgs] | | a: 4194176 0 swap | | d: 4194176 0 unused 0 0 | | | +----------------------------------------------------------+ - Format the new filesystems newfs /dev/raid0a newfs /dev/raid2a newfs /dev/raid2b newfs /dev/raid2e - Mount the new filesystems and copy the data from the old ones over: The root FS: mount /dev/raid0a /mnt (cd /; tar -lpcf - .) | (cd /mnt; tar -xpf -) The /usr FS: mount /dev/raid2a /mnt/usr (cd /usr; tar -lpcf - .) | (cd /mnt/usr; tar -xpf -) The /var FS: mount /dev/raid2b /mnt/var (cd /var; tar -lpcf - .) | (cd /mnt/var; tar -xpf -) The /home FS: mount /dev/raid2e /mnt/home (cd /home; tar -lpcf - .) | (cd /mnt/home; tar -xpf -) - Adjust the fstab on the RAID root filesystem: vi /mnt/etc/fstab Example +-----------------------------+ | | |/dev/raid0a / ffs rw 1 1 | |/dev/raid1a none swap sw 0 0 | |/dev/raid2a /usr ffs rw 1 2 | |/dev/raid2b /var ffs rw 1 2 | |/dev/raid2e /home ffs rw 1 2 | | | +-----------------------------+ - (NetBSD < 1.6 only:) Enter the following line into /mnt/etc/rc.d/swap1 (right after the line REQUIRE: disks) # KEYWORD: shutdown - (NetBSD >= 1.6 only:) Enter the following line into /mnt/etc/rc.conf (see rc.conf(5)) swapoff=yes This provides for the unconfiguration of the swap space on shutdown, so it's parity won't have to be rebuilt every time you reboot the system. - Make the filesysytems on the RAID devices autoconfigurable: raidctl -A root raid0 raidctl -A yes raid1 raidctl -A yes raid2 Step 3: Boot off 2nd disk with RAIDFrame active _______________________________________________________________________ - Now reboot the system and when the NetBSD second stage bootmessages appear, type and say: NetBSD i386 < 1.6: boot wd1a:netbsd NetBSD i386 >= 1.6: boot hd1a:netbsd (Yes, wd1a/hd1a, not raid0a. This will boot from the second disk and autoconfigure the filesystems on the raid sets) Expect a lot of errors rushing by during the boot process. The reason for those are the missing wd2 device we have in our raid?.conf files. The system will (should :-) boot nonetheless off the really existing half of the mirror, wd1! - Your system should be running from the RAID sets now. Check this by issuing 'mount'. The output should look something like: Example +--------------------------------------+ | | |/dev/raid0a on / type ffs (local) | |/dev/raid2b on /var type ffs (local) | |/dev/raid2a on /usr type ffs (local) | |/dev/raid2e on /home type ffs (local) | | | +--------------------------------------+ Step 4: Repartition/relabel 1st disk the same way as the 2nd _______________________________________________________________________ - Now prepare the wd0 disk in the same way, as you prepared wd1: - fdisk -u wd0 - fdisk -a -0 wd0 - disklabel -R -r wd0 disklabel.wd0 - newfs /dev/rwd0f - /usr/mdec/installboot /usr/mdec/biosboot.sym /dev/rwd0f - pull wd0 into mirror Take a look at 'raidctl -s raid0' now. You will see that we have a 'component0' which is failed, and a '/dev/wd1a' which is in state 'optimal'. Now, we add the devices from the first disk as hotspares to the RAID sets, and sync them. Upon the next reboot, the RAIDFrame won't remem­ ber that they were hotspares but will accept them as the first half of our mirror, according to the entries in /etc/raid?.conf. raid0: - raidctl -a /dev/wd0a raid0 - raidctl -vF component0 raid0 - raidctl -vP raid0 (not needed with NetBSD >= 1.6 anymore) raid1: - raidctl -a /dev/wd0b raid1 - raidctl -vF component0 raid1 - raidctl -vP raid1 (not needed with NetBSD >= 1.6 anymore) raid2: - raidctl -a /dev/wd0e raid2 - raidctl -vF component0 raid2 - raidctl -vP raid2 (not needed with NetBSD >= 1.6 anymore) Look at the raidctl(8) man page, to see what the 'raidctl -a', 'raid­ ctl -vF' and the 'raidctl -vP' commands do. - Edit /etc/raid0.conf, raid1.conf, raid2.conf and change the nonexistent components with the real ones: wd2a -> wd0a wd2b -> wd0b wd2e -> wd0e - reboot - Now, you should be done. Check the state of the RAID devices with raidctl -s raid0 raidctl -s raid1 raidctl -s raid2 The state of all components should be 'optimal' and Parity status should be 'clean' And, in order to be prepared when Murphy strikes: Document your instal­ lation and read the fine manuals: raid(4) and raidctl(8)! Optional Step 5: Feedback and varia _______________________________________________________________________ Did this paper help you? I would be happy to hear from you, if it did. It would motivate me to write other howtos like this. If you found errors or if you think my explanations are not clear enough, please email me also. You can reach me at ck at neverland dot ch For general RAIDFrame or NetBSD questions, please don't email me, but the appropriate NetBSD mailing list! (for example: netbsd-users@netb­ sd.org or port-i386@netbsd.org) Thanks to Greg Oster for enlightening me about autoconfiguration. Thanks to Michael Canella for the constructive feedback. I also would like to thank all those people, which make NetBSD and RAIDFrame happen! Great stuff! This document has been created using vim, groff and ghostscript. Also very great stuff!