Low Orbit Flux Logo 2 D

Linux RAID

Here is what we are going to cover:

Setup RAID 10 Array
Replace a Drive
Setting Up RAID 1 On The Root Disk
Disk Health with Smartmon Tools



Assumptions:

Setup RAID 10 Array


The Linux software RAID subsystem allows for the creation of really excellent RAID 10 arrrays. For these instructions we are going to assume you are running Ubuntu 16.04. If you are running another distro or version package names, etc. might be different but mdadm should function mostly the same.

To start, we want to make sure we install mdadm. We run apt-get update to get a fresh view of what is on the repo. We then install the mdadm package.


apt-get update
apt-get install mdadm

This will show relavant information about block devices on the system.


lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

This is what it looks like on my NAS after setting everything up.

Linux RAID 10 lsblk output

Create the actual array. Specify all four devices. Other numbers of drives are valid but we will use four for this example.


mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde

As an alternative it is also possible to create a four drive array with only two disks. The other two can be added later. You would use the keyword 'missing' in place of the second and third disk.


mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sdb missing /dev/sdc missing

Check the details of the array.


mdadm --detail /dev/md0
cat /proc/mdstat

Save everything to the config file. The path to this file might be slightly different on other distros.


mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
cat /etc/mdadm/mdadm.conf

Create the filesystem on the array and mount it.


mkfs.ext4 -F /dev/md0
mkdir /storage1
mount /dev/md0 /storage1
df -h

Add the mount to the fstab file.


echo '/dev/md0 /storage1 ext4 defaults,nofail,discard 0 0' >> /etc/fstab

Unmount the filesystem for now.


umount /storage1
df -h

Mount it with a single argument (relying on the fstab entry).


mount /storage1
df -h

You might want to change the ownership if it will be used by another user other than root. If you had copied data onto the file system from somewhere else, you might want to do this recursively.


cd /storage1/
ls -la
chown -R user1:user1 .
ls -la

If you had left two disks out, you can add them in when ready. Add the two missing disks with the following commands.


mdadm /dev/md0 --add /dev/sdb
mdadm /dev/md0 --add /dev/sdd

Watch them get synced.


mdadm --detail /dev/md0
cat /proc/mdstat

You can use mdadm for monitoring. The command can be daemonized and can send emails, etc. This isn't a real monitoring solution. If the daemon were to die you would silently lose your monitoring.


mdadm --monitor /dev/md0

Replace a Drive

assume we have two partitions

check serial number


smartctl -a /dev/sdb|grep -i serial

mark the drive failed


mdadm --manage /dev/md0 --fail /dev/sdb1

cat /proc/mdstat

remove from the array


mdadm --manage /dev/md0 --remove /dev/sdb1

Same for other partition:


mdadm --manage /dev/md1 --fail /dev/sdb2
mdadm --manage /dev/md1 --remove /dev/sdb2

shutdown -h now

physically swap the disk boot the system

partition it the same as an existing disk in the array


sfdisk -d /dev/sda | sfdisk /dev/sdb

fdisk -l

mdadm --manage /dev/md0 --add /dev/sdb1
mdadm --manage /dev/md1 --add /dev/sdb2

Watch it syncing and wait until it is complete:


cat /proc/mdstat   
mdadm --detail /dev/md0

Setting Up RAID 1 On The Root Disk


apt-get install mdadm rsync initramfs-tools

partition second disk with fdisk / cfdisk
same size partitions as first in sectors
type fd ( linux raid )

add second disk to array


    mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1  
    mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb2


format second disk


    mkswap /dev/md1  
    mkfs.ext4 /dev/md0


Add this line to the end ( not sure why )


    echo "DEVICE /dev/sda* /dev/sdb*" >> /etc/mdadm/mdadm.conf
    mdadm --detail --scan             >> /etc/mdadm/mdadm.conf  

dpkg-reconfigure mdadm

Check status:


    watch -n1 cat /proc/mdstat    # watch disk sync status if syncing

dpkg-reconfigure grub-pc    # select both /dev/sda and /dev/sdb (but not /dev/md0) as installation targets
                            # installs bootloader on both drives
                            # rebuild init ramdisk and bootloader with raid support

Copy from old drive to new one


        #mkdir /tmp/mntroot  
        #mount /dev/md0 /tmp/mntroot  
        #rsync -auHxv --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* /* /tmp/mntroot/  
        rsync -auHxv --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* /* /media/user1/e384aca9-574c-4011-8abd-6f2ee04a31e3

/tmp/mntroot/etc/fstab # wherever this is mounted, just swap sda or uid for raid disks
##/dev/sda1
##/dev/sda2
/dev/md0
/dev/md1


====================

Boot from second disk

GRUB menu, hit ā€œeā€

- will look a little different, just put (md/0) and /dev/md0 in the right places

   
    set root='(md/0)'  
    linux /boot/vmlinuz-2.6.32-5-amd64 root=/dev/md0 ro quiet



run mount or df -h
look for this:
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)



partition first disk with fdisk / cfdisk
same size partitions as second in sectors
type fd ( linux raid )


add to array


    mdadm /dev/md0 -a /dev/sda1  
    mdadm /dev/md1 -a /dev/sda2


Check status:


    watch -n1 cat /proc/mdstat    # watch disk sync status if syncing

update-grub               # update grub - probably not needed
dpkg-reconfigure grub-pc  # cuz grub update didn't do both disks

shutdown -h now

remove second drive and test booting

add it back and start up


cat /proc/mdstat    # should show second drive as disabled

mdadm /dev/md0 -a /dev/sdb1   # add it back
mdadm /dev/md1 -a /dev/sdb2   # add it back

shutdown, remove disk 1, test boot off disk2
shutdown, add disk back, boot back up

mdadm /dev/md0 -a /dev/sda1    # add disk back to array

dpkg-reconfigure mdadm   # enabling periodic RAID checks that are built into the package
OR<
echo "check" > /sys/block/md0/md/sync_action   # create a cron job

install smartmontools


/etc/smartd.conf   #  install the smartmontools package and run weekly SMART checks
/dev/sda -a -d ata -o on -S on -s (S/../.././02|L/../../6/03)  
/dev/sdb -a -d ata -o on -S on -s (S/../.././02|L/../../6/03)

smartctl -a /dev/sda    # check SMART log for errors

You can learn more HERE.




Disk Health with Smartmon Tools

Update repo info and install Smartmon tools.


apt-get update
apt install smartmontools

Get a list of disks. Either way works.


fdisk -l|grep -i "Disk "|grep -v ram|grep -v identifier
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

Check drive status.


smartctl -a /dev/sdb

Grep for errors.


smartctl -a /dev/sdb|grep -i error

Run a short test. This will run in the background and will take a few minutes. Once complete it will show in the smartctl -a' output.


smartctl -t short /dev/sde

Wait a few minutes and check the drive again.


smartctl -a /dev/sdb

Good Resources