Linux RAID

Here is what we are going to cover:

Setup RAID 10 Array
Replace a Drive
Setting Up RAID 1 On The Root Disk
Disk Health with Smartmon Tools


Assumptions:

  • You are running Ubuntu 16.04 or 18.04
  • Most commands on this page are run as root. You may want to use sudo instead.

Setup RAID 10 Array


The Linux software RAID subsystem allows for the creation of really excellent RAID 10 arrrays. For these instructions we are going to assume you are running Ubuntu 16.04. If you are running another distro or version package names, etc. might be different but mdadm should function mostly the same.

To start, we want to make sure we install mdadm. We run apt-get update to get a fresh view of what is on the repo. We then install the mdadm package.


apt-get update
apt-get install mdadm

This will show relavant information about block devices on the system.


lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

This is what it looks like on my NAS after setting everything up.

Linux RAID 10 lsblk output

Create the actual array. Specify all four devices. Other numbers of drives are valid but we will use four for this example.


mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde

As an alternative it is also possible to create a four drive array with only two disks. The other two can be added later. You would use the keyword 'missing' in place of the second and third disk.


mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sdb missing /dev/sdc missing

Check the details of the array.


mdadm --detail /dev/md0
cat /proc/mdstat

Save everything to the config file. The path to this file might be slightly different on other distros.


mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
cat /etc/mdadm/mdadm.conf

Create the filesystem on the array and mount it.


mkfs.ext4 -F /dev/md0
mkdir /storage1
mount /dev/md0 /storage1
df -h

Add the mount to the fstab file.


echo '/dev/md0 /storage1 ext4 defaults,nofail,discard 0 0' >> /etc/fstab

Unmount the filesystem for now.


umount /storage1
df -h

Mount it with a single argument (relying on the fstab entry).


mount /storage1
df -h

You might want to change the ownership if it will be used by another user other than root. If you had copied data onto the file system from somewhere else, you might want to do this recursively.


cd /storage1/
ls -la
chown -R user1:user1 .
ls -la

If you had left two disks out, you can add them in when ready. Add the two missing disks with the following commands.


mdadm /dev/md0 --add /dev/sdb
mdadm /dev/md0 --add /dev/sdd

Watch them get synced.


mdadm --detail /dev/md0
cat /proc/mdstat

You can use mdadm for monitoring. The command can be daemonized and can send emails, etc. This isn't a real monitoring solution. If the daemon were to die you would silently lose your monitoring.


mdadm --monitor /dev/md0

Replace a Drive

assume we have two partitions check serial number

smartctl -a /dev/sdb|grep -i serial
mark the drive failed

mdadm --manage /dev/md0 --fail /dev/sdb1
cat /proc/mdstat remove from the array

mdadm --manage /dev/md0 --remove /dev/sdb1
Same for other partition:

mdadm --manage /dev/md1 --fail /dev/sdb2
mdadm --manage /dev/md1 --remove /dev/sdb2

shutdown -h now
physically swap the disk boot the system partition it the same as an existing disk in the array

sfdisk -d /dev/sda | sfdisk /dev/sdb 

fdisk -l

mdadm --manage /dev/md0 --add /dev/sdb1
mdadm --manage /dev/md1 --add /dev/sdb2
Watch it syncing and wait until it is complete:

cat /proc/mdstat   
mdadm --detail /dev/md0

Setting Up RAID 1 On The Root Disk

apt-get install mdadm rsync initramfs-tools partition second disk with fdisk / cfdisk
same size partitions as first in sectors
type fd ( linux raid )

add second disk to array
mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb2

format second disk
mkswap /dev/md1
mkfs.ext4 /dev/md0

Add this line to the end ( not sure why )
echo "DEVICE /dev/sda* /dev/sdb*" >> /etc/mdadm/mdadm.conf
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
dpkg-reconfigure mdadm
Check status:
watch -n1 cat /proc/mdstat # watch disk sync status if syncing
dpkg-reconfigure grub-pc # select both /dev/sda and /dev/sdb (but not /dev/md0) as installation targets
# installs bootloader on both drives
# rebuild init ramdisk and bootloader with raid support
Copy from old drive to new one
#mkdir /tmp/mntroot
#mount /dev/md0 /tmp/mntroot
#rsync -auHxv --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* /* /tmp/mntroot/
rsync -auHxv --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* /* /media/user1/e384aca9-574c-4011-8abd-6f2ee04a31e3
/tmp/mntroot/etc/fstab # wherever this is mounted, just swap sda or uid for raid disks
##/dev/sda1
##/dev/sda2
/dev/md0
/dev/md1

====================

Boot from second disk

GRUB menu, hit "e"

- will look a little different, just put (md/0) and /dev/md0 in the right places

set root='(md/0)'
linux /boot/vmlinuz-2.6.32-5-amd64 root=/dev/md0 ro quiet


run mount or df -h
look for this:
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)


partition first disk with fdisk / cfdisk
same size partitions as second in sectors
type fd ( linux raid )

add to array
mdadm /dev/md0 -a /dev/sda1
mdadm /dev/md1 -a /dev/sda2

Check status:
watch -n1 cat /proc/mdstat # watch disk sync status if syncing
update-grub # update grub - probably not needed
dpkg-reconfigure grub-pc # cuz grub update didn't do both disks
shutdown -h now
remove second drive and test booting

add it back and start up
cat /proc/mdstat # should show second drive as disabled
mdadm /dev/md0 -a /dev/sdb1 # add it back
mdadm /dev/md1 -a /dev/sdb2 # add it back
shutdown, remove disk 1, test boot off disk2
shutdown, add disk back, boot back up
mdadm /dev/md0 -a /dev/sda1 # add disk back to array
dpkg-reconfigure mdadm # enabling periodic RAID checks that are built into the package
OR
echo "check" > /sys/block/md0/md/sync_action # create a cron job
install smartmontools
/etc/smartd.conf # install the smartmontools package and run weekly SMART checks
/dev/sda -a -d ata -o on -S on -s (S/../.././02|L/../../6/03)
/dev/sdb -a -d ata -o on -S on -s (S/../.././02|L/../../6/03)
smartctl -a /dev/sda # check SMART log for errors
You can learn more HERE.


Disk Health with Smartmon Tools

Update repo info and install Smartmon tools.


apt-get update
apt install smartmontools

Get a list of disks. Either way works.


fdisk -l|grep -i "Disk "|grep -v ram|grep -v identifier
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

Check drive status.


smartctl -a /dev/sdb

Grep for errors.


smartctl -a /dev/sdb|grep -i error

Run a short test. This will run in the background and will take a few minutes. Once complete it will show in the smartctl -a' output.


smartctl -t short /dev/sde

Wait a few minutes and check the drive again.


smartctl -a /dev/sdb

Good Resources