My good lady wife decided to make some Gluten Free bread in the Bread maker, which hasn’t been used for a long time. Long story cut short, she put the stuff in the bread maker and went shopping. I was in the garden pruning and tidying etc or maybe just enjoying the sun 🙂
At some point I decided to make myself a cup of tea and realised that the kettle wasn’t working! I quickly realised that there was no power to the (wall) sockets and found that RCD in the fuse board (Consumer Unit) had tripped. I reset the RCD and everything seemed to power up again but it only lasted some 10 minutes before it tripped again, resetting the RCD sorted it. We’re blaming the bread maker but it hasn’t tripped the power again!
At this point I realised that the Centos server would probably need restarting! It restarted but was stuck in “Emergency mode”, not good! It appears that the battery in the UPS was in need of replacement and the server didn’t shutdown gracefully 🙁 This made me think maybe something had been corrupted so I logged into the “Emergency” bit to have a look and noticed that the mount points didn’t look right! I decided to edit fstab to comment out the 2 x 1TB data drives and the 2TB USB drive, leaving just the “boot/system” drive. Rebooted and all seemed fine. Added back the 2 data drives in turn followed by rebooting in between. No problems, that I could see, then added the USB drive and got Emergency mode again on reboot. Edited fstab again and physically removed the 2TB USB drive. Server now seemed to be working fine, minus the USB drive.
This is a long story and as it turns out, my lack of (Linux/CentOS) knowledge was the biggest problem. I know what I’m doing with Microsoft Windows, I GET Windows but struggle with Linux.
Decided to stick CentOS on a spare laptop and treated myself to the Gnome desktop. I then “Hot plugged” the USB drive and it appeared, all data seemed to be intact – phew. Run fsck -f and no problems found.
OK, it’s not the USB drive, mmmm! Hopped on to server and ran fsck -f on both of the data drives (after unmounting) and again no problems found mmmmm!
Also noticed that the data on the data drives had become “jumbled” some folders that are usually on data01 were duplicated on data02 and other strange things. Things were now starting to grind me down. Decided to re-image, using an image I’d done a few days before the power loss; using Mondoarchive. It didn’t change anything, I confirmed that the image had taken as there were updates available that I’d done after I’d done the image.
Had a good trawl on the Web and and stumbled on a couple of articles about systemd, which suggested that fstab was now no longer used and that the contents of fstab should be copied into systemd/system. Apparently since version 7 of CentOS, the boot process uses “parallel processing” and the order of the drive mounts in fstab were basically ignored. I noticed that the “boot” drive was trying to use sdc1 which was what the USB drive was assigned in fstab.
Ok, I’m now out of my depth and looked on the CentOS forum for help. The people on there are superstars, turns out the solution was REALLY SIMPLE. All I had to do was edit fstab and use the UUID of the drives and NOT assign hard links etc sda1 sdb1 etc as systemd would sort this out on a reboot. Apparently using UUID or “Labels” has always been the preferred for many years but my knowledge of Unix/Linux was old school – stuff I learned administering some 200 SCO Unix boxes. Over the years I’ve played with a few flavours of Linux; Solaris, Ubuntu, SUSE and others but I didn’t keep up with the modern methods and most of the stuff I knew I’ve forgotten or become confused about. Old dog new tricks!
CentOS 7 is without any doubt very robust and reliable; if you don’t have an idiot looking after it 🙂 The last time I had rebooted the server was nearly a year ago and I always keep up with updates. Due to my lack of (Linux) skills, I tend to admin the server remotely using Webmin, Putty and WinSCP so I guess I’m lazy but I GET Windows!
It’s been a couple of weeks learning new stuff and re-learning stuff I’d forgotten about. It hasn’t been fun but I’m now more knowledgeable, I should have asked for help sooner but hey ho. I dare say I’ll have forgotten everything in a few weeks 🙂
Server now up n running, data now in all the right places, rsync backups running nicely on cron jobs.
The new UPS
It seemed more cost effective to buy a new UPS instead of a new battery. Being budget minded, I opted for a Salicru SPS 700 One which was less than £50 from ebuyer.com. It has software that is compatible with CentOS, to do auto “graceful” shutdown but it seems to need a GUI so haven’t tried it. I did try the popular APSCUPSD but it kept losing the connection; NUT (networkupstools.org) was the saviour. NUT is very easy to use, in my case I used the “blazer_usb” driver and after a reboot; it worked great. So I now have a working UPS that should do a graceful shutdown in the event of a power cut. I haven’t plucked up courage to kill the power yet but will once I know everything is backed up 🙂
CentOS 7 is awesome, it’s very robust and reliable but you need to learn how to use it! SystemD might seem a little strange but it’s worth getting to know.
It takes more than a Bread Maker to kill CentOS 7 🙂
Make sure your UPS is up for the job.