So oddly last night after not drinking, as it had been quite a heavy couple of weeks I didn’t have too much problem sleeping. Was going to take a pill, didn’t need to. But in the morning I was tingling like I was on a mission, hadn’t had that feeling since I gave up Prozac, very strange. Thankfully it did fade.
The rest of the day was productive, if wet. I really must do the write up for the SSL thing tomorrow and get that out of the way. Also must try and finish this bloody jigsaw.
So after the great power outage fiasco I decided to look at some settings and perform some tests. Sure enough the server BIOS was set to ‘power off’ after AC fail, so changed this to ‘on’. Then did a full AC outage test. It lasted well over an over, but when it got to about ten minutes remaining both the server and the firewall shutdown. I then returned the AC supply. The UPS did as it should, cut the output relay, charged for a bit and then turned the output on. The firewall dutifully powered back up, however the server didn’t. Because the server had shut down gracefully, it didn’t actually suffer an AC fail when running, so didn’t power back on. Huston we have a problem. So off to the BIOS again looking for a ‘wake on lan’ setting, nowhere to be found. Played around with a couple of Ubuntu utils and identified the network card etc. It said it could support WOL using the magic packet, but it was currently disabled, some Googling and now a service was setup to reenable WOL after every boot. However no matter how many packets I sent it and wether it was a full shutdown, sleep or hibernate it would not wake up. As the setting wasn’t in the BIOS my guess is the hardware after the NIC just doesn’t support it. Back to the BIOS. One interesting thing I did notice in there was a setting for RTC Alarm, this looked interesting. I set the clock for a few minutes later and shutdown the machine. Sure enough when the clock struck the machine rebooted. So I’ve now set it to one minute past midnight (UTC). I’m now running a fall AC fail test again, there is about fifteen minutes of power left and it’s coming up to ten-fifteen. So when it shuts down at about ten-thirty I’ll restore the power. Hopefully then just after eleven (we are on BST at the moment so UTC+1) all should be backup.
So I got up at 2AM, after going to bed at 10 to just lie awake for four hours. Dogs had already gone into kennels earlier that day. Got into car and drove to airport. First thing I knew about there being ‘trouble at mill’ was an email marked ‘UPS ALERT:ONBATT’. Panic. Okay, every now and then you do get a ‘power blip’ and the UPS cuts in, like it should and smooths things over using the battery until it passes, usually this is just a few seconds, Panic, still sets in. This was 6:25 on the one day I’m going to be away for over a week (don’t get ideas, the security is mega). Second email. ‘UPS ALERT:ONLINE’. This I at 8:04. Relief spreads over me like a heat rash. I’m reading this while at the luggage carousel. Now, all good. Until. 14.38. ‘UPS ALERT:ONBATT’. WTF. Twice in one day, what are the chances of that in the same day AND the day I’m away! And that’s the last I would hear.
Panic has now set in. I’ve lost email, both personal and company accounts, and web servers, again both personal (this one) and company accounts. I can’t ping the server. Can’t access the CCTV or the alarm (sorry folks it won’t talk to me, but is battery backed up and is loud as fuck, so again, no ideas). Panic again sets in. I can connect to the VPN. Good. Means the Zen ISP connection is good. I can log into the firewall. Check both routes, yep, 5G on three (that an anachronism?) is also up. Can’t ping the server. Look at the DHCP leases, all looks a bit dead. Panic is now very much set in. Server must have shutdown and not come back up. Panic is now in full swing. Contact my friend Elaine, she’s going to visit tomorrow and attempt a manual reboot.
Fast forward to today. Tried to connect to alarm panel, offline (again, it works fine when offline). Tried to connect to CCTV, that is back up, so the network switch is online. Checked the heating, yep, that’s good and the ring doorbell, so the WiFi is definitely up. Tried to ping the server and no luck. Connected to the firewall again and checked the DHCP leases, WiFi access point, switch, heating, doorbell, all good. Server, out to lunch. Elaine calls, she’s in the office (navigated the alarm and the lock). Now with a torch and a video call. I guide her towards the correct unit, and I have a lot in that cabinet. There is a UPS, a firewall, two network switches, the main server, backup drive and about sixty miles of cabling. After about ten minutes with a torch and pointing at things the server is identified and after more fumbling the power switch is located. It does indeed appear that it is powered down. A prod of the now found power button and the server is alive. Gave it a few minutes and then tried to refresh the email. The email server is now alive. ‘UPS ALERT: SHUTDOWN’, 15:53. ‘UPS ALERT: LOWBATT’, 15:53. So the power was out for over an hour, the UPS had run out of battery and sent the ‘shutdown’ command. There are a bunch of emails just after saying various services were shutting down, and that was it, it powered off, safely, as programmed to do so. I assume the firewall was also given the signal and dutifully shutdown. Until the power was restored. Depending when it happened the UPS would have done a couple of things. After it sent the shutdown command it knows it has to do a full shutdown and restart, to power things back up. So I assume at some point the power came back on. The UPS would have charged up to a point it was no longer in ‘low battery’ mode and would have pulled the power relays. This would have cut the power completely to whatever was connected. It would then enraged the relays again and ‘power up’ this should have caused everything connected to just ‘reboot’. Obviously the firewall obeyed and dutifully powered up, together with the ONT (the fibre interface) and the 5G modem (three, lets not go there). So connecting to the VPN and connecting to the firewall would all be okay. But, it seems the server did not power itself back on. This is probably just an over site in the bios settings. I expect it’s set to ‘last known config’, which of course would have been ‘off’ as it was told to shutdown by the UPS. It really needs to be set to ‘power on’. Or indeed, I should also configure it to ‘wake-up on lan’, I could then at least kick it from the firewall and reboot it. So there are things to simulate and test when I get back.
What are the chances of two power cuts on the same day and the same day I leave for a week? Low, very low. But it did happen. Thanks to my hero Elaine, I can ‘unpause’ my life again.
So the last couple of days I resurrected a very old laptop and I mean very old to the some of twenty plus years. It has videos my ex. husband made when we first start going out. I wanted to use it for other things, I’ll come on to that in a bit. But there was some stuff on there I would like to keep. Trying to connect it to a modern network share wasn’t happening. Trying to back up to a 128GB USB stick wasn’t happening. Eventually managed to get it to talk to an old 300GB external drive. But rather than just do a backup, instead I ‘virtualised’ it. Using VMWare converter I turned it into a bunch of files. I then transferred this to the USB stick and used VMWare tools to create a virtual machine. Works perfect, boots up into Windows XP in a window and you can use it just like the old laptop.
Now the old laptop I wanted to reuse as a Ubuntu desktop machine just for playing about. This laptop only has a 32 bit processor so had to go back quite a few versions to find one which would install. Eventually after a couple of days of pissing about and various updates I’ve got it up to 18.04. Sadly it isn’t really useable, the machine only has 512MB of ram and a very slow ‘M’ series processor. I looked and I could buy a 1GB memory upgrade and a new battery for around £60. But sadly looking around I could get a second hand laptop with much higher specs. for the same money. So I’m going to reinstall Windows XP Home on it and see how it runs, but have a feeling, with its dead CMOS battery, dead main battery, slow processor and lack of RAM it will be making its final journey to the recycling box. Shame I had many fond memories of that machine.
Still, I have found another unused laptop which has a 64 bit processor and 2GB of RAM, I think this maybe my next Ubuntu desktop target.