“2 frigging weeks this guys been AWOL! Where the hell is he? How am I supposed to manage this infrastructure myself?”
It had been two weeks since I’d heard from my server admin for one of my clients and little glitches had become bigger and bigger issues. I had already sent the emails begging for support, asking why he was blowing off the client, etc.etc. and still deafening silence. Finally yesterday, one of our sites was under attack(so tired of these attacks from Eastern Europe and China) and the server had to be rebooted by me, because despite my extensive experience with servers and tech in general, I sure as hell don’t know how to go in and resolve every potential issue regarding a Linux box at Amazon and multiple WordPress installs, out of memory issues, hacking attacks, etc. I use a lot of great tools and plugins for WordPress to fend them off, but this was one of those reboot and hope scenarios. After a couple of reboots, things eventually stabilized and the site was back to its zippy self, but not before I fired off screaming email about firing him and filing a complaint with Elance. I was steaming pissed off and began the search for a replacement all the while even more annoyed, that this was taking away from serious work I needed to get done for other clients.
At about midnight this email rolled in:
“Hello Brad, I am extremly sorry for the delay. Actually, here heavy flood for past 10 days. Now only I am able to connect to internet at least , checking the mails one by one and replying. Once again I am sorry for the delay.”
This guys been worried about his home, his family, and his community and I’m freaking out about a web site. Yes, I have every right to be frustrated, but what could I have done differently to prevent the issue and not have to go off half-cocked at a guy that for the last 10 days wasn’t thinking about a web site or me, but instead was watching his world get destroyed?
- Have 2 admins in different parts of the world. In other words, always have a backup or multiple backups to your current solution to any technical or business process.
- Have an admin here locally.
- Waited until I heard what was going on.
- Practice mindfulness and realize that while stressful, the situation wasn’t a major personal crises requiring that level of emotional involvement.
- Learn how to administer and manage the more technical aspects of the servers myself.
Number 5 isn’t practical, but the rest are and should be. Even the best infrastructure in the world can fail and you better
The moral of the story is to always have a backup plan, so you don’t have to take your lack of foresight out on someone else. Think about all the pieces of your machine and what you will do if one breaks, goes missing, or isn’t available. How will you cope with that? I already know this and practice it regularly, but this was one of those times where I missed something in the process. The good news is that a flood in India helped me get my act together, fill the gaps and forced me to learn from my mistake.
Another good lesson to remember is to see the lessons in every bad or good thing that happens. There is always an opportunity learn and the best lessons are often found in the stupidest mistakes. The 2 big lessons here:
- Don’t jump to conclusions and make accusations.
- Keep your temper under control. If it’s not life or death – literally, then take a break and walk away. Hug your kid.
- Be prepared for the worst so you want be hurt as badly when it happens. Look at every nut, bolt, gear, and break point of your business and find the places where if you have a problem you will lose your mind. Then make sure you won’t by securing an alternative.