16 February 2013

Wait and Think and IPMI

Hi, me again. About a month without a post... Let's jump straight in.

I did a stupid thing at Job yesterday.  We have a client and he has a system in the data-center. And the system is a CentOS that has nothing on it. So far so good. Then, he decides that he wants to put something on it. And the reason he wants to put something on it is because one of his production systems has just died. And now this system will take over the work. And still, so far so good. Now for some back story. I made the CentOS system, and I deployed it. And he gave me the network settings for the system, but we are a bunch of retards so we don't have access to the backbone layer 2 switch in the data-center. So we put the system in a different network and later when the administrator of the network was available we moved it in the right network. But the client seems to have missed that so he was trying to access the wrong system. So he calls me asking for the password, and we both don't know that he actually needs the IP. So I tell him the password and no dice. So I open the system he is trying to access and find that the password really doesn't work. And it takes some time to realize that this is the wrong system. So I finally open the docs and see that the system has been moved to the network where it should be. I also notice that this system has an IPMI IP. Bravo. So we are in the system, and as the champagne is poured into glasses and cigars are passed around the client calls again. His sysadmin has changed the SSH port remotely and they have lost connection to the machine. "Did you set the new port in the firewall?" I asked. To which he replied that they don't put firewalls on their systems. If you remember earlier I mentioned that I installed the system, and I do put firewalls on systems. Every system I make gets a unique SSH port, just to keep the scripted attacks guessing. They decided that a port that is unofficially associated with SSH would be better for SSH (tell me in the comments what port I am talking about). On top of that the office is 20 min away from the data-center. And the client asked if I could go to the center and fix the issue, like an idiot I said yes. I was half way there when I remembered, IPMI. I called (while driving) and the moment I entered the data-center he called back to say everything is OK.

So what was the stupid thing. When a client calls don't panic. Tell him to wait, check for yourself what exactly is going on. You are not helping by wasting time on the phone with him, your kind words are not as valuable as a quick solution. I didn't have to do anything except to send him the docs on the system. Wasting 30 min. for something that should have taken the client 5 min. to resolve on his own.

Do you know what IPMI is? You can read the wikipedia article or you can hear it from me. Imagine a network adapter that allows you to remotely open a system, reboot it and change BIOS settings. Then you can mount an ISO image and install CentOS or whatever you want over the network. And imagine if you are dumb enough to change the SSH port on a system and not to open that port in the firewall. You can use IPMI to open the system and as if you are standing in front of it, you can recover from the mess you have made. IPMI is awesome.