A reseller's team has just installed a new mainframe for a client, and the senior systems tech is justly proud that the job is almost done, says a pilot fish there. "He coordinated the whole thing from system install to software install to client training," fish says. "Everything was all hooked up, the software was installed and configured, the devices were all connected and the client was ready to go. "The system had been up and under heavy use for two days when the senior tech gathered the client's technical staff together to review some details." Once everyone is assembled, senior tech begins going on about how mainframes differ from desktop computers. He stresses that proper shutdown procedures are a must. With a final flourish, he says, "You must never treat it like a PC and do this" — at which point he pushes the mainframe power button in. "He was made to hold the button in while everyone else called all the remote locations to have them log all the users off the system for 'maintenance training,'" fish reports. "About 45 minutes later, all the users were off the system and the power button released. As the system shut down, we could only marvel at the purple colour of the senior tech's finger."
Back-ups never fail
It's the middle of a February in the mid-1990s when the server that contains all the year-end processing data and reports for a company goes down, says a pilot fish on the scene. "The techs were unable to get the server back up, so it was decided to reformat the drives and restore from backup," fish says. "One tech hurried off to start the reformatting while another looked through the backup log for the appropriate tape. "After a few minutes, the tech looking for the tape got a bad feeling." And with good reason: The last good backup he can find for this machine is dated December 31 — six weeks earlier. The reformatting is already under way, so there's no going back. A month and a half of data are gone. But how? It doesn't take long to figure out. Everyone in the IT shop knows that all the backups for this datacentre are done centrally to a single server, which also runs the backups for some systems outside the datacentre. "As some of the outside systems might not be running when the backup ran, the operators were told — and it was documented in the procedure manual — to ignore the 'backup failed' message, as it supposedly was for the systems not turned on," says fish.