A speedy recovery

Screwups happen. No matter how carefully you set up security, no matter how sensitive you are to customer privacy, a few simple mistakes can blow it all to hell.

Screwups happen. No matter how carefully you set up security, no matter how sensitive you are to customer privacy, a few simple mistakes can blow it all to hell.

That’s what happened three weeks ago to Kaiser Permanente, the giant HMO based in Oakland, California. By normal business standards, Kaiser is close to fanatical about patient privacy. Security is tight: Even if you’re a Kaiser doctor, you can’t get access to a patient’s records unless you’re that patient’s doctor.

But on August 2, a few technical problems and human mistakes in Kaiser’s IT shop pumped information on hundreds of patients to the wrong people.

That’s the bad news. The good news is that the Kaiser incident is a blueprint for the right way to handle a big privacy foul-up.

(Full disclosure: I’m not a Kaiser patient, and never have been. Computerworld US does offer Kaiser as a health care option to some of its employees, though.)

What happened? Kaiser’s Web site was upgrading servers late last month, which required a patch. That patch caused patient email to back up. Meanwhile, an IT staffer wrote a script to send email to 19 addresses. Something went wrong, and various quantities of 858 backed-up messages were sent to each of 17 patients (two of the 19 addresses weren’t valid).

The error was caught and the script stopped within 20 minutes. But hundreds of messages, some containing confidential medical information, had blown patient privacy right out of the water.

That’s what went wrong. What did Kaiser do right?

First, people in IT were paying attention: An IT staffer noticed the script was pumping out much more message data than it was supposed to, and the staffer shut down the process.

Then somebody took it personally: Anna-Lisa Silvestre, who runs Kaiser Permanente Online. There was no fight over who would clean up the mess — she just plowed into it.

There was support from the top: Kaiser CEO Dave Lawrence didn’t duck the problem and even offered to stand up at a press conference if necessary. That kind of backing makes things happen.

IT staff pitched in: Almost 900 patients had to be called. Regulators in 11 states had to be notified. IT staffers worked around the clock to deal with the results of a few minutes’ worth of mistake.

There was nothing fancy in the effort, no elegant technical hack. Human error created the problem, and a lot of hard human work was required to fix it.

The bottom line? The screwup was monumental, but it looks like it wasn’t catastrophic. It was contained because everyone jumped on the problem fast. No denial, no finger-pointing paralysis — IT people took responsibility for it, got management’s support and sweated the hard work to get it done, no matter what.

Most important, the patients seem to understand — the ones whose privacy was violated and those whose mailboxes were flooded with other people’s messages. Each one got a human voice explaining the problem. That alone went a long way to rebuild trust.

And without trust, Kaiser — or any other e-organisation — is out of business.

The “root cause evaluation,” in Kaiser’s doctor-speak, will take weeks. It will try to determine exactly what happened and how to avoid this kind of problem in the future.

But the real question isn’t what will happen next time at Kaiser Permanente. We already know they can handle it.

The real question is: What happens when this kind of nightmare strikes somewhere else?

Because screwups happen. And if they can happen to Kaiser, they can happen to you.

Hayes, Computerworld US's staff columnist, has covered IT for more than 20 years. His email address is frank_hayes@computerworld.com

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments