Over the past few weeks, I've had arguments with friends in the information security echo chamber about whether it was prudent of me to make public comments about the security the beleaguered Healthcare.gov website when I had not actually performed a formal assessment of it. My answer -- that I'd assessed all I needed to reach my conclusions -- failed to satisfy some.
Some of this disagreement stemmed from the fact that I was speaking of strong indicators, not evidence, of trouble. Did I go too far in my conclusions? Time will tell.
Could I have been less abrasive about how I stated my conclusions? Always.
But the experience raised a much larger question: I was assuming that incident assessment was more universally understood than it is. My statements therefore seemed black-box and arbitrary, and that's never my intent.
Lest I re-kindle the debate, let me move completely away from Healthcare.gov and into general incident response and how I walk into a place where I've never been and help figure out (a) if there is a problem; (b) if there is, how big it is, and (c) where do we start to fix the most broken stuff the fastest.
In the past, when an organization asked me to help them understand a compromise, I started by asking questions designed to disqualify avenues of investigation. I often start with two big ones. Of course, these are not the only questions I ask, but they are among the first ones, because the wrong response is so strongly correlated with other horrible practices and habits of security guttersnipes:
Can we take a look at your network logs, flow records or analysis and/or traffic capture for the last few days?
Can I see your latest network scan results?
The answers to these questions are often telling. In many cases, I'm far less interested in what the logs and captures say than the fact that they exist in a form that is accessible by someone in a reasonable amount of time, and that they can in fact be accessed and presented in a form that allows them to be analyzed.
You would be shocked at just how many organizations fail completely at this request.
The "latest scan results" is asking the all-telling question, "Do you know, within a 25% margin of guesstimation, how many endpoints are connected to your network?"
If the company fails these two basic questions, I am certain to find a major mess. Those things didn't cause the mess, but their presence indicates an inattention to security detail, lack of procedural integrity, failure to backstop and a directionality of spending that historically have shown to be conditions present in an organization that has been compromised.
Do I know it, factually? Of course not. But I know it enough to bet the client a case of Dublin Dr. Pepper that I'll find it out right quick.
Many incident responders and penetration testers have similar big-picture disqualification questions they ask. Dave Kennedy, CEO of TrustedSec, goes one level deeper into the logs on his first question: "Have you got logs of DNS requests from the past week or two?"
That's a question I usually ask too, once we've got other evidence that something is up. Once we know that there's something on the network, the DNS logs are a great first place to look for where it's calling to (and, inferentially or specifically, then, what it is and who put it there).
There are a whole bunch of other questions you can ask once you start digging, and most of them are designed to disqualify a line of questioning to avoid getting into it. Eric Olson, VP at Cyveillance, tells the story of how his firm asks a single question to determine whether an email was a possible banking phish: "Does the email contain the FDIC symbol?"
Now, not all email with the FDIC symbol is a banking phish, but pretty much no banking phish doesn't have an FDIC symbol. With one question, then, Cyveillance is able to disqualify 97% of the email it sifts through on a daily basis, allowing them to concentrate on the three per cent that may be bank phishing.
Here's another example: on a recent incident we asked about a certain flow analysis product. The first question was highly disqualifying: "You're not using [product] now, are you?"
"Oh, yes I am," came the response, which was in itself highly interesting.
"Oh," we said. "So how often are you using it? Weekly? Monthly?"
"Oh, every day," came the reply. "We love it."
"Oh, I see, that's great," we said. "So when you used it today, did everything look okay?"
"Yeah, you know, things looked pretty normal."
I bet they did. Earlier that morning, I had personally seen the box in question sitting, unplugged, on a cardboard box in the data center.
An engineer told me it had been unplugged for more than a month.
In this case, it was clear I didn't need to do any further digging to understand that no analysis had been done. It was also clear that, if the person would cheerfully lie about that, he would cheerfully lie about other things.
It was therefore no big intuitive leap for me to conclude that previous things the guy had signed off on had likely also not been done. Did I know this factually to be true? Not yet.
But I had a wicked-strong indicator. Later we confirmed it, but the indicator itself was strong enough for me to base certain statements and actions.
This stuff is not rocket science. By asking high-level, disqualifying questions, one can easily make some broad assessments. Then it's simply a matter of drilling down into the indicated areas, and finding the problems.
Kennedy offers a few more questions, like, "Are you testing for security threats and doing things like external/internal penetration tests and social-engineering efforts to test your controls and your incident response?" and, "Have you ever done a source code analysis or dynamic testing of applications to determine what risks they pose?"
A number of us are working together to make a list of the top ten questions that can be asked in any organization. We feel that this is a great way to share knowledge and experience with the community. Like that idea? Drop me a line, using the contact form at nickselby.com.
Nick Selby is CEO and co-founder of StreetCred Software. A veteran information security consultant, Nick has provided incident response services for F500 companies.