Sony's apparent difficulty in figuring out the extent of the damage from the recent intrusion into its PlayStation Network, while frustrating for those affected by it, is not too surprising given the bag of tricks that hackers employ to hide their tracks.
All too often companies simply don't have the forensic tools or enough log data to be able to reliably piece together what might have happened and to determine the true scope of a breach. Sometimes, it can take weeks and months to get an accurate picture, and even longer for a breached entity to entirely clean out its networks.
Sony itself has offered no reason why it waited more than six days to inform consumers that their account information including name, address, birth date, purchase history, online ID and possibly credit card data, had been compromised.
And it has said nothing about why its taking so long to restore the network. In all, a staggering 77 million consumer records , including that of many minors, were potentially exposed making it one of the largest data breaches ever.
It's possible that Sony's initial silence was prompted by PR worries, a law enforcement request or both. It's also possible that the company did not have the data it needed to quickly determine the true scope the problem, IT managers and security analysts said.
That's because often the security tools that companies deploy are oriented towards discouraging and preventing data breaches, said Matt Kesner, chief technology officer at the law firm Fenwick & West. "Most haven't focused on instruments that would create a great record if you were hacked or breached," he said.
While companies probably look at log data from their firewalls and other security devices, "it's very difficult to build a trail" without more data, he said.
"A lot of organizations keep and monitor logs from security devices [such as] firewall, anti virus , intrusion detection," said Johannes Ullrich, chief technology officer at the SANS Internet Storm Center. "But they fail to create and collect application logs, in particular from custom applications, with the same rigor."
Custom applications, especially Web applications, are a huge target for malicious attackers, Ullrich said. Yet because companies don't often collect and maintain these logs, "a lot of intrusions are not detected and the damage can not be quantified." he said.
Another problem especially in large companies is that old log data often gets overwritten with fresh logs by the time an intrusion is detected, said Alex Cox, principal research analyst at Netwitness, a security vendor that was recently acquired by EMC .
Though it is relatively inexpensive for companies to store multiple years worth of raw log data if they want to, many don't. As a result, log data that might have revealed critical data of a break-in might get overwritten by fresh data over a period of time.
"If you are lucky you can get to a point where you find some piece of information you need to put the puzzle together and sometimes you don't find it," Cox said.
In addition to log data, companies also need to have the right host-based and network-based forensics tools to be able quickly sift through and correlate event data to figure out what might have happened.
Commercial and open source tools are available currently that allow companies to do full packet capture of all traffic on a network for future analysis purposes.
Other technologies from companies such as Netwitness and Solera Networks allow companies to record and store every single network event, and then to replay them back in DVR-like fashion if needed.
Although such tools can give companies invaluable insight into security incidents such as the one that hit Sony this week, they are relatively expensive and only now beginning to get deployed in significant numbers.
A lot of times, companies also gbecome stymied in their investigation of a breach because of the initial manner in which they react to its discovery, said David Amsler, president and CIO of Foreground Security.
It's not unusual for enterprises that discover a breach, to get into a panic and start immediately shutting systems down and unplugging them from the Internet. One example is Oak Ridge National Laboratory, which quickly shut down its email systems and disconnected itself from the Internet after discovering intruders in its network earlier this month.
Such measures can be critical to preventing data theft, but they can also make it harder to determine what happened, Amsler said. Oak Ridge for instance, is still without Internet access nearly two weeks after it pulled the plug on it.
Often, the ones behind such intrusions have already established a presence deep inside the network by the time their intrusion is discovered. Any actions that show their intrusion has been detected typically causes the attackers to launch measures for erasing their tracks, wiping logs clean, altering time stamps and for going even deeper into hiding, he said.
"Many times victims don't even know what data was breached because the artifacts from the breach are encrypted and password protected," by the time the intrusion is detected, said Marcus Carey, community manager at security vendor Rapid7.
In many cases, attackers take control of multiple systems and multiple accounts once they get into a network. They can drop multiple malware packets, each carrying a different payload. They also often disguise themselves to appear as legitimate users on the network and often delete log files or put in fake logs to throw administrators off their trail.
"If you suddenly take a subset of host systems offline they are just going to switch their MO midstream," Carey said. "They will change their attack vector. They will drop multiple different tool kits. They'll even throw stuff out there that they'll want you to find so you think you have found them."
"It's no surprise at all that some of these big companies are taking weeks to find out what's going on," he said.
Jaikumar Vijayan covers data security and privacy issues, financial services security and e-voting for Computerworld. Follow Jaikumar on Twitter at @jaivijayan or subscribe to Jaikumar's RSS feed . His e-mail address is firstname.lastname@example.org .
Read more about data security in Computerworld's Data Security Topic Center.