- I was happily dreaming about a well-secured network when the beeping of my cellphone woke me. It was the office. I'd been away from work for a few days and was making an effort not to read my email or check on the security team. The call gave me an excuse to find out what had been going on.
The security staffers were calling me to see if they had responded appropriately to an incident. We have a script that regularly analyses the firewall logs to identify any changes in behaviour, which we cross-check with our intrusion-detection system. I normally review the script results myself, but while I was away, the staff took over.
The latest report was full of blocked connection attempts. We expect many probes to come from the internet, but these were a little worrisome because they came from inside the firewall.
The security staffers were thrilled to think that they had uncovered a new worm. They analysed the behaviour and could see many machines trying to connect to the same handful of external machines; a few internal machines also appeared to be conducting sequential outbound port scans.
This behaviour suggested that many machines were infected with a stub virus that was going out to a central location to get updated virus code and download the rest of the worm. Then the virus would scan to find new hosts.
Our protections were working to a certain extent: The firewall was blocking most of the infected machines from downloading further code, and the virus was retrying over and over again. A few copies had apparently managed to get the full package and had begun to search for other machines to infect.
The staffers called me because they had been unable to collect a sample of the infected code to send to the antivirus teams. They hadn't uncovered any applications that shouldn't be on the attacking machines, nor could they determine what was opening so many network connections.
I reminded them of our firm's virus-handling instructions: Had they carried out all the steps for a suspected virus outbreak? We segregate the network to halt the spread internally by splitting off critical servers, delay all file attachments in the email system in case they're being used to spread the virus and notify key contacts in our business units of the attachment delays. They had indeed carried out all of these steps.
We discussed using incident-handling tools to help find the subverted application. I also asked them to track down the systems from which the virus was trying to download the code. If we could shut down those systems, not only would we protect ourselves in the event that the code found a way around our firewall, but we might also help protect other companies. I suggested that the staffers contact our key security product vendors and check www.incidents.org to see what other people were saying.
From Bad to Worse
The deeper the security workers dug, the worse the situation looked. They found different applications initiating the connections from each desktop, and the checksums (values used to check file integrity) on the executable files matched those on fresh installs from the original installation CDs. Did this mean the original media was infected, and some kind of time bomb had been initiated? Perhaps the virus was clever enough to patch the kernel so the checksum software returned bogus values? We worried about how dangerous this issue could be. How do you clean up an infection when you can't tell the infected from the clean applications?
No other company was experiencing the same problems. The staffers did have a couple of names, however: AvantGo and RealNetworks.
I paused and asked them to repeat what they had just said.
Now I knew why the application checksums matched. The applications hadn't been tampered with: This was designed behaviour. RealPlayer downloads updates and playlists automatically, and AvantGo retrieves new items to be loaded onto handheld computers. These are two examples of the many applications that "phone home" without asking the users, or at least without the users understanding what they are accepting on the default install.
Until recently, we had forced all browsers to use the proxy settings but had allowed some applications to bypass the proxy to download updates. After the Nimda attacks, we changed that and blocked all nonproxy access.
This explained the machines that were trying to connect to outside servers. But the port scanning remained a mystery. I asked the security staffers to repeat the list of anomalies they were seeing. They mentioned that the sequential scans weren't to different ports, as I had assumed. Instead, the number of the source port was rising incrementally, but the destination port was always Port 80.
Again, this is normal behaviour for some badly written software. Our firewall returned a warning page explaining that users needed to use the proxy settings. When these applications received this warning page, they knew they had received something, but it wasn't the pages requested, so they kept retrying. The web server operating system was supplying a new source port with each retry, and the steadily incrementing value in the firewall log had caught the eye of the reviewer as the classic symbol of a port scan.
By the time the situation was resolved, it was late in the evening. The security team had learned a valuable lesson, and it has a strong incentive to not get carried away with an incident the next time.