As job descriptions and personnel changes a lot due to a ongoing company merger, I was less and less involved in desktop support, and temporarily taking care of the servers I was already supporting until I migrate them to the new centralized servers they're trying to set up. As I had more free time (and could hardly charge all that "Documentation" time on company-time), I got assigned to projects, which brought me to a remote-city meeting for the whole day. That was in July, and I managed to escape the meeting early (I was replacing someone else, they didn't need my input anymore, I had to get the train back home, you know?). Wonderful afternoon writing lyrics in the train ride home. Since the train station is so near to the office, and since it was still early, about four, I decided to make a quick stop just to check my e-mail and leap-off. As one of my colleagues used to say "Do like Gordie Howe, and get the puck outta here!" Well I got there not a minute too soon. I have a desktop (only) support guy from the remote office who left me a voicemail message about disappearing files on one of their servers. He also mentions that this could be related to a new virus that appeared earlier in the morning, Worm.Explore.Zip. I didn't touch a computer all day, so I didn't get the news, but that snapped me out of my clouded lyrics from the train ride. This is a nasty. It replicates itself by replying to unread messages found in Exchange, Outlook Express and Outlook. That's 1-0 for me on the nasty. I don't know yet how many people are infected, but I know they won't cross-contaminate each other, that's a relief. What else does it says... Copies itself in predictable places, that's 2-0 for me, because with this information, I figure I will be able to quickly craft a searching batch file to scan for it and remove the infected file (deletion was the cure). I would redirect the output of a "if exist" file check conditional delete to a log file, along with username and machine name, and have it run from the login script. What else does it do? Delete .doc, .xls, .ppt, .c, .cpp, .prg and some others like that, sneaking everywhere to find them, and attacking on brute-mode. Actually, it doesn't *erase* the files, rather it puts them to 0 size. NASTY! He definitely scored a goal on that one, 2-1 for me. Well, it's past 4:30 now, at least most systems will be down by an hour, this buy me time. OK, time to call the support guy back.
John was anxiously waiting for my call, apparently overwhelmed by the situation. I try to calm him down and try to see if he could give me any help since he's on-site, and at least he's not a user, right? All he knows is that he received that trouble ticket call by a guy who complained about a bunch of files in the departmental drive *ouch! the biggest repository of documents in our network environment*. And that's all he could tell me! OK kid (and I was only 24 at the time!), stand by and watch a pro in action. First, I open Windows Explorer and connect to the departmental shared drive. This drive is a mess. It was set up as a mean to keep the file servers to be used only by local users to reduce traffic, but still have a mean to exchange documents between offices. This was at the dawn of corporate internet mail, when only few privileged employees had it at the time, and the main corporate messaging system was through a mainframe system that we connected to with a TN3270 client. Not really called for file exchange. So this drive is the place where everyone in city A places a document a person in city B needs to read, and vice-versa. It is free for all. When there's no room, clean up. Don't put your single copy of a document here (but you can't rely on users for that one). So I initiate a Find files request with search criteria *.*, 0 bytes in size. WHOOAAAA!!! Over 200 files showing, and a few refreshes of the view shows me that it keeps going up. 2-2. Nasty!
As I keep doing this, I am still on the phone with John, who is now in the server room. I tell him exactly what I'm doing, step-by-step, live to the minute, and I suggest him to do the same from the console. OK, so anybody who knows a little bit in networked operating systems and virus infections or computer security knows that for a malicious program to deliver effectively its payload, it relies on the access rights of the infected username. Crunching a file to 0 bytes in size leaves a trace, NT can tell me who owns files and who has made last modifications to it. I check a few files and this gives me one username that seems to be deleting the files. There's still more and more getting deleted as we speak over the phone (I told you there were a *lot* of files in that drive). As we make the investigation, we discuss about if this could have been due to the server being infected. That was rejected for obvious reasons:1) the file servers don't run any mail package, so could not receive the virus and execute it, 2)the fact that files on the server were being deleted was no indicator that the deletion process was executed on the server, 3)the username and the date of last change gave us an excellent view on the virus origin and progression. It was amazing to sort them by time to realize how steadily the files were going away. It is sad that I did not keep a snapshot of that to include it here. You could clearly see it plunder files in a stubborn and quite effective alphabetic fashion, putting in our imagination the image of a small furry electronic beast, with only one obsessive thought totally occupying his small but determined mind:
Do {
Crunch (*file)
file++
} While (Files_exists()==1)
"OK", I told John, "go to that guy's office and tell him to shut down is computer NOW! Disconnect it from the network and put it aside for further inspection." In the meanwhile, I know enough, now all I want is to stop the murder. I close all connections to the server and remove all share access to the attacked folder. Quick Find files request on other important shares (\\servername\users\*; \\servername\groupdata\*) for 0 bytes files. That doesn't bring up much information, gladly, because that means no damage there *yet*, so I keep these services live, but I keep a close eye on it. I do see two 0 bytes files in the same guys user directory, though, I'm pretty sure I'm on the right track. So tell me, Worm.Explore.Zip, can you feel my breath on your neck? I quickly craft a batch file that I launch from my faithful login scripts. In the batch, in put in a quick check for my previous idea, and I add some DIR to search local drives for 0 bytes sized files. I redirect all output to a writable file on a server that I keep around for testing (so if something bad happens to the machine, it's no big deal). In corporate environments, these machines are sometimes hard to get. Obtain them from when servers are migrated and upgraded, when older machines are released. When such machines get free, TRY to get your hand on it. Make up false requisition, invent fake project, or if it happens that one of your current server gets upgraded, it is even easier to keep the old machine also by invoking some "legacy-application" that can only be run from this machine, whatever. You're not stealing them, you're simply trying to do your job. You just have to play their game with their own rules if you want to stand a chance.
OK, batch file ready, 4 phone calls to 4 wonderful ladies that have the ability to reach everyone with the contact of one single touch on a key on the phone pad. The word spreads fast: Logout now! Log back in if you really intend to... It's nearly 5 PM now, and I imagine that I won't get to see too many outputs to test the efficiency of the batch file. Well, it appears that some people are zealous tonight, because the results start coming in already. So far, so good, all signs of further file destruction are gone, and I am pretty sure by now that I've singled out the culprit. 3-2 for me after a nice breakaway goal! A quick look for last minute .dat files... not yet. Mental note to check first thing in the morning. And last thing before quitting tonight. I would have wished they all went home right now, but I think I have enough alarms to warn me quick enough if another breakout happens. Let's go back to the departmental drive to see if I can't squeeze a bit more info from it. This is where the virus has done the most damage, this is where I can learn most about it. What??? There's even more files getting deleted? But I thought I cut access from it??? IT'S BACK??? HOW THE HELL DID THAT SHARE RE-SHARED ITSELF ALL ALONE LIKE THAT??? #$%#!^%!^&*^&($*^&@$^!!! Trusting my instinct, I call John to see where he's getting at with our infected friend. He tells me that the guy is not at his office, and knowing he has a laptop, he could be connected anywhere. No way to reach him (he didn't respond to my URGENT mail). Things are not looking good: Nasty's got a powerplay with a few minutes left remaining on the clock. I then ask John if he would know anything about the departmental share being back on. "Well, yeah!" he says, cheerfully. "I had received a trouble ticket call from a user saying he couldn't access the shared drive, so I investigated the problem and I found that the share service was removed. So I put it back in place."
-YOU DID WHAT!???!! YOU PUT BACK THE FILE SERVICE THAT YOU KNOW IS BEING SEVERELY ATTACKED RIGHT NOW AND IT DIDN'T COME TO YOUR INVESTIGATIVE WANDERINGS THAT IT WAS I WHO MIGHT HAVE DISCONNECTED THE DRIVE? YOU MORON! $$^*(%^($^(&**&!@^!!!!
-But you should have told me you did that! (Remember about the obvious: Use your head? Here's the perfect example).
(Dimwit)
-OK, I am going to cut access again, John, please don't put it back until I say so. Better yet, let me put it back when I decide it so. If people ask you about it, explain that we're under severe virus attack and that the drive has been disconnected until further notice for protective reasons. Some people, you have to tell them everything.
-OK, he says. Oh, by the way, did you know the backups aren't running on these machines?
!!!!!!!!!!!
-WHAT!!! TELL ME I HEARD WRONG!!! PLEASE TELL ME I HEARD WRONG!!! WHAT ARE YOU GUYS DOING UP THERE YOU BUNCH OF $#$^&% MORONS??? STOP PLAYING WITH YOUR $%^@$& A$$E$ AND TRY TO BE USEFUL FOR A CHANGE!!!!!&%*&%*$^!@^%!^&@!!!!
-Hey, don't blame me, I didn't even know there were servers in that closet before I received that ticket this afternoon.
Looks like some information went down the drain during the exchange of responsibilities between staff members. After I calm down, I get a better picture of the real situation. The normal file server have backup running, but nobody changed the tape since who-knows-when, so all I got on this one is a 24 hours old backup. Better make sure this doesn't get overwritten. The departmental server, on the other hand, has its backup software crucified on the screen console: General Protection Fault, since who-knows-when for that one too. Kick in the groin goal! 3-3, Nasty evens the score. It will be impossible to recover any erased files, hundreds of them. This story is true to the bone. The worst is this shouldn't have happened. Backup tapes are the first daily chore of server administration, and it's last resort solution if the absolute worst happens. Well, bad things happened, and this is one more proof that the chain can be only as strong as its weakest link. I just thank the Lord this didn't happened at 8 in the morning, when I was on the train on my way to meeting.
Epilogue: Overtime
I spent the next few days closely monitoring my output files to see if the batch files would catch anything new. I get the newest .dat files as soon as they're available. I ask people to reboot this once to activate as many logins as soon as possible, in order to have something a bit more reliable that batch-files for protection. Having been a witness of another IT fiasco in the same department, I was expecting some directors to go out for blood on this one. Surprisingly enough, I had ZERO complaints about the lost files. I will never be sure of why, but I will put to the credit that they followed our advice and not put the single copy of an important document on that drive. So the games end in a tie, it was a fierce matchup. It could have been worse. I could have been beaten badly by Worm.Explore.Zip. Many others faced many days of reconstructing and ensuring sites. Let's just say that I got a bit lucky, but like every good goalie, I made my luck.
8. Strategies to adopt
10. The brown stuff
Table of contents