Mornin’ kids. Any of you wondering what I’ve been up to over the past few days? Yes? Good.

Well, eC’s server was broken into again. Leif figured it out when he ssh’d in to see why it had been so slow over the past few days and most of the commands segfaulted. After that, he initiated operation back-everything-up-within-the-next-four-minutes.

One of the first things extracted from the server was /var/log, which I got a copy of. When I went to examine the logs, I found that anything recent had been trunctuated. Useless.

Or so I thought. On a closer examination, very valuable information was found in /var/log/apache/error_log. Enough information to allow me to track the compromise down to one man: br0k3d of #nova@irc.rizon.org.

He was involved with some Brazilian warez d00d group, and they needed servers for their warez chan. He was nice enough to tell me that his modified version of the Anti Sany Worm was used to get into the server via phpBB, and then the server joined his IRC room and was scanned for vulns. Since it was running RedHat 7.3 with the shittiest repos in the word, vulns were numerous.

After that, shv4, a rather nasty rootkit, was installed. This thing replaced a number of binaries and libraries. It also managed to damage the system beyond reasonable repair. (Damnit, when you write a rookit, CAN YOU PEOPLE HAVE IT CHECK TO MAKE SURE IT’S COMPATIBLE WITH THE OS? Honestly, IT HOSED THE SYSTEM! What the HELL is the point of keeping root on a useless box?)

Even though RH7.3 was hopelessly out of date, hopelessly damaged, and just generally stupid, I tried to repair it anyway. I sort of half-fixed it. Httpd could start, so meh.

That, however, was only a temporary solution. Having a server that broken running several websites is like asking to be poked in the eye with a dagger.

We had two options: Require an OS restore for $30 bucks and get RH7.3 back (The best you can get is RH9 for $100, but that’s EOL’d, so it’s still pretty useless), or switch hosts. A host with decent prices is a rare find, so that wasn’t looking like much of an option.

We were going to go for the restore. But I had an idea: Installing Debian via SSH. SInce there was a good chance that it would fail, we could just proceed as planned and have the OS reinstalled. But if it worked, $30 could be saved, and eC would have decent packages on apt.

And so began a two-day Debian installation. Oh boy, was it weird. The guide we went by can be found here.

Two reboots were required, and both failed. All I can say is kudos to EV1’s DataCenter team. Despite their lack of any proper English and spelling whatsoever, without their stupidity, this installation could have never happened.

You see, RH7.3 had configuration files for lilo and grub. I thought the bootloader it used was grub (grub is more noticable. Sue me.), so I had reconfigured grub instead of lilo. When we rebooted, it tried to go into RedHat 7.3, and it choked to death.

EV1 made an offer: A free four-hour keyboard-mouse-video setup through a java-enabled browser. It would be like sitting in front of the server. At this point, I went to sleep, thinking all hope was lost. Leif realized that the server was using lilo and fixed it.

In the morning I pinged the server, ssh’d in, and my jaw dropepd in shock. Debian was running. I removed some packages we didn’t need, and hurried off to school.

When I got home, Leif explained the magic to me. He also told me it was loading the wrong ethernet module, and that he had fixed the module configuration to load it on boot.

So, I finished the install and rebooted. It didn’t come back up. Tickets were sent to EV1, and another four hours of KVM useage was aquired. The config file Leif had fixed for loading the ethernet module had been regenerated by something, and a proper fix was put in place. The server was rebooted once more into Debian.

And here we are now. Apt-get dist-upgrade is running to get us to testing, because stable packages are hopelessly out-of-date. Testing will be moved to stable within the next two months anyway. Again, sue me.

Yup. Back to work.