j0434750 Around two weeks ago, I found this email in my inbox, with the subject “Complaint about Robozzle”:

Hi Igor

Robozzle is really cute, I like it, but why on earth is it polluted with hundreds of invisible links to porn sites? From a guy like you I don’t expect to do such dirty things. pls remove them.

David

Hoping for a simple explanation, I looked at the source of the RoboZZle front page. And my heart sunk as I saw hundreds of spam links on the bottom:

<a href=”<a hijacked site>.com/files/cms/7/pornhud.com-20076.html”>porn hud.com</a>
<a href=”<a hijacked site>.com/files/cms/7/www.tube.com8-5852.html”>www.tube.com 8</a>

The links were inside a <p> tag with visibility set to “hidden”, so the links were only visible to search engines, not to human visitors.

I wondered if this affected how my site shows up in search engines. So, I searched for RoboZZle on Google (results in Bing or Yahoo! were not affected), and here is what I got:

robozzle_bad

Oh crap. This sucks. How did it happen?

Site infestation

j0438025

So what did the hackers do to my poor game? I looked around to find what has changed:

  1. Planted links
    Multiple pages on my site have been modified to import a planted config.ini file. The config.ini file contained hundreds of fake links, and was updated every couple hours with a new set of links. The links all pointed to spammy content planted on another hijacked site.
     
  2. Planted content
    My site also contained thousands of small planted HTML files, similar to the spammy stuff that the planted links pointed to. An odd folder (something like http://robozzle.com/old/old2/) contained the HTML files, all with links to suspicious content, and all infested with spammy keywords.
     
  3. Backdoor PHP scripts
    And finally, my site also contained a couple of PHP backdoor scripts. If you visited a particular URL on the robozzle.com domain, you’d get a file manager that lets you upload, delete, and manage files on my site, no passwords needed. As I found out later, the hackers actually knew my FTP password so they didn’t need the backdoors, but they left the backdoors so that they can get in once I change my password.

I carefully checked my site and removed all of this stuff. Even after I removed the PHP backdoors, the config.ini file continued to get updated until I changed my FTP password. From that point, I was pretty sure that the attackers somehow got my FTP password.

Looking for other hijacked sites

After cleaning up my site, one of the first things I did was notify the other compromised site, which hosted the planted content linked from my site.

I also wondered if more sites have been hacked in a similar way. I tried various online tools to find other sites that contain the same links that have been planted on my site.

I found a handful of hacked sites right away. The fact that the links change every few hours made the task more difficult, though. The newest links generally point to content that has not yet been crawled by search engines. But, there is a simple solution. By looking at older versions of several compromised sites in a service like Google Cache, I eventually found older links that lead me to many more hijacked sites.

After repeating the process for a couple hours, I ended up with a list of over a 150 infected sites, including some fairly major sites. The larger sites generally removed the infestation quickly, though. Here are a few sites that haven’t cleaned up, despite the fact that I alerted them at least two weeks ago:

  • bayonnenj.org – City of Bayonne, NJ
  • steinercollege.edu – Rudolf Steiner College
  • dillard.senategop.org – GOP Senator Kirk Dillard
  • egnc-ibm.gov.eg – Egypt-IBM Nanotechnology Research Center

Don’t go to these sites unless you know what you are doing. At the time of writing this post, these sites appear to be compromised, so they may well contain viruses or malware.

UPDATE: Most of the sites are suddenly not showing the links. My guess is that the hacker group distributed an empty config.ini file after this story became popular on reddit. I don’t believe that so many obscure sites on my list would be fixed over night when they have been infected for months before. Some sites still contain the planted links, but those seem to be the ones that haven’t been updating the links regularly. These are probably the sites that the hackers only have partial control over at this point (e.g., FTP password changed, but there is still a backdoor on the site somewhere). You should be able to view the planted links on all infected sites by looking at Google Cache.

I did my best to contact owners of as many hijacked sites as I could. Looking for contact information on that many sites – most of them not in English – is a time-consuming endeavor, though.

Tools I used

I found these tools useful when tracking down other sites that have been hijacked:

Yahoo! Site Explorer This is the best tool I found to search for sites that link to a particular URL. At least for my purposes, it worked much better than “link:” queries in Google.
 
Proxify When visiting sites controlled by hackers, it is worthwhile to be cautious, since the site may be infested by malware. Proxify sends the request on your behalf, and in the Source mode, sends you the HTTP response as text. So, the infested site won’t see your IP address, and any HTML it sends back will not be rendered, just shown in text format.
 
Web caches As you probably know, all major search engines (Google, Bing, Yahoo!) let you view the version of the page cached by the service. This gives you a version of the page as it was a few days or weeks ago.
 

 

And how did I get hacked in the first place?

Of course, I have been wondering about how the hackers got my FTP password in the first place. It wasn’t really guessable or discoverable by brute force, and I didn’t use the same password on other sites.

And then I got an email from my webhost, notifying me that FTP passwords may have been stolen due to a vulnerability. They tracked down the issue to a particular software package they use.

So, I assume that my password got stolen this way. If not, it is also possible that I logged into my site on a computer with a particular virus. Apparently, that’s how many FTP passwords get stolen.

If you liked this article, check out these ones:

Human heart is a Turing machine, research on XBox 360 shows. Wait, what?

Numbers that cannot be computed

18 Comments to “RoboZZle hacked, and 100+ sites are still compromised”

  1. Thomas says:

    The hackers might as well be on the same shared hosting package (assuming you’re using shared hosting). In that case, they don’t even need your FTP password to hack your site…

  2. […] Has the city Site been Hacked? On 10.29.09, In Current Events, From the Web, By BCAL RoboZZle hacked, and 100+ sites are still compromised. […]

  3. anon says:

    I hope you are using some sort of secure FTP like sftp otherwise your password is being sent in the clear over the wire; easily taken.

  4. rg says:

    Use ssh or at least sftp or your password will be easily found….

  5. Tim says:

    I’ve seen backdoor PHP scripts a handful of times. TL;DR, Remote file inclusion or unprotected file uploading.

    The two times I’ve seen them, its been like this:

    index.php?loadfile=woot.php

    index.php has include($_GET[‘loadfile’])

    OR

    You have a file upload somewhere and put the incoming files into an executable directory.

  6. Kg says:

    I work in a web dev studio, and one of my staff alerted me that one of her FTP site has spammy links. Double check that no rootkit or holes.

    Turn out she got rooted on her Windows machine, (very likely due to a trojan or virus from Outlook, or some other downloads). From “netstat”, I saw a bunch of established TCP connection to multiple machines.

    Very likely that those machines captured the saved password from Filezilla, and grab those series of passwords.

  7. Anonymous says:

    This is the big danger of using shared hosting. All it takes is one other user on your shared host that is vulnerable to someone uploading a .php or a .gif.php file with PHP embedded in the comments and all the sites on the server are compromised. Then the hacker just opens the file and the server executes the embedded PHP code. Any where on my sites that accept file uploads, I put a .htaccess file that contains: RemoveHandler .php to completely disable the ability to execute an uploaded PHP script.

  8. Brendan says:

    it could just be a cross-site scripting attack or a SQL injection. You can check the DB for the url’s or code or look into sanitizing your user’s input/use prevention techniques against XSS from affecting your site.

  9. Brendan: Those are indeed the most common ways sites get attacked, but that wasn’t the case this time. I know that the hackers had my FTP password because until I changed it, updates kept getting automatically uploaded to the site.

  10. Ezra says:

    I turn off FTP on all of my sites as FTP passwords are sent unencrypted and can be sniffed by any server in between the server and client — probably at least 2 dozen servers. If you use SFTP and don’t save your password you should be in good shape.

  11. Michael Hill says:

    This is exactly why people should not use ftp and instead use a versioning tool. I work with clients who go through the same issues you have and it’s a direct result of not implementing proper procedures in both your scripts and the method you use to push code and content to your site.

    All professional sites need to have versioning control in their workflow. A system like git would have save you countless hours of finding exactly what files were changes(and when) which in turn gives you the method in which your site was exploited. This is obviously not true with all attacks such as SQL injection which will not change your files but puts harmful data into your database.

    In your case you could’ve taken 2 seconds to find out what files were changed/added by simply doing this:

    git status

    In addition you should NOT be using ftp for site file transfers for anything you value. FTP passwords are sent in plain text and it’s not uncommon for them to be compromised. I’m assuming however, that the tool that was used allowed for the exploiters to see your username/pass directly from the backdoor php script. Transfering your files and changes to a site is also horribly inefficient. Copy and Paste is a tool for people who don’t understand the importance of security and their websites. Instead you should be doing something like this:

    Locally:
    git commit -a -m “Adding latest files and changes”
    git push origin master

    On the server:
    git pull origin master

    And about 75% of your security issues would be resolved.

  12. Michael: Thanks, you definitely make a valid point.

    However, using versioning control in a shared hosting setting is problematic. E.g., the webhost for robozzle.com specifically does not support versioning control, at least not for Windows hosting.

    I agree that the ability to use versioning control is an important thing to have in mind when choosing the host for a serious site, though.

  13. Hans says:

    I had the same problem with a shared hosting service I used to use. The hosting service merely blamed it on a weak, guessable password. I tried to help them by pointing out the complexity of my password was mathematically impossible to guess at random in a thousand years on top of the fact that I was aware of other clients and friends using their hosting service who had the exact same thing happen. All I cared about was helping them sort it out so it wouldn’t happen again (as this was the second time in a few months my site got hacked). It became incredibly infuriating that, despite all the evidence I was laying out in front of them, they refused outright to admit that it was anything else other than a ‘weak ftp password’. Whatever the case, I came to a similar conclusion that either they had bad security and passwords were somehow leaked. In our case, it was a script that would update all index.php/html files with a hidden iframe and, in another case, adding an htaccess rule that would redirect to an external site if your referrer was a search engine (presumably to disguise it from the owner of the site, until the seo damage has been done). I then informed everyone I knew that hosted with their service and most of us left. It’s one thing to hide the real truth from a casual user, but another to continue denying things from a group of people, who have evidence and display an obvious understanding of what’s really going on. I now host my own servers and haven’t had a problem since. Ubuntu server has a nice feature of letting you decide whether you want updates to be automatically applied as they come in, keeping your system patched, as well as ksplice, which allows you to patch your kernel without rebooting.

  14. Hi guys, ive got a problem too, i have an online auction site that ran perfectly well until about a month ago, then it started to get really slow and pages where not opening as quick as they should. i checked my files and found files for halifax and other banks and a cleverly written page that has been planted on my server amongst my files, it looks like they where using this program page to get any info they wanted from my site and probably any other site on the shared server, i deleted just about everything i could find and kept all the files for future ref. now i have an ugly link appearing over my home page at http://www.antiques-market.net it leads to a directory page somewhere / could someone shed some light on this and possably how i can get rid of this link, ive looked in my header.tpl but i file and cant find the code when i look into it although i can see it when i look at the original source. this is the code that appears over my header code,<!–
    document.write("“)
    //–><!

  15. Eurania says:

    That’s the worst thing that can happen to a website, I think this might help http://www.eduardobaret.com/20.....tack-site/

  16. b makowsky says:

    Top-notch article it is definitely. We’ve been seeking for this update.

  17. i says:

    Still Prolems….!!!!

    Only People who have solved at least 10 puzzles can leave a comment.

Leave a Reply

You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>