Yep, client hit for the second time today too, after full re-installs on WP and the non WP sites affected last week. so much for their new scanning procedures eh?
its exactly the same as the incidents with Godaddy a few weeks ago, you spend hours and hours putting it right to wake up and find it done again.
at least Godaddy have a "revert to date" feature on their backup system so you can just revert to your last known safe date files, MediaTemple is a f'in nightmare by comparison.
I began dealing with this on 8/6. My affected site is using Wordpress [newest version. In fact, I upgraded the night before this began] via Media Temple. Media Temple didn't give me much help, aside from posting a bunch of links in reply to my support ticket of what "might be" the issue. I ended up reading about "johnnyA" after doing tons of searches. I deleted the user, found malicious code in a sidebar.php and deleted it, requested another Google review. My site began working fine. Now, a week later, I'm right back where I started - with my site being blocked and Media Temple "looking into it".
The problem for me is I'm an idiot when it comes to this stuff and I'm not tech savvy at all. So, while those of you who know what you're doing had a rough time searching for code, etc - imagine what it's like for me, someone who hasn't a clue what to do, what to look for or even what the hell "grep" is.
The kicker is that my personal site is running Wordpress NOT via Media Temple and I haven't had one issue at all. I feel like I've received some sympathy from Media Temple, but not any help. Whether it's a security issue on their end or not [I still believe it is], I can't possibly figure out how to fix things by any instructions I'm finding online because when it comes to this stuff, I'm about the intellectual equivalent of a toddler.
I'm really aggravated. I was able to look at a dropdowns.js within my Wordpress theme and did find the var st1 = 0;document.write(unescape .. thingy. I'm not even using the dropdown feature so I deleted it, but also have a fresh one that I just downloaded in case I do need it.
Seriously - this entire situation has been frustrating for many people who KNOW what they're doing, yet what about us who don't?
PS: I've been told so many times by Media Temple to make sure my Wordpress and plugins are up-to-date, that it truly makes me believe they feel I am the reason for this happening. My plugins and Wordpress were up-to-date, which makes this even more frustrating. I'm basically at the mercy of someone at Media Temple "possibly" helping me. The $20 credit they've given me over this doesn't nearly cover the time and aggravation I've spent trying to understand how to fix this myself.
I've been dealing with this problem too. Found the JohnnyA as a wordpress user.
The problem with not knowing how the initial upload occurred is that it is difficult to seal definitively.
It might be worth looking around on hacker sites to see if the bragging reveals any details. I could imagine, for instance, an exploit that used some of the file upload plugins for TinyMCE.
By the way, we updated our blog post regarding security. I invite you to please read it and let us know if you have any further questions:
(mt) Media Temple Weblog - Security Facts
Thanks!
I've been tearing my hair out over this. I was thrilled to find this page after desperately typing "1RqLcpt" into Google. Now, I'm back to hair removal after finding quite a few affected files across multiple domains.
There's a lot of very useful and important information here. I'm hoping it will eventually be possible to compile all the known resolution methods into some kind of tutorial for systematically tackling the problem. I've already lost one client due to their site being blocked/blacklisted, so I'm not happy.
The main thing that concerns me is that (mt) doesn't know where the hole is - or precisely how it's being exploited yet. With all due respect to them and their efforts, I have a sinking feeling I'm going to be hunting for obscure snippets of code until they (and the rest of us) do know for certain.
Based on what I've read here and seen on my own account, I'm convinced that TinyMCE does play a significant role in this. Here's something from last year that seems disturbingly relevant:
http://yehg.net/lab/pr0js/a...
I could be wrong, but I suppose we'll find out.
I'm a tech support agent over at (mt) Media Temple.
We can confirm that this blog post is correct. From internal investigations, we're finding several variations of injected obfuscated code. Right now there's no sign of our infrastructure being breached, or WordPress itself being breached. The vast majority of compromised sites are running 3rd party plugins and themes. These themes appear, at this point, to be the primary point for the malware to be injected into. This is still under investigation. We cannot confirm this 100% as there appears to be other vectors but, for now, themes should be audited for security loopholes.
A recent Softpedia article states:
"The mass compromise was detected by Websense's ThreatSeeker Network, and even though the affected websites are hosted at Media Temple, this does not imply any security problems with the hosting company's servers or infrastructure. Similarly to other hosting providers, Media Temple has had its share of compromised websites under its roof in the past and this is because hackers systematically scan entire address spaces for vulnerable targets, before proceeding to infect them."
You can read more about Websense' findings here: http://bit.ly/91vTvU
(mt) Media Temple is currently working on a blog post to address what customers are facing, working on helping customers scan and clean up their blogs from these attacks. We're committed to being as transparent as possible with our findings to aid others being affected, regardless of where they are being hosted.
Attacks like this affect the internet community as a whole, it's important to work together and share information. For the time being, we share our information and resources at http://mediatemple.net/secu...
If you're a (mt) customer and have any questions, contact us 24/7.
Thanks for chiming in, and sorry about the bizarre formatting on your comments (I've got some Markdown issues to smooth out :). The issue, though, is that my neighbors' carelessness (domains/ permissions) has cost me over 15 hours of cleanup time across 20 some domains, and I have no solid guarantee that the access points are sealed off. I'm thrilled that you found the source of the breech (honestly), but can we get a solid statement that the problem is solved and I won't be spending my evenings grepping anymore?
I'm very sorry that you've had to go through that.
Honestly, we could not find any consistent connection between users who had been compromised by JohnnyA and users who had edited their /domains/ permissions. Thus, we can't say for certain that it was the source of the breech. Also, unless you had chmod'd the perms for your /domains directory yourself, your sites would not be vulnerable from your neighbors. With regard to the guarantee that access points are sealed off:
The ACLs that we applied across the entire (gs) Grid-Service has sealed up any possibility of neighboring users from viewing any files regardless of permissions.
If you want more information on security at (mt) Media Temple please visit our security wiki page.
I've been hit with this hack. Incidentally, the second hack since I've been on MT. I'm intrigued as to why you think the fault doesn't lie with MT and lies elsewhere (or is it simply because MT says it's not their fault?)? My experience of dealing with them suggests that at least part of the blame lies at their door. The fact that they haven't detected the attack vector yet, despite there being (at least) two bouts of this attack, one being after we know all users changed their passwords, seems to call into question the quality of their tech staff. Either that or their covering up.
I'm still searching for the trojan files. My solution for the time being is moving to a different host after cleaning my theme files (not that I've found anything in them) and cleaning my database.
I'm growing less and less sure that it was a security flaw in my software and not an overall problem at MT. I found a new rootkit this morning, and a whole new series of injections. Not as elaborate and easier to clean up, but I'm quite sure I had any permissions problems taken care of, and software up to date.
The new rootkit was in a file called css.php, and located in a somewhat random folder in a Sunshop install. It's detectable using the 255 character greps above, or a search for "eval(gzinflate(base64_decode(". It's a binary that's base64 encoded and gzipped that becomes executable when the file is accessed via php. Not cool, and how it got there is beyond me at this point.
<div markdown="0">
Hi guys, so I am facing the same problems and I am also on (mt) GS platform. I think old versions of wordpress are the point of entry, but there is no certainty.
Anyhow, I wanted to add few lines of bash that I use to search for these exploits across 20 sites...
To find the stuff in the JavaScript files I use:
find . -name "*.js" -exec grep -q "%[0-9A-Z][0-9A-Z]%[0-9A-Z][0-9A-Z]%[0-9A-Z][0-9A-Z]%[0-9A-Z][0-9A-Z]%[0-9A-Z][0-9A-Z]%[0-9A-Z][0-9A-Z]%[0-9A-Z][0-9A-Z]%[0-9A-Z][0-9A-Z]" '{}' \ -printf "%Td-%Tm-%TY %TT %p\n"
(displays path with a timestamp to a file containing the exploits while searching for these %TWOCHARS%TWOCHARS% which are characteristic when 7 in a row)
That covers all bases and I run it daily and monitor file changes just in case.
I improved security of some WP installations too.
Plus some resources:
I use this to confirm the problem is gone - http://www.avg.com.au/resou...
This helps me understand what hackers want - http://www.tareeinternet.co...
And this is something that helped me with RegEx - http://www.gskinner.com/Reg...
Good luck guys!
Then to make sure and extend my search, I use:
find . -name "*.php" -exec grep -q "[a-zA-Z0-9\/\+]\{255,\}" '{}' \ -printf "%Td-%Tm-%TY %TT %p\n"
find . -name "*.js" -exec grep -q "[a-zA-Z0-9\/\+]\{255,\}" '{}' \ -printf "%Td-%Tm-%TY %TT %p\n"
(which finds lines longer than 255 characters... kinda accurate)
Then (maybe an overkill...) I do this to scan the PHP files:
find . -name "*.php" -exec grep -q "base64(" '{}' \ -printf "%Td-%Tm-%TY %TT %p\n"
find . -name "*.php" -exec grep -q "eval(" '{}' \ -printf "%Td-%Tm-%TY %TT %p\n"
find . -name "*.php" -exec grep -q "gzinflate(" '{}' \ -printf "%Td-%Tm-%TY %TT %p\n"
find . -name "*.php" -exec grep -q "str_rot13(" '{}' \ -printf "%Td-%Tm-%TY %TT %p\n"
find . -name "*.php" -exec grep -q ";document.write(unescape(" '{}' \ -printf "%Td-%Tm-%TY %TT %p\n"
(these give some false positives, but I'd rather check to make sure)
</div>
Sucks man. I had a big meeting today where a contact hyped up what a great tech team we're on and my site had been malware blocked by google.
Anyway,
Another thing to remember to do, if you can't even get to your site anymore is the following:
Log in through cmd line in mysql and do:
SELECT * FROM wp_users ORDER BY ID DESC;
In my instance the last two users in there were "johnny" users. Make note of the user ids and delete the johnny rows.
ALso remove corresponding rows from wp_usermeta that match the user_ids you just deleted.
I have 5 sites I manage on Media Temple each with their own account and a wordpress install with version 3.0. All were compromised with the JohnnyA malware attack.
This appears to be more and more related to Media Temple's lack of ACL rather than a wordpress exploit. However, it greatly disappoints me that they have failed to come forth and admit they "screwed up"...again.
I found the offending rootkit exploit located in folders nested deep without anything in common to the locations with other sites I manage. You can add these file names to the list:
chmod.php
fopen.php
mkdir.php
Still the only way to locate and eradicate the exploit is to use the following "grep" commands with the above mentioned string both for the javascript and php rootkit.
grep -R "document.write(unescape" *
grep -iR --include "*.php" "[a-zA-Z0-9\/\+]\{255,\}" *
After you're done make sure to send a nice note to Media Temple demanding a years free hosting for the trouble...it wasn't too long ago we all had the hassle of changing our MySQL passwords due to a lack of security measure by Media Temple as well.
I'm a tech support agent over at (mt) Media Temple.
I just want to clarify that the ACL's and permissions issue that we mentioned in the update, were the result of internal security auditing. Basically the issue was this: if a user intentionally granted global read permissions on their OWN /domains/ folder, it was possible to be read by immediate neighbors.
We treat ANY kind of security vulnerability in our architecture very seriously, needless to say this discovery was top priority and fixed within a day using ACL's.
I would like to be extremely clear about what was found:
• This did NOT affect all (gs) users
</li>
• I personally spoke with the techs who wrote the proof-of-concepts the night before the ACL's rolled out. A very, VERY small percentage of users had messed with their /domains/ folder permissions and were therefor susceptible. Some clusters had a few as a few dozen users with self-modified permissions.
For *nix savvy users, I'd like to put this into perspective:
• Default ~/domains/ permissions for every single (gs) Grid-Service, on any Cluster is 750.
• These users essentially ran something like chmod 777 ~/domains, some users ran 755.
• The ACL's that were rolled out essentially stopped any chance of local neighboring users from viewing any files regardless of your /domains/ permissions, even if set to 777.
• We found no evidence that anyone had found this or used this previously. There was certainly no indication that the few users making their own sites potentially vulnerable could cause WordPress exploits for users with proper permissions on entirely different clusters.
I completely understand your concerns, but NOBODY benefits from hiding information about these exploits. We're not in the business to cover up information that could help us, our customers, and compromised users on other hosts make informed decisions before and after a hack.
I've been dealing with this issue today as well, and thought I'd share some of what I've encountered.
I wasn't able to find "is_writable.php" until later, but another file called "fopen.php" caught my attention first. The code inside wasn't found with your find command above, as there were spaces (eg: "$o = " instead of "$o="). So the search I used was :
find ./ -name "*.php" -exec grep -l "$o = '1RqLcpt" {} \
With this search I'm finding a bunch of files named after different PHP functions; chmod.php, mkdir.php, eregi.php, etc.
Frustrating, as I manage a bunch of sites and it's tough to track down where the initial intrusion occurred.
I had a modified version of the hack pop back up, so I went looking for the rootkit again. I ran Chris' modified find
statement above and found about 50 PHP files named after UNIX functions (eregi, content, is_file, fclose, etc.). I double checked the list for false positives (none) and then piped it out to a text file (badfiles.txt) and just ran:
for file in `cat badfiles.txt`; do rm $file;done
Dangerous thing to do, but I definitely did not feel like rm'ing them one at a time. Worked great. Still have a lot of cleaning to do, apparently, but thought I'd throw that one out there.
Thanks much for the additions, hope they help some more people out. The most frustrating part for me was that each site of mine that got hacked had unique aspects to it, not something I could automate very well. Highly annoying.
Did you notice any files added to the TinyMCE folder that's included in WordPress? I noticed that this folder path was the most common across all of the affected domains. This path in particular:
(domain name)/html/wp-includes/js/tinymce/plugins/inlinepopups/skins/clearlooks2/img/
I wonder if that wasn't the point of entry, or if it's just a coincidence.
That's exactly where I found them in one install. I didn't list it verbatim as I didn't find them there in all installs and figured it was random.
load more (1 remarks)
Do you have anything to grep for to find the "is_writable.php" file? I don't have a file of that specific name.
I posted my steps to remove the malware, too:
http://www.uhleeka.com/blog...
Alika
I'm afraid I don't. I had a bit of a eureka moment when I found the files, and proceeded to gleefully delete them and seal off the directory. I forgot to check for strings to help locate them in the future, which sucks. They did, however, have a current timestamp, so the easiest way to locate those files would be to do a find for files within a certain date:
find . -name "*.php" -newerct "2 weeks ago"
adjusting the date to whatever would include the approximate time you got hacked…
thanks for the great info, one thing i'm having a hard time finding out though is how they got in and performed this hack in the first place, what / where is the weakness that needs to be hardened? someone's ftp password? a hole in wordpress? what?
I wish I could be of more help in that area, but I'm still trying to figure that out myself. I recommend changing all of your database and admin passwords, and making sure that you don't have world-writable (777) files accessible from the web. MediaTemple doesn't seem to know the common point of entry, either, so I've just been trying to harden everything all around…