First I'd like to ask you to do 2 very important things. First, I'd like you to change your WordPress (wp-admin/wp-login) password to a newer and more secure one, if you haven't done so already and second, while I know my message is rather long, please take a minute to read it and of course if you have any questions please feel free to ask me. Let me explain what all this fuss is about.

As you may know, this past Tuesday, a widespread "brute force" attack against WordPress sites started impacting sites across the entire internet. This attack is leveraging a botnet which looks to have more than one hundred thousand different computers at its disposal. Its intent is very simple: to find and compromise WordPress sites with simple passwords, to likely later use them to distribute malware (and further increase the size of the botnet). You can read more about this on the net: http://www.computerworld.com/s/article/9238377/Wide_scale_attack_against_WordPress_blogs_reported

On Tuesday, our backbone service provider, Level 3 Communications discovered this attack as I investigated increased load and decreased performance on several of our hosting servers. I had already been dealing with installation of a new hexacore-super-server to offset unrelated server loads. I quickly identified this as a widespread attack on the WordPress login page. The attack was a large one (hundreds of hits a second to many WordPress sites spread across our infrastructure). It became quickly obvious I needed to act fast. At this point, the fastest solution was to drop all traffic to the WordPress login page (wp-login.php) while myself and my Information Technical Services (Yes, while I'm basically a 1 man gang I do subcontract help, more than you may think) worked on a better plan.

The downside to this, of course, is that we blocked legitimate access for customers who wanted to login to WordPress. We knew that was not an acceptable solution for very long, so I immediately went to work on a better solution. I truly apologize if I kept you from logging into your WordPress, but I felt that keeping your site up (but not allowing you to login), was the better option.

With the infrastructure stabilized, we dug in and started investigating better solutions. We reached out to some partners and other groups on the web, and collaborated on some security rules that would help mitigate the attack. These security rules are, in a sense, rules based on behavior: if a single IP address or browser used the wrong password on a WordPress site more than a handful of times in a few minutes, we would ban that IP address for a period of time. This rule would help us allow legitimate customers to login to WordPress, but would stop the attacker after a number of bad attempts.

I rolled these changes out Tuesday afternoon. It took a few tries to find the right balance to block the bad guy but not keep a legitimate user from logging into their WordPress site. The attack subsided overnight.

The attack returned in all-out-hell-force on Wednesday as we reached peak business hours and with a never-ever seen before assualt on one of my client's many websites, "the most popular boy in class" (bet, you can't guess who I'm referring to?). This made it obvious that the attack was based off a botnet—likely using the computers of unsuspecting office workers coming in for a normal day of work! I spent Wednesday tweaking rules and working with other folks in the industry to share tips, tricks, and findings.

By this point, between ourselves and our partners, we were approaching having flagged nearly that hundred thousand IP addresses, and more new IP addresses were showing up every second. Even though we were stopping much of the attack, it was so large that simply handling the traffic was starting to impact our servers.

By Thursday, it was clear that the attack was not subsiding. The first thing I did was to roll out a new heuristic-based set of rules, that would look historically at our growing set of log data, identify patterns, and block the attack based on that data, not just on current bad behavior, but combinations of bad behavior.

That put a big dent into the attack. But the attack was still big enough to be causing our servers to run at a higher than normal load.

My breakthrough happened on Thursday, as my tech people out in Colorado looked through data on the web and data in our logs. We found a difference between the way the attack accesses WordPress and legitimate customers access WordPress. Friday morning I rolled that change out to our cloud servers (before the traffic even reaches the web server that might be hosting your site) to drop any traffic that didn't look legitimate.

Hundreds of hits a second dropped to nearly none.

As we head into the weekend in good shape, but vigilant against a returning or altered attack. In the meantime, my support desk is ready to help you if you are feeling any lingering effects (the most common one might be if your IP got marked as a possibly bad IP).

As always I appreciate your business and know that I'm dedicated to give you the very best hosting expierence by personally paid attention to nearly every detail of your website and IT needs.

Again, please take a moment to change your Wordpress login password.

Truly yours,

Mark Wilson

Saturday, April 13, 2013







« Back

Language:
Connect With Us
Copyright © 2016 Great Atlantic Media Group. All rights reserved.       
  Site Map | Terms of Service | Privacy Policy