Has your host got Ransomware?

Ransomware has steadily become an issue over the last few years with individuals, organisations, hospitals and whole towns affected. A shift from targeting and extorting money from individuals to organisations has clearly been lucrative. Indeed now your insurance company may well have a fund to pay your ransom for you.

Just like any infrastructure, web hosts are going to be a target and a big juicy one, because taking out even part of a host’s infrastructure means you are holding thousands of companies to ransom.

This isn’t a hypothetical either; several hosts over the last year have fallen victim to targeted ransomware attacks and the latest is Managed.com who report via their status page on November 18th.

November 18, 2020 – On Nov.16, the Managed.com environment was attacked by a coordinated ransomware campaign. To ensure the integrity of our customers’ data, the limited number of impacted sites were immediately taken offline. Upon further investigation and out of an abundance of caution, we took down our entire system to ensure further customer sites were not compromised. Our Technology and Information Security teams are working diligently to eliminate the threat and restore our customers to full capacity. Our first priority is the safety and security of your data. We are working directly with law enforcement agencies to identify the entities involved in this attack. As more information is available, we will communicate directly with you.

Managed.com Status Page, November 18th

Eeeek!

Beeping Computer claims the type of ransomware is REvil which is a virus that encrypts files. Now the “good news” is that it genuinely encrypts rather than destroys the data. No point paying if you’re unsure you are going to get your files back. And the blackmailers are asking for $500,000; that’s a lot of money. REvil is a business, indeed it’s a “ransomware as a service”, with a core group of maintainers building the virus while others are carrying out the infections. This also means how it gets on to machines is one of many ways. For a more detailed explanation of REvil both secure works and Krebs on Security have better write-ups.

If I was putting a security pundit hat on, I would now be hazarding a guess as to how they got into this state, but the problem there is the answer in so many ways. Instead I want to focus on what can you do?

It feels like it’s only a matter of time before one of the larger WordPress Hosts falls under similar issues. Not because they don’t take security seriously but that an attacker needs only be successful once. 

So what can we do to potentially mitigate this?

Backups

A good host will take backups for you, a great host will do that and then tell you to take your own. I cannot stress this enough, you need to be in control of your data, take your own backups AWAY from your host’s. Think of your host’s backups as spares.

With backups we can move. I’ve talked before about backup strategies but here are some brief important things to consider when taking backups:

• Backup! Step 1 if you don’t have some sort of backups go set one up.
• Make sure your backups are backing up away from your host
• Make sure you can restore your site with your backups
• Test this process
• Automate as much as possible, but verify and test wherever possible

The first 4 everyone can do, and while the other article goes in depth as to types of backups, when, etc, for now I want you to go check you have external backups, not stored on your host.

If you don’t test your backups, you don’t have backups, you have prayers.

So with backups the worst case scenario, should our site or our host have ransomware, we can rebuild quickly elsewhere. 

I’m hoping most of my readers were already there with backups, but there is another piece in this puzzle, your domain name and DNS.

Managing your Domain and DNS

So disaster has struck, your host has ransomware, you need to move, you visit another host, sign-up, upload and restore your backup.

Then try to log back in to your old hosting company to move the DNS and, oh yes, they have ransomware.

It is really common for hosting companies to sell you the domain and then to have you point your DNS to their nameservers. The problem then is if you ever need to make changes (say to move away from that host) you are reliant on your host.

So here are some tips to keeping control of your domain and DNS:

• Buy your domains separately from your hosting, and ideally from an entirely different provider.
• Do not use your hosts name servers, instead use either the DNS facility offered by your domain provider, or a third party solution (see below). This will mean you will need to manually edit DNS records to point to your A/CNAME/MX records etc but you are keeping control.

So even while I worked for a hosting company, my domain portfolio was held across several companies and I have for many years used DNS Made Easy (affiliate link) for my DNS, this means all my eggs are split into several baskets. 

If my host has issues, I can still manage the DNS from DNS Made Easy.

If DNS Made Easy was to have issues I could change my nameservers with my domain name provider.

Now the weak link is the domain name registration itself and where that is held. Ultimately these can be moved and updated by contacting the registrar, for example for UK domains that is Nominet.

For a DNS provider, I would recommend going with a specialist provider or one of the larger cloud providers not only for added peace of mind, but they are normally significantly faster than your average hosts DNS, come with more features, for example for additional security many support secondary DNS setups allowing you to have 2 providers responding to DNS queries, if one has issues the other takes over. For some sites I have DNS Made Easy and Google DNS setup in this manner.

Email

Many hosts offer free email, and I think many clients see hosting as email hosting with free website hosting. Trust me nothing upsets end users more than downtime on email. Worst, if our host is having an outage and we and they are both trying to communicate with each other on the now not working solutions it’s just not going to work. Worse, it’s not just hosts who communicate with us that way, but so much of our digital work lives managed through email. Moving hosts, guess what, they are going to email you things!

So here are some email tips:

• Email is nearly always mission critical; pay for it. If you are not paying for it, it’s either a bolt on service with no SLA and comes with strings, or you are the product. 
• Separate it, almost certainly when you took the choice to pay, your host won’t be the best option for this. The exception, when they are reselling things like GSuite (or whatever it’s called this week). In those scenarios the services are still provided by the third party and DNS and delivery is determined by their status, not your hosts. 

Process, Document & Test

OK folks, I’m hoping some people are feeling smug, having ticked everything off in this article. They are taking backups, they have separated their DNS away from hosts etc.

But do you have a plan, process and documentation?

When issues arise it becomes stressful and things get forgotten, now is the time to sit down and work out the process, how would we recreate and move our site quickly.

How quickly?

Well, find out. Here is my challenge to you, next week pick up a free 1 month free trial of some hosting and run through the entire process. Assume the worst, you cannot touch your live site or any of your host’s controls. 

How long does it take to migrate?

I can’t lay down such a challenge without also doing it myself, so here is how my disaster recovery test went which was completed earlier today.

• Grab cup of tea and start the clock – 00:00
• Visit the new host home page, find the product I want, register – 00:005
• Wait for email to arrive to create account  – 00:07
• OK, account created, waiting on the server being provisioned – 00:09
• While waiting, checked the TTL on the A Record, found it was super low so left it.
• This server is taking some time – 00:30
• Email arrives with a fraud check, please EMAIL them a picture of the card :( – 00:35
• OK, phone call later and thankfully no emailing pictures of cards and we are up and running – 00:55
• SSH into the box – 01:02
• Add appropriate new user for an ansible run – 01:03
• Run ansible playbook called server-provision.yml (don’t worry if you don’t know what Ansible is or playbooks, I’m just setting up the server from scratch)  – 01:04
• Playbook fails, some dependencies are missing a change between Ubuntu 18 and 20 – 01:12
• Re-run playbook – 01:15
• Success, playbook has run, we have a working server – 01:23
• Run deploy playbook – 01:24
• MySQL user for the site has no privileges for the table –  01:27
• Re-run playbook  – 01:29
• Success the site is deployed but is missing it’s database – 01.30
• Run command to restore DB on site – 01:31
• Site is up and running on new host – 01:33
• Change the DNS record in DNS Made Easy – 01:34
• Visit site, SSL fallen back to a local certificate :( so showing error – 01:35
• SSL fixed itself, the auto-provisioner has provisioned an SSL certificate – 01:37

So in total it took me 1 hour and 37 minutes, to go from sitting down ready to sign up, to migrating the site to a new host. At no point did I need to touch the old host for anything. The backups worked correctly, and, on the whole, the provisioning scripts did too. 

But even though that was a success I needed to tweak some bits. The provisioning of SSL fell back to a local cert, but perhaps a smarter move would be to backup the LE certificate on the old host and use that before re-provisioning.

I genuinely want to know how well you do, I’m expecting many of you to beat my time. I also included signing up to the provider there, but especially if using cloud providers this can be pre-done, for example, I keep accounts both on AWS and Google Cloud so that I can spin up service on both.

So, some takeaways

Ransomware is a growing problem that can strike anywhere, even good hosts and indeed yourselves. The best defence, as the person at the end, is to be portable. Always manage your own backups, and if you don’t do backups, go do them. Separate your concerns by keeping DNS and email separate and finally test your disaster recovery process.

If you do, do a disaster recovery test let me know how it goes!