If you work in IT or even just run a business with an online presence, chances are you’ve heard this equally shocking and baffling story – 123-reg have managed to delete a number of their customers’ websites (and some backups). And some of these may have no chance of being able to be recovered; there could be companies that might have to rebuild their websites and systems from scratch.
Put simply, a number of their customers’ website have completely disappeared – some with no chance of data being recovered. It has affected a number of websites on Virtual Private Servers (VPS).
Even six days later, things are not looking good. The hosting company have so far managed to recover 39% of the affected servers, which leaves almost two-thirds of their customers with absolutely nothing; it’s also been confirmed that some (though not all) of this data may have been permanently lost.
@ianhamilton_ @123reg this should hit the news. Just to make others aware . This will wreck my business & plenty of others I am sure
— Its a Puzzle Thing (@itsapuzzlething) April 18, 2016
123-reg have tried to play down the situation somewhat, presumably to alleviate fears in the rest of its customers not hit by the issue. It says only 67 of its 115,000 servers around Europe have been affected, although they haven’t revealed just how many customers this relates to.
How Did It Happen?
According to 123-reg, it’s the result of an error in the code of a clean-up script. When this was run, entire virtual servers hosted by the company completely vanished. Since these sites were stored on a number of 123-reg’s VPS, this catastrophic error in its routine clean-up script means that multiple websites were wiped out for every individual VPS affected.
What Does It Mean For Affected Businesses?
If 123-reg didn’t have a backup of your virtual servers or websites then you could be completely out of luck; the host has told some customers that tens of gigabytes of data has been permanently lost and cannot be recovered. As a gesture of goodwill, 123-reg have offered affected customers 6 months of free hosting – but this hasn’t gone down particularly well with a number of those whose sites and servers have fallen foul of the scripting error.
So official email from @123reg saying no chance of restore my VPS and trying to offer me 6 months free VPS hosting… #fail
— Martin Chatwood (@Martin_Chatwood) April 20, 2016
We’ve heard from one of our customers who only had their own virtual server backups, and who were directly affected by the 123-reg scandal. Their website is used to receive customer payments; with every hour that passes while it’s still down, it’s costing them very real amounts of money.
While we do provide them with backups of all their other servers, they had the hosting of the affected sites were covered (they thought) by their hosting provider. Had we had a backup of all of their servers, they would have very quickly been able to be up and running again – making this a perfect example of the importance of using a single independent backup company, like Backup Systems, to handle all of your servers and/or sites!
Would Separate Backup Servers Have Helped?
Basically, yes. Those with no backups could well be out of luck, and even those with 123-reg backups aren’t guaranteed to be out of the woods just yet.
Here at Backup Systems we have had several of our customers affected by this hosting issue. One of these customers, a large UK recruitment company, had two websites hosted with 123-reg – one for their main site, and a second that ensured everyone got paid by contractors uploading time sheets.
This issue didn’t just affect the sites themselves – the offending script also eradicated their backups they had made themselves. Thankfully, we always prepare for these situations; we had another backup of our own of both of their sites, which meant they were back up and running again incredibly quickly.
Back Systems’ Mark Ridley has said:
“123 not only deleted the [customer’s] site but also their backups. Luckily we had another backup which got them up and working quickly.”
Without this additional backup of their sites, however, it would be a very different story – one which could have had very real financial impacts not only for the company but its customers and employees as well.
What Should I Do To Protect My Business?
If you’re affected by the 123-reg hosting issue, or even if you’re not, we can’t stress this enough – always keep backups. While backups with the hosting companies can be useful, it’s not always worked as in this case; third party backups (such as the services we provide here at Backup Systems) give you an extra level of certainty that you’ll be up and running quicker than others affected.
The next step for any business, particularly those who rely on websites to manage, receive and send out payments, should be to get a backup separate from your hosted website – ISPs aren’t infallible! We offer an extensive managed backup service that would have protected anyone affected, and kept their online presence running.
Featured image source: Pexels – Pixabay. Licensed under CC0 Public Domain