By Bruce Rowe, CSS head programmer
It looked like it was going to be a quiet Saturday.
There was beautiful fall weather outside. A couple of short emails were waiting to be read and answered. Several pleasant programming projects were waiting to be dealt with. Then Twila called. “The Tix 5 server is down,” she said. All thoughts of a pleasant, productive day vanished in the wind.
I checked the Tix 5 server. It looked good – I could see all the sites. However, when I clicked on a link, I got a message, “Site is down.” I looked at the other sites. They were also down. I checked the database server. I couldn’t get in. That was a helluva note. Webtix is not much good unless it has data to draw from.
Now, what do we do? I logged into the management console and looked at the situation. The database server was running. But unless we could connect, there was not much we could do. Then, I had a thought. We do have redundant servers. What would happen if we restored from one of them? Could we read the restored data? We had nothing to lose. Why not try it?
We got started restoring. An hour later, it was done. And since there had not been any changes to the data, we knew it was the latest. We pointed our DNS service to the restored data and voila. Everyone was back in business. Total time lost: about two hours
Was this the same kind of problem that happened to the East Coast on Friday? Probably. Will it happen again? More than likely.
Friday’s attack is called a “DNS attack.” What this means is that the guidance for your browser is no no longer working. For instance, if your browser wants to go to https://help.centerstage.com, it first looks it up in a DNS service. The service looks up “help.centerstage.com” and finds that the IP address is 126.96.36.199. Your browser goes there and you are happy. When the DNS service goes down, your browser does not know where to go. It’s like being in a strange city – without a map or GPS.
What is the cause of this? Insecure devices are connected to the internet.
These include routers, security cameras, networked printers, and pretty much anything else connected to the internet. These smart devices also known as the “internet of things.” This is where the real mess is. These gadgets have very little security, hard-coded passwords in the firmware, and they come with default passwords that do not need changing. The makers of the gadgets do not have an incentive to put in better security (that costs them money). The buyers of the gadgets have no incentive to change the default settings. They don’t care and it’s too much trouble.
Can we stop this problem?
Not really. Laws requiring better security built into the gadgets are probably the quickest and most efficient solution. Requiring us to change the default settings on our routers would be a good step. However, that requires a law and the current crop of politicians running around is not interested. Their statements indicate a profound ignorance of everything digital.
In conclusion, the problem is not going to go away. Everyone could purchase their own servers. But, there is more to web hosting than the hardware. You need someone to set it up, configure, and maintain it. These are expensive people ($100-200 per hour). Next, you need the data connections so your customers will have a pleasant experience.
We are stuck. If Facebook, Netflix, PayPal, and Constant Contact could not stop these attacks, we are not likely to do so, either. On the other hand, we do let everyone know when our service is down. That is more than Authorize.net or Constant Contact does.
These days, the internet is too useful and completely ingrained in our world. Occasional outages are part of the price we pay. We should get used to it.
Bruce Schneier, DDoS Attacks against Dyn
Brian Krebs, Hacked Cameras, DVRs Powered Today’s Massive Internet Outage