Drafting my next blog post:

"The internet today relies TOO MUCH on just a few big players. When one of them stops working, half the world is impacted because too many services, in my opinion, depend on them.
“Too big to fail,” some might say.
“Single Point of Failure,” I respond."

#ITNotes #OwnYourData #TooBigToFail

I am thinking one should never trust a single supplier no matter how big and trustworthy you think they are.

One opinion I have heard a few times goes like this: It's not a problem if we are completely dependent on CloudFlare, GCE, or Amazon. If they go down many other systems will go down as well, and customers won't blame us.

The problem with that line of thinking is, that the bigger the provider is, the less important each customer is to the provider. There have been stories about individual customers of big providers getting into trouble.

@kasperd I know what you mean. Many just care of being a part of a big group. "If we're down and Google is down, too, nobody can complain"
I'm still trying to explain we can stay up if Google is down, and be proud of it, claiming we can stay up even when Google is down.
Some understand it, some don't.

@stefano @kasperd It’s a strange calculation - I’m sure the businesses affected by cloudstrike would’ve preferred their revenue that day over being able to avoid finger-pointing.

@stefano @kasperd However in a decision makers role I understand the herd mentality. If doing what everyone else does turns out to be wrong then you don’t own it personally, if you buck the trend and get it wrong you’ll be labelled incompetent

One example from my own experience is with DNS. Back in 2015 we had a series of outages due to the provider who was hosting our DNS zones.

First the provider had a six hour outage during which we could do nothing, and we couldn't even get in touch with the provider.

Then we tried redundancy across two different DNS providers, which lead to another two outages when the first provider did not handle redundant setups correctly.

At that point I said we will never want to go through this again. And we decided to operate our own DNS servers with redundancy across four different hosting providers.

We have now used that setup for over nine years and not had a single outage of our DNS zone. We have of course had individual DNS servers be down for some time every now and then, but we never had all four go down.

Such a track record of course helps when I am making arguments for multi provider redundancy. I can point at this setup and say over nine years with 100% uptime, had we used Cloudflare we wouldn't have had such a high uptime.