Three weeks ago I wrote about an entry on our status page that nobody needed to worry about: a fibre cut in Slough that rerouted in under a second and cost nobody a single call. Prior to that I wrote about a similar incident in Volta where two legs of our network were cut - same non-event.
The point of writing about these, as I said at the time, is that they're the only way to show the work and investment behind an architecture that's designed to survive this stuff. I do so not to gloat (although some will disagree) but because it is a genuine point of differentiation and I live in no small hope that those who have other priorities for investment (Ferrari-cough, cuddly-dogs-cough) might be motivated to up their game. This industry improving benefits us all, not least end-users and our customers who cannot solely depend on us in an industry that is by its very nature interdependent.
I write now because you’ll see another entry on our status page for Monday (13 July 2026 if you’re reading this in the future) relating to our Slough Availability Zone (AZ). It might have cost some of you a portal login, an API call, or a chunk of an afternoon trying to update your locked balance; or you might not have noticed. It is more likely you did notice because this actually affected some services ancillary to our main services - actual call volumes were normal and not a single 999 call failed (we specifically alert on that, however caused). There’s learning here for us though, and everything I said above would be hypocrisy if we didn’t similarly share. I’d also sooner you heard the truth from source rather than a distortion from a Mustard Moron in some dark corner at the CCUK Christmas party.
So… there are four factors at play here that set some context:
- Firstly we’ve been refreshing hardware around the network. You won’t have noticed because that’s how we’re designed. The London AZ, the Manchester AZ, Newport which isn’t an AZ but regrettably still exists - all done. Slough, next on the list and thus still running old gen hardware and build.
- Secondly, you’ll have seen us talk about new auth and you’ll have probably seen that relating to the API already in the portal and you may even be using it. The portal itself hasn’t been moved across yet - for you at least, we’re using it in staging. That means it is dependent on an API you’ve never heard of (v4) which should never have existed and was written way back in the day by a French man - our resident chaos monkey, a title he’ll still be proud of - because he was fed up of waiting on another former colleague to make the changes he needed to API v3 - the one you do know and love. It has sat as middleware for selected portal functions (like login) for way too long. v4 is completely replaced by the new API architecture, which I’ll call v5 for simplicity even though it introduces independently versioned services starting at v1. So v4 never existed and v1 comes after v3; with me so far? Good.
- Thirdly, we rely heavily on DNS for global service discoverability where local load-balancing or internal anycast can’t cut it. For that we use DNSMadeEasy and in particular their Constellix product which allows automatic failover of records and we’re hooked into its API for dynamic updates. They used to happen within the TTL (15 seconds) which was great. Constellix was bought by Digicert in 2022 and it's apparent their standards have deteriorated.
- Lastly, we recently parted ways with a long-serving and very capable engineer. That left gaps in ownership and institutional knowledge we hadn't fully backfilled - which matters below.
That's the context. I hasten to add not excuses, the ownership comes later!
So, at 14:27, an old host in our Slough AZ died. Not gracefully - disk I/O error, no IPMI, no way to even power-cycle it remotely. It was one of the older boxes on the estate, with less monitoring wired into it than anything we've bought in the last few years, and it had passed a health check a month earlier. It just went, the way hardware does, with no warning anyone could have acted on.
But that is what we plan for and nobody outside of our DevOps team should have noticed. Other hosts have failed the same way more than once this year and it's been a non-event every time, because they're on current hardware with proper out-of-band management and deployed to our current standards. This one wasn't, and that's the actual story here.
We have hundreds of container images, deployed dozens of times in some cases, making for a large estate. Within AWS we use Kubernetes to orchestrate this, but in our three on-net UK AZs, they use Docker with our own network stack. This is something I’ve blogged and presented on many times in the past as, uniquely, a container is a first class citizen on the network, speaks BGP to its host, pushes its own firewall rules to the edge etc. etc. All good but a lot to orchestrate and therefore we have strict policies around deployment to ensure availability between AZs and across hosts within an AZ. Anycast makes light work of this for many things - just run them everywhere - other things such as our database of record Galera is a cluster with a node per AZ, ElasticSearch is sharded and distributes shards according to its own strict policy and so on. We also make enormous use of Redis - like 500,000 requests a second levels of use - around the network. Read nodes are anycasted so everything consumes it closest, often local, usually on the same box, for minimum latency. If you know Redis though you’ll know it has a single master under which everything else is a slave. To manage the master-candidates around the network we use Sentinel, which seamlessly promotes a new master and ensures everything uses it. Much code is hooked in to use Sentinel to benefit from this but certain code which can’t instead uses DNS which is updated automatically on a state change.
The box that died had all of the above on it, as well as loads of other services. They all, even Galera, did what they should and just got handled. Manchester and London AZs, as well as the many on AWS, and our Global Edge, all continued completely uninterrupted exactly as they should. That is worth celebrating I think because I don’t think any of our competitors would have gone without a major outage losing primary database nodes, let alone primary nodes across dozens of services. But this isn’t about them, and we hold ourselves to a higher standard.
What didn’t work was Redis. Sure, the slaves carried on working, as designed, which is why other sites didn’t skip a beat and why call volumes remained normal (voice runs on different hosts altogether - Voice Compute Nodes (VCNs) vs Compute Nodes (CNs) in our parlance), but nothing could write to the master because Sentinel hadn’t failed over. Also, a load of services which rely on Sentinel to discover the new master failed to connect because there simply wasn’t one - the old one was in Slough on a host which was now dead. It's fair to say all hell broke loose in Slough and a cascade of weirdness ensued.
Notably portal logins failed on Carrier Services because API v4 couldn’t connect to Redis. Portal logins failed on Hosted for similar but independent reasons. API endpoints like ‘calls in progress’ stopped working because call events weren’t being processed and customers relying on the balance-update endpoint were understandably concerned, since the API writes to Redis as well as Galera and couldn't. It didn’t matter though because millions of CDRs weren’t being billed because requisite services couldn’t reach Redis. Finally, we also lost a SIP proxy in Slough which was wrongly deployed on the Compute Node which had failed, but I’ll return to that.
On investigation we found out that Sentinel didn’t exist anymore. Rather than existing in every site per our spec, it only existed in Slough on the old host which had failed. Let's say that was an ownership problem in origin - Sentinel hadn't been deployed elsewhere as hosts had been upgraded, so Slough was the only node - Sentinel hadn’t been deployed elsewhere as hosts had been upgraded so Slough was the only node. I believe in decentralised command - or trusting people to do their job - especially where people are technically so capable, but the reality is you can’t delegate accountability and we did. We have to own that and make sure it can’t happen again.
In terms of mitigation, we didn’t just want to stand up a new Sentinel with unknown consequences for the now unmanaged cluster so we promoted a new master manually. That triggered a DNS update but the services which use DNS for discovery of the write node didn’t update. It turns out that despite a 15 second TTL, Constellix took over 30 minutes to push the change out. Digicert’s support team wasn't aware of the product last time we contacted them recently and we were in the middle of forcing changes in other ways when the change took. That left the other services which consumed Sentinel in order to discover the write node; Sentinel which wasn’t there anymore. They needed patching, testing and deploying one-by-one to resolve. That had to be done in priority order and meant some things, themselves 2 minute fixes, took longer to resolve than they otherwise would have done.
Out of hours after the incident we redeployed Sentinel and, with appropriate escape routes in place, let it control the cluster. It did so with the poise and lack of drama we know and love it for, and which it would have done during the day had it been there.
That covers pretty much all the weirdness customers saw with two exceptions.
The first was “calls in progress” and “locked balances”. The latter was a simple case of Redis writeability being restored and that was one of the earlier services to be fixed. However, customers polling the calls in progress endpoint and firing off updates to locked balances were seeing an out of date balance simply because calls in progress wasn’t updating - a low priority thing to address. It wasn't low priority to them as they believed they’d run out of credit any second or that their change hadn’t worked; even when writability was back and CDRs were catching up, they were updating their locked balance to an impossible figure by virtue of decrementing a historic figure.
The second was the outbound carrier proxy in Slough. The customer facing proxy was working as was the rest of the stack but on the way out of the network there is another edge proxy in every AZ. We knew this was down from our monitoring, and some customers in Community Slack even told us, but it was low priority as voice calls were working fine according to our own automated testing and volumes. In days gone by the very first thing we’d have done in an incident like this is modify DNS to fail traffic away from the affected AZ, regardless. On this occasion we didn’t and that was a mistake, even though it would have been far more latent than designed due to Constellix issues. When a new carrier proxy was deployed onto the VCNs, the correct home, Slough was back to 100% even with the other Redis based issues. I stand by our record since 2018 that voice continued working across the network as other sites were unaffected and even Slough telemetry was fine. However, we did have a couple of tickets from sensible customers who experienced odd issues and we owe it to them to work through those thoroughly and mitigate them in future if there is a way of doing so.
There are two books I love. The first, Black Box Thinking, contrasts airlines with healthcare in terms of how one learns from mistakes and the other covers them up. I’m determined we’re more like the airlines and continue to improve wherever we can, and like the airlines, share that knowledge. The other one is Extreme Ownership by Jocko Willink. Jocko recorded the intro for the SimCon just before the pandemic for those who were there. I hope this post demonstrates how we embrace his teachings. One of his expressions is “discipline is freedom” which I think really captures the true root cause here. Yes, it was a Redis failure resulting from a hardware failure, but that was designed out of being a problem 10+ years ago. The reality is that for all the designs, and playbooks and testing, we ultimately relied on someone to do their job. The fact he took shortcuts is our problem and we should have identified and remedied them before they had an opportunity to cause an issue. The spine running through all of that is a lack of discipline.
There are a few lessons and actions here which we’re applying or have already applied:
- Sentinel is everywhere, per the design
- Services which were patched to bypass Sentinel are being corrected and redeployed
- We need to look at DNS to find an alternative or at least mitigation for Constellix. We already use another dynamic DNS provider for our Kubernetes services.
- We need to investigate and address those presumably carrier proxy derived edge cases.
- We need to fail-over DNS away from affected sites in such an incident in future, as we used to, rather than being overconfident in the local redundancy in place.
- The staging portal will be rolled out to customers ASAP, enabling v4 API to be retired.
- The chaos monkey role is very helpful and we’ve missed the sharpness having one causes. The longer stuff works, the more scary it breaking becomes and the rustier you are at responding. We intend to program in regular datacentre visits (even if no work is planned) and part of their function will be to trigger failures, to test redundancy and keep everyone sharp. Impending legislation requires annual reboots anyway so this is a necessary direction of travel everyone will have to take.
- Containerisation, as I’ve opined many times before, ensures every version of every container is identical, which was a massive win for us a decade or more ago. We can’t have an instance like some of our peers where two magic boxes have different manual configurations. But, only policy directs the placement of those containers and requires them not to be changed without going through a change control and deployment process. It is very clear we need to use some of the amazing tools at our disposal now to enforce that policy. We will have a single pane of glass, separate to monitoring, which visually enforces our policy and would have told us that Sentinel wasn’t compliant with it in terms of running instances and their placement, with the bonus of verifying the signature of the code running to affirm it hasn’t been “hot fixed” since deployment. That wasn’t an issue here but could have been and can be easily mitigated at the same time. Exposing policy violations enables them to be addressed when it isn’t an emergency and necessary changes made to ensure they don’t happen again. As we say in Bitcoin: trust, but verify.
- Slough upgrades are being prioritised to complete the rollout.
I hope that provides some assurance and explanation to all those who wondered what was going on. This isn’t an RFO as the O didn’t happen, but it was far closer to happening than we’ve been in a long long time and shouldn’t have been. We pride ourselves on our architecture and technical performance so flog ourselves more than anyone outside ever could. We appreciate you trusting in us to do so.