By Simon Woodhead
Over the bank holiday we performed some short-notice changes to the way our web properties are served and promised an explanation as to why. Before doing so, if you were unaware of these please ensure you subscribe for updates at https://status.simwood.com – you can get them by email, SMS or webhook.
The change we made was to put CloudFlare, the CDN, in front of our web properties, i.e. this blog, our web site, portal and API. To do so not only involved a change of DNS, it required us to move primary DNS authority for simwood.com to them. This was done on the Friday evening and took effect in most parts of the world within a few hours. We were then able to activate their service in front of them.
For most customers CDNs offer performance improvement due to bringing content closer to the end-user. Whilst true for international customers, this is not our reason for doing it. It wouldn’t be relevant for the API and of negligible benefit to UK customers where 86% of our traffic flows over our peering sessions with them or their ISP anyway. Our reason was security, plain and simple.
As a result of this change, all of our web properties were (and are) marshalled through local nodes with caching where sensible (e.g. static elements not the API!) and our true origin servers are hidden. In testing since this has only improved performance although a handful of customers did experience issues with their traffic being blocked. In one case this was due to using a very old library to access our API which did not support modern SSL versions; in another it was due to certain other aspects of the HTTP headers tripping alarms. Suffice to say though, a one-armed man who’s lost three fingers could have counted the support tickets that resulted! That said, we’d have preferred it to be zero and to have spent more time testing.
So why didn’t we? We found it necessary to increase our security posture given recent events. You’ll be aware of recent attempted non-technical fraud, our substantially increased security competence through Cal Leeming joining us (and as it turns out one of our Mauritius team’s hidden skills!), our substantially increased profile in recent years, and finally our inclination to put our head above the parapet by being unique in doing something to combat VoIP Fraud. Whilst we’re refactoring our entire application stack to take advantage of modern code and infrastructure techniques (that’s for another day) which’ll dramatically improve our security agility, that is an ongoing process and our web properties are last in the list. All these things combined with some concerning activity against our API on Friday morning, lead us to not wait and to make a switch over the long-weekend – the extra day giving us more time to roll back out of any issues caused by amending the authoritative DNS.
We’re quite pleased with where this leaves us, although one thing I learned about security many years ago is that the more you learn the more paranoid you become. We’re not quite donning foil hats and living in a Faraday cage yet, which shows we still have a lot to learn, but security is central to everything we’re coding, and we’re recoding everything. We have an awesome team driving this now and improved security is just one product of some really exciting changes coming down the line.
Naturally, we can improve our security and do our best to ensure the uptime of services and network on which you depend. You need to be able to reach those services though so we’d again remind customers of our post “Surviving somebody else’s DDoS“.
