Articles tagged "RFO"

Incident summary

During routine maintenance, the movement of a network cable brought to light a dormant issue in one networking component, resulting in a degraded connection. Later, while trying to fix this issue, an unrelated network cable was accidentally disconnected for a few seconds. As our infrastructure is built redundantly, both incidents should have had no impact. However, a software malfunction prevented another connection to take over on both system. This put them into an unstable state that resulted in a partial unavailability of our services between 15:20 and 19:45 CEST.

Resolution...

What follows is a relatively detailed and technical description of the reasons for the outage. Because we understand not everyone wants or understands this much detail we added a simpler and shorter summary of the problem:

There was a problem with the devices (switches) that regulate the (redundant) internet (network) connections to our data centre. In trying to solve this isolated problem unfortunately a feedback loop occurred that clogged the multiple connections.

We have multiple connections going to the data centre to prevent losing access to our machines, however this issue managed to...