1329

cross-posted from: https://lemmy.world/post/14465880

It's the endgame.

you are viewing a single comment's thread
view the rest of the comments
[-] SwingingTheLamp@midwest.social 11 points 3 months ago* (last edited 3 months ago)

A couple of years ago, I was working at a grocery store picking orders using a web app. The store had enterprise-class Internet service with a provider that had two power utility feeds from two different electrical substations a couple miles apart, for reliability.

One day, though, our service went down. One of the power substations had exploded. Shortly, thereafter, the increased load on the grid caused the other substation to explode, too. The cascading electrical failure took out the ISP's backup generator.

That didn't even take nuclear war, just a faulty transformer. (ETA: The disaster preparedness lesson is to look for hidden dependencies between your backups.)

[-] spikespaz@programming.dev 1 points 3 months ago

Can you please elaborate on the technical details of the failures? What was the hidden dependency?

[-] SwingingTheLamp@midwest.social 2 points 3 months ago

The ISP had redundant electrical grid connections for reliability, but the two connections were not isolated at the electrical utility level. A failure in one substation cascaded to the other substation. The operation of one electrical feed depended on the operation of the other, so they were effectively only a single feed.

[-] spikespaz@programming.dev 1 points 3 months ago

But I don't understand why them being connected makes one dependent on the other, unless half of the supply alone can't support the workload. What is the "electrical utility level"

[-] gaiussabinus@lemmy.world 1 points 3 months ago

Both substations feed in parallel. Demand exceeded the capacity of station 2 to supply demand and it tripped the cutout. Only rural connections are radial. Electrical utility starts at the point of demarcation to the generators that make the power.

[-] SwingingTheLamp@midwest.social 0 points 3 months ago

The "utility level" is Madison Gas & Electric's infrastructure. Our ISP had two independent electrical service connections based on the idea that if one went down, they'd still get power sufficient to run their data center from the other. That would be the case if each connection reached all the way to the generating station completely independently. However, the two substations to which the ISP was connected were linked in such a way that a catastrophic failure of one caused failure of the other, so it got no electrical power.

this post was submitted on 19 Apr 2024
1329 points (98.7% liked)

Comic Strips

11685 readers
1132 users here now

Comic Strips is a community for those who love comic stories.

The rules are simple:

Web of links

founded 1 year ago
MODERATORS