I used to do some freelance work years ago and I had a number of customers who operated assembly lines. I specialized in emergency database restoration, and the assembly line folks were my favorite customers. They know how much it costs them for every hour of downtime, and never balked at my rates and minimums.

The majority of the time the outages were due to failure to follow basic maintenance, and log files eating up storage space was a common culprit.

So yes, I wouldn't be surprised at all if the problem was something called out by the local IT, but were overruled for one reason or another.

I'm this person in my organization. I sent an email up the chain warning folks we were going to eventually run out of space about 2 years ago.

Guess what just recently happened?

ShockedPikachuFace.gif

Ballast!

Just plonk a large file in the storage, make it relative to however much is normally used in the span of a work week or so. Then when shit hits the fan, delete the ballast and you'll suddenly have bought a week to "find" and implement a solution. You'll be hailed as a hero, rather than be the annoying doomer that just bothers people about technical stuff that's irrelevant to the here and now.

The real pro tip is to segregate the core system and anything on your system that eats up disk space into separate partitions, along with alerting, log rotation, etc. And also to not have a single point of failure in general. Hard to say exact what went wrong w/ Toyota but they probably could have planned better for it in a general way.

Even better, cron job every 5 mins and if total remaining space falls to 5% auto delete the file and send a message to sys admin

10GB is nothing in an enterprise datastore housing PBs of data. 10GB is nothing for my 80TB homelab!

Bad things can happen but that's why you build disaster recovery into the infrastructure. Especially with a compqny as big as Toyota, you can't have a single point of failure like this. They produce over 13,000 cars per day. This failure cost them close to 300,000,000 dollars just in cars.

not to mention the lean process effed them during fukashima and covid, with a breakdown in logistics and a shortage of chips, meant that their entire mode of operating shut down, as they had no capacity to deal with any outages in any of their systems. Maybe that has happened again, just in server land.

It was forever ignore in backlog

Lean philosophy is supposed to account for those dice-rolling moments. It's not just "keep nothing in inventory", there is supposed to be risk assessment involved.

The problem is that leadership doesn't interpret it that way and just sees "minimizing inventory increases profit!"

I work in a manufacturing company that was owned by the founder for 50 years until about 4 years ago when he retired. He disagreed with a lot of the ideas behind lean manufacturing so we had like 5 years worth of inventory sitting in our warehouse.

When the new management came in, there was a lot of squawking about inefficiency, how wasteful it was to keep so much raw material on the shelf, and how we absolutely needed to sell it off or get rid of it.

Then a funny little thing happened in 2020.

Suddenly, we were the only company in our industry still churning out product. Other companies were calling us, desperate to buy our products or even just our raw material. We saw MASSIVE growth the next two years and came out of the pandemic better than ever. And it was mostly thanks to the old owners view that "Just In Time" manufacturing was BS.

Lol good bot I guess

Wow, what a useful bot!

Human error....lol, classic.