131
submitted 4 months ago by lemmee_in@lemm.ee to c/technology@lemmy.world

Earlier this month, Google Cloud experienced one of its biggest blunders ever when UniSuper, a $135 billion Australian pension fund, had its Google Cloud account wiped out due to some kind of mistake on Google's end. At the time, UniSuper indicated it had lost everything it had stored with Google, even its backups, and that caused two weeks of downtime for its 647,000 members. There were joint statements from the Google Cloud CEO and UniSuper CEO on the matter, a lot of apologies, and presumably a lot of worried customers who wondered if their retirement fund had disappeared.


UniSuper's mistake was relying too much on Google Cloud even for critical backups.

top 13 comments
sorted by: hot top controversial new old
[-] otacon239@feddit.de 47 points 4 months ago

I think about how many times things like this happen to individuals that don’t have the weight to warrant an article where their entire collection of online data just gets wiped out and Google responds with: ¯\_(ツ)_/¯

Don’t trust your data to the cloud, folks.

[-] Imgonnatrythis@sh.itjust.works 10 points 4 months ago

My thought exactly. I read the headline and thought that this probably happens to people fairly often, I can't believe Google admitted to this... Oh wait there it is - that's why. I have an old Google apps for your domain account. They switched it to an "apps for work account" then they switched it to "apps for business". Then switched it to "G suite". They then switched it to "Google workspace". Each time they made changes in the infrastructure and that account is all whacked up. There's lots of things including Google made things that just don't work with it and features are late to arrive or never do when standard Google accounts are changed. I'm amazed they haven't just accidentally deleted all my stuff in one of these change overs.

[-] SeeJayEmm@lemmy.procrastinati.org 0 points 4 months ago

I also have one of these old Google apps accounts. I wish they would accidentally delete it so I could get the last couple stragglers off and take my domain elsewhere.

[-] redcalcium@lemmy.institute 26 points 4 months ago

During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again.

Your data is safe in the cloud with multiple redundant backups, unless your account is marked as delinquent which will be deleted immediately and irrevocably.

[-] ech@lemm.ee 5 points 4 months ago

That's not what I got out of that at all. It looks more like an errant setting made the account expire automatically after some amount of time, triggering the wipe.

[-] redcalcium@lemmy.institute 14 points 4 months ago* (last edited 4 months ago)

Errant settings that marked the account as delinquent/unpaid at the end of the month, triggering immediate and irrecoverable account deletion. Basically, the scariest part of the google cloud is if they think you can't pay anymore, even if it's a mistake, your account will be wiped along with the backups. They did say they'll have more safeguard after this, but finger crossed.

[-] ech@lemm.ee 3 points 4 months ago

"Fixed Term" does not mean "delinquent". It just means there's a hard cut off. All I know is the snippet you posted, though, so maybe there's more to the situation.

[-] redcalcium@lemmy.institute 0 points 4 months ago* (last edited 4 months ago)

When the account is marked as missed the assigned fixed payment term, it's basically a delinquent account, right?

[-] ech@lemm.ee 1 points 4 months ago

Unless I'm misunderstanding it myself, fixed term means they had a set period of time that's not up for renewal. Ergo, when the term runs out, that's it. There's no chance of delinquency because there's no additional payments.

[-] JeffKerman1999@sopuli.xyz 3 points 4 months ago

What I got out of that is that gcp is not a stable product and should be avoided.

[-] hal_5700X@sh.itjust.works 14 points 4 months ago

Don't trust the Cloud, people.

[-] Jyrdano@lemmy.world 6 points 4 months ago

Abe Simpson was right all along.

[-] autotldr@lemmings.world 4 points 4 months ago

This is the best summary I could come up with:


At the time, UniSuper indicated it had lost everything it had stored with Google, even its backups, and that caused two weeks of downtime for its 647,000 members.

Two weeks later, Google Cloud's internal review of the problem is finished, and the company has a blog post up detailing what happened.

During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank.

This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period.

During its many downtime updates, UniSuper indicated it did not have access to Google Cloud backups and had to dig into a third-party (presumably less up-to-date) store to get back up and running.

These backups have minimized data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration."


The original article contains 651 words, the summary contains 170 words. Saved 74%. I'm a bot and I'm open source!

this post was submitted on 30 May 2024
131 points (97.8% liked)

Technology

58574 readers
4785 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS