Never attribute to malevolence that which can be explained by incompetence.

Including the end of humanity at the hands of the robots apparently

That reminds me of a fairly recent article about research around visualisation systems to aid with interpretable or explainable AI systems (XAI). The idea was that if we can make AI systems that explain their reasonings, then they can be a useful tool, especially in the hands of domain experts.

Turns out that actually, the fancy visualisations that made it easier to understand how the model had come to a conclusion actually made subject matter experts less accurate in catching errors. This surprised researchers and when they later tried to make sense of it, they realised that they had inadvertently dialled up people's likelihood to trust the model because it looked legit.

One of my favourite aphorisms is "all models are wrong, some are useful." Seems that the tricky part is figuring out how wrong and how useful.

It’s worse now than ever though, many managers have been steeped in tech optimism their whole working careers. The failures of “revolutionary new systems” have been forgotten about while the success of other things are lauded.

They’ve been primed to jump on any new “innovation” and at the same time B2B marketing has started adopting some of the most manipulative practices that used to be only used on consumers. They’ve crafted a narrative that shapes discourse so the main objections that appear are irrelevant to the actual issues managers might run in to.

Stuff like “but what if it is TOO good?!” and “what if the wrong people get their hands on this AMAZINGLY POWERFUL new tech?!”

Instead of “but does this actually understand anything or is it just giving output that looks correct?” or “ Wait, so, how was this training data obtained? Will there be legal issues from deliverables made with this?”

The average manager has been primed by the zeitgeist to ask the sales rep the kinds of questions they want to answer.

Seems to me that a lot of the world's problems start with "well, the managers think..." They all seem extremely bad at the whole managing thing, good thing we don't overpay them or anything like that.