Technology

People weren't wildly supportive of nuclear power before Chernobyl / TMI either, but after? 40 years of virtual moratorium on new construction in most of the world, absolute rollback in Germany. We've continued to poision ourselves with coal and wreck the climate with CO2 instead of learning to do nuclear right. If building of new plants weren't so difficult, older and less stable plants like Fukushima could have been decomissioned before having major problems.

stock market

Algorithmic traders flash-crashed the market over 15 years ago. AI in trading has been a thing since before then. In that field the latest LLMs are an evolution, not a revolution.

The AI threat to cybersecurity isn't (so much) people accepting vulnerable AI code, it's AI finding the vulnerabilities in widespread legacy code.

As for "lowering the bar" - we've had skript kiddiez since forever, the availability of AI to skript kkiddiez just raises the bar for finding and fixing zero days by the White Hats before the Black Hats find and exploit them.

An unsafe design that wasnt even fully tested.

Still bang on.

I find that the AI tools worth using are the ones that the AI agents themselves configure properly for you.

Learning how to connect an MCP server myself feels like an extremely ephemereal skill - one that will become nearly worthless several days before you start to learn how to do it.

There were lots of "expert systems" developed and deployed in the 1990s, and they have been undergoing modest development improvements since then.

Who’s to say these system are being closely monitored and studied.

Mostly: the FDA, EUMDR and similar agencies and regulatory systems around the world.

There could statistically be patients who have died or been maimed who otherwise wouldn’t if they had real human professionals instead of someone who used a system

Works both ways, and, statistically, the data says the machines are improving outcomes overall. Hotshot doctor I knew had a childhood friend die rather suddenly in his 60s in the late 1990s. Really bothered him so he dug in to the case post mortem, did all the research using then present state of the art computer search tools, and determined what really happened. His friend went to the best possible ER given the circumstances, was seen by the best available human doctors in that site, who made the best possible calls based on the available information, and they made the wrong calls - wrong diagnosis led to interventions that made a bad situation worse and within less than 3 days after initial presentation his friend was dead. It took my hotshot doctor using keyword search tools he was very skilled with months to untangle all the information, and his conclusion was: the only thing that would have saved his friend's life would be blind luck that a specialist (who didn't practice in that hospital) might have been called to his case at random, because the standard of care didn't trigger bringing in the right specialists in this situation, and if the right Dx were made and the right intervention was done, his friend could have lived another 30 years. But, if they did nothing, he would have still been dead within a week, maybe sooner, so they made the (wrong) call, did the (well meaning, but actually harmful) intervention, and made him die faster. LLMs could have saved his friend, turned those months of research using 1990s state of the art search tools into a few hours, timely enough to have made the right interventions when they were needed.

Machines make mistakes too, just not as often as people. Since tthe 1970s and even before, the depth of medical knowledge available in libraries far exceeds the capacity of any human, or even reasonably sized teams of humans, to access quickly on demand. Machine assistance in searching that information has been improving dramatically over the past 30 years. I'd much rather have a doctor that uses it than one that just gives his opinion based on what he learned in school and practice. But, on the flip side, I definitely don't want a doctor who soaks his brain in gin at dinner and expresso at breakfast and relies 100% on what the machine says rather than engaging his own brain. Unfortunately, doctors like that, and others who are more concerned with filling their high paying surgery schedules than actually caring for patients best outcomes are all too common.

Hold on a minute… the field of AI and in general algorithms and models is quite old. LLM is relatively new in there. For science and healthcare we’ve been using specialist ai for a long while for things like image processing, sequencing and what’s not. Those are absolutely monitored have have generally high standards because they are healthcare related already. Don’t mix those things.

They're always going on about how dangerous their AI is.

"Our hey our AI found 3,000 0-day security vulnerabilities in top software, we're only giving it to top software companies so that they can patch their products before we release"

We're all probability machines, from bacteria through the great whales.

I did a calculation about the 3rd generation Claude Sonnet engines... they had similar processing capacity to about 6 bumblebees - using millions of times more energy, of course.

Chernobyl spread significant radiation over 8 million people with effects lasting for decades / generations. It spread detectable radiation across most of the globe.

I truly feel for the tragedy of the schoolgirls, but as Stalin said: one death is a tragedy, a million deaths is a statistic. Chernobyl was a statistic level event, so many individual tragedies that people can't feel them anymore.

In Soviet Chernobyl, the power plant design tests YOU.

I don't think it needs to be an actual AI to cause a "Chernobyl Moment" maybe almost literally.

We have tech bros and random clueless idiots trying to shoehorn these glorified chat bots into basically everything, and an administration that doesn't seem interested in putting meaningful guardrails on the technology. What happens when it gets put in charge of something it really shouldn't and starts to hallucinate?

If it somehow ends up in charge of critical safety systems in a nuclear plant? (Remember that some of these tech bros want to have their own plants to power their data centers)

Or air traffic control and causes multiple crashes? (Pretty sure I've already seen that idea being floated)

Maybe every cybertruck goes haywire stuck on self-driving mode and they cause tons of damages and deaths for a few hours until their batteries run out.

Maybe it gets used to route navigation in Google maps and it ends up causing massive gridlock in every major city around the world for a day or two

Perhaps it gets used for identifying vehicles on traffic cameras and automatically issuing citations, but ends up citing everyone on the highway for a week whether or not they actually did anything plus a bunch of cars that it hallucinated before someone catches on. Millions of tickets are issued and need to be sorted out, if anyone actually pays the fines refunds need to be issued, some people maybe get their license revoked and maybe even get arrested because of it and all of that needs to be set straight

If any of these kinds of things happen on a big enough scale, I think that could be a Chernobyl Moment for AI

Biowarfare has been publicized as something evil geniuses can cook up in a garage for decades now. I think that narrative has been perpetuated by the governments and other large organizations which run clandestine labs...