Researchers find a way to reliably spot AI writing : technology

[–] AnAmericanPotato@programming.dev 33 points 6 hours ago (1 children)

93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution

This isn't what I'd call "reliable".

I'm also not impressed with their methodology, which is heavily based on Gemini.

To generate mirrored AI stories, we reverse-engineer writing prompts from each human story by prompting Gemini 2.5 Flash (Gemini Team, 2025a) to infer the underlying premise

That's too many steps removed from anything you'd encounter in the wild. They're not even testing against human-prompted output.

And then they use Gemini again to analyze all the stories. Relying on proprietary cloud models for the core of your analysis is like building on sand.

[+] Allah@piefed.world -7 points 6 hours ago (3 children)

only 7% of human work got flagged for ai it is reliable though

[–] GenosseFlosse@feddit.org 1 points 1 hour ago

Dude if my code had an error rate of 7%, my boss would say that this "good enough".

[–] technohacker@programming.dev 26 points 5 hours ago* (last edited 5 hours ago)

Percentages are somewhat misleading here. That's around 2 in 25, enough to falsely accuse a student in a fairly standard classroom for example.

I remember somebody mentioning a quote along the lines of "A 95% success rate sounds good, until you become the 1 in 20 that it kills"

[–] Damarus@feddit.org 6 points 6 hours ago (1 children)

That's not good enough. Just a thought experiment: Only 7% of humans died because their work got flagged incorrectly

[–] palordrolap@fedia.io 15 points 6 hours ago

... which will be fed into the next generation of LLMs in order to stop sounding quite so much like LLMs.

[–] Exec@pawb.social 19 points 8 hours ago (1 children)

I wonder how does this stand to writers who have learnt English as a second language

[–] Waterpumpee@lemmus.org 0 points 4 hours ago

At this point, unless you write prose, i'd certainly use AI to improve formulations. See it as proof readers.

[+] Allah@piefed.world -27 points 8 hours ago (1 children)

[–] sbbq@lemmy.zip 30 points 8 hours ago (3 children)

I can't even imagine what dickhead is saying there. Is he implying that the mean real people are bullying the sensitive AI babies?

[–] AbouBenAdhem@lemmy.world 19 points 8 hours ago (1 children)

Since Ben Garrison is the one human whose output most closely resembles AI, I imagine this is him having an existential crisis.

[–] davidgro@lemmy.world 14 points 7 hours ago* (last edited 6 hours ago) (3 children)

You're saying an actual human called that thing both "pen & ink" and a "typewrite"?

... Also labeled the clearly working lamp as "power outage"

... And doesn't understand how USB works

I don't buy it. Even if Ben did real drawing in the past, he clearly isn't now. Edit: Or as a reply pointed out, the signature could be fake too with Ben not involved.

[–] AnAmericanPotato@programming.dev 8 points 7 hours ago

Is this real? Or is it a fake comic making fun of Ben Garrison? It's hard to tell. I'd expect better even from him, which is saying a lot.

[–] frongt@lemmy.zip 3 points 6 hours ago (1 children)

I dont think this is real. Definitely AI generated, very unlikely to be his actual work.

[–] edgemaster72@lemmy.world 1 points 4 hours ago

I agree, reverse image search isn't coming up with anything for this one, but it works on other comics from his site. Seems to be an imitation.

[–] AbidanYre@lemmy.world 2 points 7 hours ago

He's always been a big fan of labels.

[–] ThePowerOfGeek@lemmy.world 9 points 8 hours ago

Why did I think he had died? Probably getting him mixed up with Scott Adams.

I agree with you, it's a typical example of a Garrison cartoon. It's so boringly obvious on a surface level that you can't help but assume there's some idiotic alt-right subtext.

Technology

Our Rules

Approved Bots