595

Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.

Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.

The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

you are viewing a single comment's thread
view the rest of the comments
[-] SkyNTP@lemmy.ml 42 points 2 months ago

LLMs == AGI was and continues to be a massive lie perpetuated by tech companies and investors that people still have not woken up to.

[-] ContrarianTrail@lemm.ee 10 points 2 months ago

Who is claiming that LLMs are generally intelligent? Is it just "they" or can you actually name a company?

[-] kautau@lemmy.world 10 points 2 months ago

I think the idea is that every company is dumping money into LLMs and no other form of alternative AI development to the point that all AI research is LLM based and therefore to investors and those involved, it’s effectively the only only avenue to AGI, though that’s likely not true

[-] exanime@lemmy.world 2 points 2 months ago

You mean the stuff currently peddled everywhere as "Artificial intelligence"?

Yeah, nobody is saying they are intelligent

[-] ContrarianTrail@lemm.ee 9 points 2 months ago

AI and AGI are not the same thing.

A chess playing robot is intelligent but it's so called "narrow intelligence" because it's really good at one thing but that doesn't translate to other things. Human are generally intelligent because we can perform a wide range of cognitive tasks. There's nothing wrong at calling LLM an AI because that's what it is. I'm not aware of a single AI company claiming to posses an AGI system.

[-] exanime@lemmy.world 2 points 2 months ago

Yes, I missed the implied meaning when you said "generally intelligent"

[-] TheGrandNagus@lemmy.world 7 points 2 months ago

In game NPC actions have been called "AI" for decades. Computers playing chess has been called AI for decades. Lots of stuff has been.

Nobody thought they were genuinely sentient or sapient.

The fact that people lumped LLMs, text-to-image generators, machine learning algorithms, image recognition algorithms, etc into a category and called it "AI" doesn't mean they think it is self aware or intelligent in the way a human would be.

[-] exanime@lemmy.world -1 points 2 months ago

The person I replied to said nobody was claiming LLMs were intelligent. I just posted that the people behind the push for this overhyped bubble are indeed making that claim

Whether people believe it is something else. But also, many people do believe it

[-] Grimy@lemmy.world 5 points 2 months ago

He said generally intelligent, In the context of the first reply using the term AGI. There is a difference between artificial intelligence and artificial general intelligence.

[-] exanime@lemmy.world 3 points 2 months ago

I see... At first read I thought the generally was implying somewhat. I missed the meaning in aGi

[-] jaybone@lemmy.world 1 points 2 months ago

The fact that we even had to start using the term AGI when in common parlance AI always meant the same up until recently, shows how goal posts are being moved.

[-] AFKBRBChocolate@lemmy.world 3 points 2 months ago

What people mean by AI has been changing for as long as the term has been used. When I was studying CS in the 80s, people said the holy grail was giving a computer printed English text and having it read it aloud. It wasn't much later that OCR and text to speech software was commonplace.

Generally, when people say AI, they mean a computer doing something that normally takes a human, and that bar goes up all the time.

[-] AA5B@lemmy.world 2 points 2 months ago

It might also be a question of how we define “intelligence”. We really don’t have a clear definition and it’s a moving target as we find out more

  • “reading aloud is something only a person can do. It requires intelligence”. Here’s a computer doing it. “Oh, that’s not really intelligence, is it”
[-] Silentiea@lemmy.blahaj.zone 2 points 2 months ago

To a degree, but, like, video game ai has been called that for decades, I don't think anyone ever thought it was agi. It's a more specific term, and it saw use before the big LLM craze started

[-] ContrarianTrail@lemm.ee 2 points 2 months ago

The thing with 'common parlance' is that it's used by people without a deep understanding of the subject. Among AI researchers, there’s never been confusion about this. We have different terms for different things for a reason. The term AGI has been around since the early 2000s.

It's like complaining about the terms jig, spoon, spinner, and fly, and saying that back in the day, we just called them fishing lures. They are fishing lures, but these terms describe different types. Similarly, AGI is a form of AI, but it refers to a specific kind.

this post was submitted on 04 Sep 2024
595 points (94.5% liked)

Technology

59179 readers
2152 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS