Until you solve prompt injection, they are indeed extremely bad for security and should never be given permissions that would allow them to do anything catastrophic.

[–] verstra@programming.dev 1 points 6 hours ago

I say mantra because there is a large amount of people just hating AI outright, without a grounded reasoning.

Granted, coding agents are insecure by default - they are built to execute remote code - but that does not mean they are generally useless/harmful/bad. I run them in a container, with access to the codebase only.

Also, they hallucinate, produce over-convoluted abstractions, do not know when to reject instead of blindly trying to find a way trough a brick wall.

But also, they can answer questions about gigantic codebases way faster than I could. They can generate tests, find missing test coverage, review code, and many other things.

[–] kingofras@lemmy.world 7 points 1 day ago

mantra

The way LLMs work is that they actively will make multiple attempts to get past hurdles (because they have no intelligence or methodology) so guardrails need to be extremely tight for them to work, other wise the model will simply see it as one of the challenges to overcome.

That’s the mantra, and that is very poor technology to put in the hands of people who don’t understand how it works.