chat

You could argue in the same way that car safety technologies are "bandaid after bandaid to try and compensate for the fact that cars are fundamentally huge hunks of metal going really fast".

The parallel doesn't really work (there is nothing fundamentally incompatible between the purpose of a car and the basic design idea); but to try and answer, this would be a viable answer if the huge "AI labs" admitted the tech doesn't work and were collectively working on entirely new architectures (likely ones that include actual symbolic representation of the world, not just statistical links between words).

But they're not, they're using the same fundamental approach and doing ridiculous shit around that approach - ultimately a genuine, and interesting, NLP breakthrough - to try and sell it as what it's not and will never be.

you still need to do quite a bit of handholding and double checking it gets stuff right but it definitely can get medium-high complexity stuff right and overall it is a productivity boost.

See, that's one part I disagree about. First as a direct productivity boost; I realize you probably think it's making you more productive; but that's something you have to demonstrate, at scale. And the few genuine studies I've seen suggest the effect, if it exists, is marginal at best; at least two I remember suggest it decreases productivity (and at least one, possibly both, can't remember, said self-reporting from devs was that it increased it despite the effect).

More importantly, though: very little of the code you produce this way will be maintainable. Very little will also be secure, too (both "direct" vulns - they're not actually good at finding them, FYI - and more fundamental logic vulnerabilities - they'll never find these at all outside of trivial cases). If you want a concrete example of that, just look at the leaked Claude harness code: it's genuinely pathetic. They're vibe coding it all and it shows.

Also, I notice you say "you still need to..."; do you think this will eventually get fixed ? it's been nine years. There's a point where "it'll get better" is starting to become more of a meme than an answer. It was getting better, then a lot less, then lately almost not at all, despite greater and greater cost increases.

Shit, lately I've even started to wonder if they're even training new models anymore; I've been wondering if they're just not shipping the same with a different harness / different bullshit surrounding it. It's not like they can improve on the models themselves anyway (not without at least one, likely several, genuine fundamental breakthrough) - they're all out of human-produced data by now.

Yes it is, just yesterday I was having trouble with a script, so I googled it, the Google ai told me to use a command that doesn't exist. It just made it all up. Completely worthless.

China isn't counting the same numbers i tell you that

That's true; the very quality of results you can obtain with some recent models locally suggests the possible optimizations are huge. But it's also diminishing returns: the moat between state of the art models and one I run on a P40 locally is really small these days; the amounts they need to sink to get even slightly better results at this stage are more and more. They're text generators; yes, if you run ten of them in parallel, vote; have them cross-check their results, introduce harnesses at every steps, etc. (all examples of actual bandaids I evoked above) you can improve results. But all of this is trying to make a tech that fundamentally doesn't answer the problem answer it nevertheless. And all of it costs a lot.

And I dearly hope China is not spending too much effort on backing these domestic initiatives, because again: outside of a few limited use cases (easily identifiable, I've listed a few in another comment in this very thread: those where having no relation to the truth in a portion of the generated text is acceptable), the tech doesn't work.

It's incredibly useful for letting humans interact with a computer system in natural language and maybe you shouldn't take from that "hey we should have it do highly sensitive stuff where the slightest error could have great consequences"

There are use cases where having the computer completely invent actions or do shit randomly isn't that bad, I guess; games come to mind. But ultimately and more generally I disagree, it's shit for that too. Try one as a daily runner, just for laughs. Or just try an entire shell session where instead of typing the commands, you complete a description of what you want to do through one and then do it. It goes bad very fast, let me tell you.

And I get it: what you describe would be awesome - a SF dream. I like tech and I wish all of this would work; and like many initially I genuinely wondered as well if scaling/attention was all you needed, and had some measure of hope; quickly dashed, though.

it does a great job of summarizing reddit threads

Again, no it does not. You have no guarantee it won't pull shit out of its virtual ass; and just as crucially (and even more likely), no guarantee it won't ignore significant parts of the source material. You'll get a result that seem like it summarizes the thread, with no confidence level, and no guarantee of its reliability.