TechTakes

luis villa, who as a lawyer really should have known better, being self-reportedly a friend of behdad and a confabulation/war machine promoter decided to come to rescue, calling the above (a) attack, and (b) slander.

how… what… how… why… why would you think…

scott jumpscare

yeah it's bad

otoh awareness I think is spreading

swedish public broadcasting has regular "spot the fake" pieces on their website

I think giving a sensationalist bit of news 6 hours to "mature" is a good idea before amplifying.

animals are like sentient beings y'know, a clanker is a… matrix or a bunch of matrices or something

Even in Knuth's account it sounds like the LLM contribution was less in solving the problem and more in throwing out random BS that looked vaguely like different techniques were being applied until it spat out something that Knuth and his collaborator were able to recognize as a promising avenue for actual work.

His bud Filip Stappers rolled in to help solve an open digraph problem Knuth was working on. Stappers fed the decomposition problem to Claude Opus 4.6 cold. Claude ran 31 explorations over about an hour: brute force (too slow), serpentine patterns, fiber decompositions, simulated annealing. At exploration 25 it told itself “SA can find solutions but cannot give a general construction. Need pure math.” At exploration 30 it noticed a structural pattern in an earlier solution. Exploration 31 produced a working construction.

I am not a mathematician or computer scientist and so will not claim to know exactly what this is describing and how it compares to the normal process for investigating this kind of problem. However, the fact that it produced 4 approaches over 31 attempts seems more consistent with randomly throwing out something that looks like a solution rather than actually thinking through the process of each one. In a creative exploration like this where you expect most approaches to be dead ends rather than produce a working structure maybe the LLM is providing something valuable by generating vaguely work-shaped outputs that can inspire an actual mind to create the actual answer.

Filip had to restart the session after random errors, had to keep reminding Claude to document its progress. The solution only covers one type of solution, when Claude tried to continue another way, it “seemed to get stuck” and eventually couldn’t run its own programs correctly.

The idea that it's ultimately spitting out random answer-shaped nonsense also follows from the amount of babysitting that was required from Filip to keep it actually producing anything useful. I don't doubt that it's more efficient than I would be at producing random sequences of work-shaped slop and redirecting or retrying in response to a new "please actually do this" prompt, but of the two of us only one is demonstrating actual intelligence and moving towards being able to work independently. Compared to an undergrad or myself I don't doubt that Claude has a faster iteration time for each of those attempts, but that's not even in the same zip code as actually thinking through the problem, and if anything serves as a strong counterexample to the doomer critihype about the expanding capabilities of these systems. This kind of high-level academic work may be a case where this kind of random slop is actually useful, but that's an incredibly niche area and does not do nearly as much as Knuth seems to think it does in terms of justifying the incredible cost of these systems. If anything the narrative that "AI solved the problem" is giving Anthropic credit for the work that Knuth and Stapprrs were putting into actually sifting through the stream of slop identifying anything useful. Maybe babysitting the slop sluice is more satisfying or faster than going down every blind alley on your own, but you're still the one sitting in the river with a pan, and pretending the river is somehow pulling the gold out of itself is just damn foolish.

Baldur Bjarnason's essay remains evergreen.

Consider homeopathy. You might hear a friend talk about “water memory”, citing all sorts of scientific-sounding evidence. So, the next time you have a cold you try it.

And you feel better. It even feels like you got better faster, although you can’t prove it because you generally don’t document these things down to the hour.

“Maybe there is something to it.”

Something seemingly working is not evidence of it working.

• Were you doing something else at the time which might have helped your body fight the cold?

• Would your recovery have been any different had you not taken the homeopathic “remedy”?

• Did your choosing of homeopathy over established medicine expose you to risks you weren’t aware of?

Even when looking at Knuth's account of what happened, you can already tell that the AI is receiving far more credit than what it actually did. There is something about a nondeterministic slot machine that makes it feel far more miraculous when it succeeds, while reliable tools that always do their job are boring and stupid. The downsides of the slot machine never register in comparison to the rewards. Does it feel so miraculous when I get an idea after experimenting in Mathematica?

I feel like math research is particularly susceptible to this, because it is the default that almost all of one's attempts do not succeed. So what if most of the AI's attempts do not succeed? But if it is to be evaluated as a tool, we have to check if the benefits outweigh the costs. Did it give me more productive ideas, or did it actually waste more of my time leading me down blind alleys? More importantly, is the cognitive decline caused by relying on slot machines going to destroy my progress in the long term? I don't think anyone is going to do proper experiments for this in math research, but we have already seen this story play out in software. So many people were impressed by superficial performances, and now we are seeing the dumpster fire of bloat, bugs, and security holes. No, I don't think I want that.

And then there is the narrative of not evaluating AI as an objective tool based on what it can actually do, but instead as a tidal wave of Unending Progress that will one day sweep away those elitists with actual skills. Random lemmas today mean the Millennium Prize problems tomorrow! This is where the AI hype comes from, and why people avoid, say, comparing AI with Mathematica. To them I say good luck. We have dumped hundreds of billions of dollars into this, and there are only so many more hundreds of billions of dollars left. Were these small positive results (and significant negatives) worth hundreds of billions of dollars, or perhaps were there better things that these resources could have been used for?

ooh gooods nooo now all the Claude slurpers are going to refer to this forever as definitive proof of how legitimately useful LLMs have got, it "solved" a math problem for Donald Knuth! :<

oh hey I remember reading that Donald Knuth paper earlier today, when it got posted by an AI youtube channel as 'proof' AI is on the path to AGI

Istg this has come up before, i am just too lazy to prove it. Still. Why would anyone want this

It's curious how, in terms of utilitarianism, the 2014 post has people doing arithmetic to estimate QALYs but the 2018 post is more of a handwave where Scoot repeats the 2014 numbers verbatim. Advocates of decriminalization and legalization have long argued that the QALYs saved by releasing people from prison and no longer sentencing them (easily 20+ QALYs/person) and not arresting people for possession in the first place (0.5 QALYs/person-arrest) are significant to society at large, even if there were quantifiable health risks.

TBH I think that Scoot got a bit of a tough surprise when data actually came in on cannabis usage; it's now accepted cannabis lore that cannabis can cause onset of e.g. schizophrenia, at a rate of something like 1 in 2000 users, but the numbers on causing cancer never materialized. Meanwhile the case studies treating e.g. epilepsy have multiplied to the point where, again, it's now accepted lore that some epileptics find relief by using products made from high-CBD strains.

Choice sneer from the second post, from somebody with an extremely-relevant Moray avatar:

Yeah but you know what would achieve better results? Criminalizing driving.

Edit: grammar and also the extremely-relevant link. Pass the Moray, please~

while OpenAI deserves every bit of flack they get, it's comical to see people who criticise OpenAI for creating a 'war machine' turn around and praise Anthropic when they were-by their own admission no less!-the first people to start using AI for military purposes

From the comments:

Today I got yet another AI huckster email offering to promote my book, but that book turned out to be AI slop published under my name on Amazon. (I have contacted Amazon.) The AIs are eating their young. This would be funny if it wasn’t really happening.