Thanks for that article, it was a very interesting read! I think we're mostly agreeing about things :) This stood out to me from there as an encapsulation of the conversation:

I don’t think LLMs will approach consciousness until they have a complex cognitive system that requires an interface to be used from within – which in turn requires top-down feedback loops and a great deal more complexity than anything in GPT4. But I agree with Will’s general point: language prediction is sufficiently challenging that complex solutions are called for, and these involve complex cognitive stratagems that go far beyond anything well described as statistics.

"Statistics" is probably an insufficient term for what these things are doing, but it's helpful to pull the conversation in that direction when a lay person using one of those things is likely to assume quite the opposite, that this really is a person in a computer with hopes and dreams. But I agree that it takes more than simply consulting a table to find the most likely next word to, to take an earlier example, write a haiku about Danny DeVito. That's synthesizing two ideas together that (I would guess) the model was trained on individually. That's very cool and deserving of admiration, and could lead to pretty incredible things. I'd expect that the task of predicting words, on its own, wouldn't be stringent enough to force a model to develop "true" intelligence, whatever that means, to succeed during training, but I suppose we'll find out, and probably sooner than we expect.