AI - Artificial intelligence

This is, well, just dumb. It doesn't seriously consider the possibility that near-future "seemingly sentient" AI is, in fact, actually sentient. There are indeed many reasons why it would be bad if a non-sentient AI were to be treated as sentient, but simply assuming that AI is definitely not sentient and treating any belief that it is as a sort of delusion isn't a solution to that problem. Maybe the author is religious (or rejects materialism for some other reason) or maybe he thinks that the Church-Turing thesis is false and the brain is doing something that a Turing machine can fundamentally never do, but if so he should come out and say it. Otherwise he's stuck with human consciousness as a process of computation and that's not a foundation that can support his argument.

And his mention of how a simulation of a thing is not in fact that thing is a pet peeve of mine. (I do computer modeling.) No one thinks that a simulation of a storm is a storm in the physical world - that's a strawman. What I for one think is that something doesn't have to exist in the physical world (as more than data in a computer) in order to be in its own way real. A simulation of a storm is not a storm, but it is something in the same way that a human being is more than just some atoms despite being made entirely of atoms. Computation is both real and substrate-independent.

My suspicion is that this essay is marketing. It would be bad for Microsoft if people thought its product could be sentient, so a Microsoft employee is trying to convince people that it can't. I'm not saying that he's deliberately being disingenuous, but he wouldn't have his job if he was going around saying "Maybe someday soon Copilot will have human rights."