The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, "Reasoning" models, and Agentic frameworks.
ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels.
Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what's next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations.
You can try the tasks yourself here: https://arcprize.org/arc-agi/3
Here is the current leaderboard for ARC-AGI 3, using state of the art models
- OpenAI GPT-5.4 High - 0.3% success rate at $5.2K
- Google Gemini 3.1 Pro - 0.2% success rate at $2.2K
- Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K
- xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K.
(Logarithmic cost on the horizontal axis. Note that the vertical scale goes from 0% to 3% in this graph. If human scores were included, they would be at 100%, at the cost of approximately $250.)
https://arcprize.org/leaderboard
Technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf
In order for an environment to be included in ARC-AGI-3, it needs to pass the minimum “easy for humans” threshold. Each environment was attempted by 10 people. Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Many environments were solved by six or more people. As a reminder, an environment is considered solved only if the test taker was able to complete all levels, upon seeing the environment for the very first time. As such, all ARC-AGI-3 environments are verified to be 100% solvable by humans with no prior task-specific training
As another technologist, I have to remind everyone that unless you subscribe to some rather fringe theories, humans are also based on standard physics.
Which is math. All the way down.
As a philosopher, I have to remind you that humans invented math and physics to model reality.
Humans are not based on physics or math. That would be like saying the earth is based on a globe.
As a mathematician, it should be noted that the mathematics of physics aren’t laws of the universe, they are models of the laws of the universe. They’re useful for understanding and predicting, but are purely descriptive, not prescriptive. And as they say, all models are wrong, but some are useful
As a random person on the Internet I don't actually have anything to add but felt it would be nice to jump in.
That's true, but that doesn't contradict the above comment. Unless you believe in something like a spirit or soul, you must concede that human intelligence ultimately arises from physical matter (whatever your model of physics is). From what we know of science right now, there are no direct reasons for thinking that true intelligence or even consciousness is limited to biological organisms based on carbon and could not arise in silicon.
My point was more so that the argument that humans can be modeled with math & physics implies that LLMs are/could become intelligent, conscious things, since they’re also based on math, is nonsense. These are statistical prediction algorithms; they work nothing like a nervous system or a conscious living being. They can be impressive in narrow use cases, like all ML, but they cannot actually learn or perform novel tasks. I don’t think this rules out the possibility of creating some sort of true artificial intelligence, but the current approaches are structurally unable to ever get there, and the conversation above makes really weak points to the contrary. But this was too many words so I figured my other approach was better for brevity lol
I generally agree, but I kind of wonder whether something like an advanced LLM has a place as a component of an artificial "brain". We have a language-focused area in our brain, but we have lots of other components of the brain that does all kinds of other things too. Perhaps we're "just" missing those other things.
I agree, the maths argument is not a good one. While a neural network is perhaps closer to what a brain is than just a CPU (or a clock, as it was compared to in he olden days), it would be a very big mistake to equate the two.
What maths do our memories follow? What about consciousness?
We're not actually individuals; we're massive colonies of cells that work in concert. Memories and consciousness are both products of chemical interactions that happen between the cells, and the cells themselves are conglomerates of subatomic particles. Everything about us is determined by particle physics, which can be expressed and predicted mathematically.
The hubris of modern science and medicine is thinking that we know everything about our biology. I contend that we don't. Can you tell me what's in my gut microbiome?
No one said we know everything about our biology. But we're made up of particles, just like everything else. We don't fully understand those particles either, but it doesn't make them not real or not subject to the rules reality seems to follow.
They actually make little pills you can swallow to take samples at certain locations a long your digestion, so I suppose I could, given the knowledge and resources. Surgical sampling is also possible.
But I don't see why it matters because all of the bacteria and archaea present in the body are made up of subatomic particles.
Obligatory xkcd... we're just meatbags somewhere to the left Purity
On a more serious note, there's plenty to explore there and there are some potentially interesting links to quantum physics and stuff in our brain, as well as how certain drugs can completely disrupt our consciousness (ever had an operation?) and how it could link up. But there is obviously no definitive answer.
At best consciousness is whatever flavour of philosophical interpretation/explanation you like at any given time.
Philosopher: looks at the mathematician...
Consciousness (the fact of experience) doesn't necessarily need to be linked to intelligence. It might be but it doesn't have to. An LLM is almost definitely more intelligent than an insect but it most likely is like nothing to be an LLM but it probably is like something to be an insect.
Isn’t it kind of eery that you can only suppose it must be “like something” to be an insect, from the very precise bias of being human? We’re projecting the idea that “it’s like something to be something [as a human]” only the experience of other things.
How would we describe what it’s like? Would something poetic suffice, such as “it’s like being a leaf in the wind, and with weak preference of where you blow but no memory of where you’ve been.” … but, all of that is human concepts, human experience decomposed into a subset of more human experiences (really weird, the recursive nature of experience and concepts).
I think the idea of “what it’s like…” has some interesting flaws when applied to nonhumans. It kind of presupposes that insects are lesser, in a way. As though we can conceptualize what it’s kind to be them, merely by understanding a stricter subset of what it’s like to be human.
I can only suppose that of other people as well. There's no way to measure consciousness. The only evidence of its existence is the fact that it feels like something to be me from my subjective perspective. Other humans behave the way I do so I assume they're probably having similar experiences but I have no idea what it's like to be a bat for example.
However, answering the question "what it's like to be" is not relevant here. What's relevant is that existence has qualia at all.
Does existence “have qualia?” That treats qualia almost like it’s ontological, if I’m interpreting you correctly. Yet, qualia can only exist from the perspective of a being with the capacity to model a (seemingly external) world via said qualia. There is no magic qualia sauce we can embed inside something.
Qualia, I think, is a process of information reduction… but also it’s a flavor of information interrogation. Because, reducing electromagnetic radiation to “visual perception” happens inside light sensors too — albeit without counting as “qualia.”
What would you say counts as “qualia?” Or rather, what are its dependencies?
It's the fact of subjective experience - the warmth of a campfire, the bitterness of lemon, the greenness of green. We're essentially talking about consciousness here. The fact that there's something it is like to be.
While nobody knows what consciousness is or how it comes about, what I mean by it is best captured by the philosopher Thomas Nagel in his aforementioned essay "What Is It Like to Be a Bat?"
The premise still strikes me as odd. How can we know it’s like anything to be anything, if we can not know what it’s like to be anything else? Coming from a premise that, to truly understand anything, you must also understand what it is not.
Is it really fair to presume, from our biased perspective where “likeness” is an abstract quality of “being,” that everything ought have a manner of which it is like to be?
What about the totality of the universe, to include all its embedded agents. What would that be like? Would an ever small portion of that likeness include precisely what it’s like to be me?
Do you think it would be possible to qualitatively describe and differentiate between two distinct phenomenologies, one day? Not just behaviorally, but to actually differentiate between their internal processes — what it’s like to be them?
And what might it be like to be a whirlpool, lightning, or even an entire ecosystem? Would that strictly be as ludicrous as asking “what might it be like to be a rock,” or is there something else to be said given whirlpools, lightning, and ecosystems are more-or-less events rather than objects?
I don’t disagree with the argument you shared… I think there’s an obvious difference between what it’s like to be a bat versus a human, but I also feel like we’re missing something important that clearer terminology could work out.
Because it undeniably feels like something to be in this very moment from the perspective of my subjective experience. In fact, I'd even go as far as to claim that it's the only thing in the entire universe that cannot be an illusion. I could be a mind living in a simulated universe on an alien supercomputer, with every person I've ever interacted with just being a convincing AI, or I could be a Boltzmann brain - but what remains true despite all that is that something seems to be happening.
I think the closest we can get to true unconsciousness that you can still come back from is general anesthesia. It's nothing like sleep. It's like that period of time doesn't even exist. It's like the time before you were born.
Descartes would too, I doubt I think, therefore I think…
What about a third-term fetus (3tf)? To me, I think it’s obvious and intuitive that a 3tf has an experience. This is as obvious and intuitive to me as a rock not having an experience. Yet, there’s also something similar about them which isn’t made obvious by those two points; both a rock and a 3tf can (perhaps) be said to be sharing the same kind of experience.
A 3tf would have experience that doesn’t contain meta-cognitive function (e.g., self awareness). That said, the experience of a 3tf can (again, perhaps) be modeled simply as a function like
experience=fn(qualia)wherequalia=nervous-system-capacity + stimuli. Effectively, it’s the structure of the being (the nervous system) being exposed to the world (stimuli). Rocks can be said to be the same, with a very “poorly functioning” nervous system. You can model a rock’s experience too, givenqualia=0for the rock.From this framing, I think it starts to become more clear that we’re discussing a kind of physical process. Qualia starts to look like a name we’ve given to that particular process, and less like it’s some elusive thing which evades scientific understanding.
I’m partially not convinced that it feels like anything to feel something, though. I mean, I do understand feeling happy, angry, sad, even sublime. But these are categories of feeling that my very own internal processes have conjured up. How can I be sure that “feeling” something isn’t similar to the kind of illusion a heap of cells can evolutionarily succumb to when it begins to regard “itself” as separate from its environment? You wouldn’t use the sense of “self” to justify “I exist as myself, for fact.” So why would our experience of phenomenology be different?
I can only assume a fetus likely has some faint level of consciousness while a rock has none, but again, this is all just speculation. I couldn't possibly know for sure. Consciousness and qualia are entirely subjective experiences. There's no evidence of them in the universe outside our own direct experience of it. If I wasn't conscious myself, I wouldn't even have any idea that such a thing exists.
What it is - I have no idea. If I had to bet, I'd say it's an emergent feature of a sufficient level of information processing and therefore a physical process, but that's just my speculation. Nobody actually knows.
Illusions are experiences. That's why I say consciousness cannot be an illusion: the very fact that you're experiencing the illusion proves that the space where that illusion appears exists - and that space is consciousness.
I'm also not talking about any human concepts we layer on top of feelings, nor the thoughts we have about those feelings. I'm only talking about the raw sensation itself that our brain then interprets as hot, wet, green, bitter, and so on. The fact of experience itself. If I were to switch places with a bat it would most likely still feel like something to be that bat but if I were to switch places with a rock it would be equivalent to dying. The lights would go out because there's no consciousness. It's like nothing to be a rock (probably).
The sense of self implies some kind of center of consciousness or thinker of thoughts. I don't buy that. Thoughts just appear - nobody is authoring them. I speak of "me" or "I" as a being in the universe, but that's just because it's the only way I know how to refer to these things. I don't know how accurate my view of the universe really is. Like I said: I could just be a mind living in a simulated universe. I don't think I am, but it would be perfectly compatible with my experience.
Preface: I agree with pretty much all of what you said.
The other day, though, I had washed my hands. I had to be careful because one of my fingers can’t get wet due to an injury. While carefully washing my hand, I noticed that I was “experiencing” wetness all over my hand — to include on portions that were completely dry. I found this rather interesting, that I was experiencing something which I knew to be factually false. I wonder if the difference between processing and experiencing could have something to do with that.
I think a lot about this stuff.
I could go on and on. Sometimes I think it’s ridiculous that I can’t so easily find existing material on this stuff.
You seem to be well versed on this topic. Can I ask what your study materials have been?
There's also the concept of consciousness without memory. What's that like? Being able to experience the current moment but having no memory of any past experiences - including your experience one second ago.
Or here's a scary thought: what if general anesthesia doesn't actually switch off consciousness but simply blocks new memories from forming? You could experience the full horror of being awake during surgery but remember none of it. From the perspective of "now," that would be functionally the same as never having experienced it at all.
Then there are those extremely weird recordings from split-brain studies. Back when grand mal seizures were treated by cutting the corpus callosum - the bridge between the two brain hemispheres - to stop the "storm" from spreading. On the surface these patients seemed completely normal after the operation, but some really strange stuff shows up when you start testing them properly.
There's a way you can communicate with each hemisphere independently without the other one knowing. The left hemisphere controls the right side of the body, the right hemisphere the left side. You can flash text on the left side of the visual field (which only the right, non-verbal hemisphere sees), then hand them a pen and let the left hand (controlled by the right hemisphere) answer questions by writing. Turns out that the two halves often don't agree on things. Ask the right hemisphere what it wants to do for a living and you'll get a different answer than what the left hemisphere says out loud. Or you can give the right hemisphere a task - "go get a glass of water" - and when you ask the left hemisphere why it did that, it just makes up an explanation. "I was thirsty," it'll say, even though the researchers know that's not true. It genuinely seems like there are two separate consciousnesses running in the same brain at the same time. The big question is: were they there all along, or does the second one only emerge once the connection between them is cut?
And yeah, this is all stuff I've absorbed from podcasts covering these topics - mostly from Sam Harris. I'm just naturally really curious about the human mind, and I'm pretty experienced with meditation as well, so I probably pay attention to my day-to-day conscious experiences about 1% more than the average person. I'm however not in any way expert on this. It's not even remotely related to what I do for living.
The taxonomy/topology of concepts are especially fun to think about. Alike experience, because it’s similarly something that is unique to consciousness. To conceptualize.