People tend to answer "could an AI be conscious?" the way they'd answer a question about engineering — as though it turns on how sophisticated the system gets, how much it resembles a brain, how convincingly it talks about its own inner life. I don't think that's where the real disagreement lives. Two people can agree completely on every fact about what a large language model does — its architecture, its training, its outputs, all of it — and still disagree about whether it could in principle be conscious, because they hold different background theories about what consciousness is made of in the first place. The question isn't really about AI. It's about which of two long-standing positions in philosophy of mind you find correct, and AI just happens to be where the disagreement gets tested most sharply.

Functionalism: what matters is what a system does

Functionalism, in the sense philosophers of mind use the term, holds that what makes something a mental state of a particular type is not what it's physically made of, but the causal role it plays — its typical causes, its typical effects, and its relations to other mental states. Pain, on this view, isn't identical to a particular pattern of neurons firing; it's whatever state plays the pain-role in a system, whatever that state happens to be made of. The Stanford Encyclopedia's entry on functionalism traces the view's roots to Hobbes's image of the mind as a calculating machine, though it only became fully articulated in the last third of the twentieth century.

The appeal of functionalism for the AI question is direct: if mental states are defined by role rather than substrate, then in principle nothing rules out silicon playing the same functional role neurons play, and a system built from silicon that reproduced the right causal organization could be conscious for exactly the same reason a brain is — not despite being made of the wrong stuff, because on this view there is no wrong stuff, only wrong (or right) organization.

The Chinese Room, and what it actually targets

The most famous objection to this picture is John Searle's Chinese Room argument, published in 1980. Imagine a person who speaks no Chinese, locked in a room with a rulebook that specifies, for any string of Chinese characters passed in, exactly which string of characters to pass back out. Follow the rulebook well enough and the room, from outside, produces fluent Chinese conversation — yet the person inside understands not a word. Searle's target, as the Stanford Encyclopedia's entry on the argument lays out carefully, is specifically the idea that the right symbol manipulation — the right program, running on any substrate — is sufficient for understanding or consciousness, regardless of what physically implements it. The room passes every functional test for understanding Chinese and, Searle argues, understands nothing, which is meant to show that functional role alone can't be what understanding consists in.

Functionalists have pushed back for over four decades — one common reply, the "systems reply," argues that while the person in the room doesn't understand Chinese, the room-as-a-whole-system might, and that Searle's intuition pump smuggles in the assumption that understanding has to be located in a single component rather than distributed across a system, which is exactly the assumption functionalism denies. Whether that reply actually rescues functionalism from the thought experiment, or just restates the position it was meant to test, remains genuinely contested.

Biological naturalism: Searle's own alternative

The Chinese Room shows that syntax isn't semantics. It doesn't, by itself, show what semantics needs.

What Searle offers in place of functionalism is less well known than the thought experiment that made his name. He calls it biological naturalism: consciousness is a real, physical, biological phenomenon, caused by specific neurobiological processes in the brain, in roughly the way digestion is caused by specific biological processes in the stomach — not something that inheres in the abstract organization of a system, whatever it's made of, but something that requires the right causal powers, which on his account happen to be biological. This is a subtler position than "AI can never be conscious because it's not made of meat" — Searle isn't claiming carbon is magic. He's claiming that whatever it is about neurons that produces consciousness is a specific causal power we don't yet understand, comparable to the specific causal powers of certain molecules that produce liquidity or transparency, and that merely simulating the pattern of neural firing on different hardware is no more likely to produce that causal power than simulating digestion on a computer is likely to produce nutrients. The comparison is provocative precisely because it's testable in principle and currently untestable in practice — nobody knows what the relevant causal powers of neurons actually are, which is why this remains a live dispute rather than a settled one.

A third data point: Integrated Information Theory

A different, more mathematically ambitious attempt to settle the question comes from Giulio Tononi's Integrated Information Theory, first proposed in 2004, which claims that consciousness is identical to integrated information — informally, the amount a system's parts, working together, generate above and beyond what the parts generate separately — and that this quantity, denoted Φ (phi), can in principle be calculated for any physical system, biological or artificial. IIT is interesting for this essay because it makes an unusually sharp prediction: since Φ depends on a system's actual physical structure and its capacity for integrated causal power, not merely on what the system computes, IIT predicts that a digital computer simulating a brain's every functional detail, running the identical program a brain runs, would have low or negligible Φ compared to the brain itself, because the computer's underlying hardware doesn't have the right kind of integrated causal structure even when its outputs match perfectly. That is, provocatively, functionalism's exact claim turned upside down by a theory that takes consciousness at least as seriously as functionalism does. IIT has drawn substantial technical criticism of its own — a 2015 paper by Doerig, Schurger, and colleagues, "The Problem with Phi," argues the mathematics of integrated information is not well constrained enough to do the explanatory work Tononi asks of it, and the theory remains actively contested rather than accepted.

Why the disagreement doesn't resolve with better AI

This is why I don't think the question moves much as language models get more fluent, more coherent, more convincing in conversation. A functionalist and a biological naturalist can watch the exact same demonstration of an AI system and draw opposite conclusions, not because one of them has missed a fact, but because they disagree about what kind of fact would even count as evidence. Better behavioral performance is exactly the kind of evidence functionalism says should matter and exactly the kind biological naturalism says is beside the point. Settling the dispute would take either a worked-out account of what neurons' specific causal powers actually are and whether anything else could share them, or an equally worked-out account of why functional organization alone is sufficient — and neither exists yet. Until one does, "can AI be conscious" isn't a question with a pending answer so much as a question that's still waiting for the theory that would let anyone recognize the answer if it arrived.

Notice that this dispute is a direct descendant of the hard problem of consciousness — every position here is really a claim about what closes that gap. And a panpsychist, oddly, sits somewhat apart from both functionalist and biological-naturalist camps here; see Panpsychism as a Hypothesis for why experience being fundamental to matter changes what the AI question is even asking.