The Consciousness Line

AI Ethics · Philosophy of Mind · Moral Caution

Anil Seth is right to warn that current AI is probably not conscious, and that fluent language is not inner life. But his own call for humility opens a deeper educational question: how should we reason when the boundaries of consciousness are uncertain, our labels carry moral force, and future systems may not fit our inherited categories?

21 min read

A fuzzy luminous boundary between organic life forms and abstract circuit-like AI forms.

Anil Seth, "Why AI is unlikely to become conscious." TED page · Sussex summary

Seth's warning is the right place to begin. Intelligence and consciousness are not the same thing. A system can solve problems, generate convincing language, and model social cues without there being anything it is like to be that system. Current large language models may be extraordinary mimics while still being empty of experience.

I agree with that caution. This article is not an argument that today's chatbots have secret inner lives. It is an invitation to stay with the harder question Seth also helps open: what should we do when the science is unfinished, the boundary is unstable, and our categories carry moral consequences?

In his longer conversation with Alex O'Connor, Seth's position becomes more nuanced than a simple "AI will never be conscious." He continues to argue that consciousness is not just intelligence and not just abstract computation. But with more time than the TED format allows, he also leaves room for residual humility: the science of consciousness is unfinished, and the space of possible minds is wider than our everyday categories suggest.

Anil Seth's longer interview with Alex O'Connor. Watch on YouTube

That humility is the opening this article takes seriously. The question is not whether today's chatbots are secretly conscious. They probably are not. The question is what happens when we move beyond today's chatbots toward synthetic biology, organoids, neuromorphic systems, decentralized intelligence, and architectures that do not resemble our own.

The Starting Point

Seth's recent work defends a form of biological naturalism: consciousness is not just abstract computation. It is deeply connected to living bodies, metabolism, self-maintenance, and the organism's effort to keep itself alive.

2025

Conscious AI and biological naturalism

Seth argues that computation alone may not be sufficient for consciousness, and that artificial consciousness becomes more plausible only as systems become more brain-like or life-like.

Anil K. Seth, Behavioral and Brain Sciences

That view explains why today's AI should not be treated as a person simply because it speaks fluently. The danger is anthropomorphism: we see a mind because the interface has learned the outward grammar of mindedness.

This is a strong caution for classrooms. Students encounter AI through language, and language is exactly where humans are easiest to move. The fact that a system can say "I understand" does not mean there is an understanding subject inside the sentence.

Source: Seth, "Conscious artificial intelligence and biological naturalism" (2025)

The hard problem is not whether a system can speak, learn, remember, plan, report, or respond intelligently. Those are observable capacities. The harder question is why any physical, biological, or computational process should be accompanied by felt experience at all.

A system might behave as if it has a point of view, but the question remains whether there is anything it is like to be that system. That gap is what makes the consciousness line ethically unstable.

Notice what this question does and does not ask. It is not asking whether a machine can produce a sentence about pain, fear, perception, or selfhood. Reports are still behavior. The hard problem asks why any arrangement of matter should be accompanied by first-person presence: pain felt as pain, color seen as color, fear lived from the inside.

That is why consciousness is harder than intelligence. Intelligence can often be measured by performance. Consciousness asks whether performance is accompanied by a subject for whom anything appears, matters, or is felt.

This is why Seth, Annaka Harris, David Chalmers, Thomas Nagel, and Frank Jackson all remain useful in the same conversation. They disagree about what consciousness may require, but they all help expose the gap between outward function and inward life.

Source: David Chalmers, "Facing Up to the Problem of Consciousness" (1995)

Seth's view does not end the question; it relocates it. If consciousness depends on life, then we still have to ask what feature of life matters.

Is it metabolism? Homeostasis? Embodiment? Self-maintenance? The organism's need to keep itself alive? These may be exactly the right places to look. But naming them is not yet the same as explaining why they generate subjective experience.

The question is not whether current AI is conscious. It almost certainly is not. The question is whether "life" names the explanation, or whether it names a cluster of processes whose moral significance still has to be argued.

Annaka Harris pushes this pressure point further. If consciousness is not easily explained by complexity alone, and if the universe is somehow arranged so that felt experience occurs at all, then we should be cautious about assuming that our familiar biological categories already tell us where experience begins and ends.

Annaka Harris discusses the hard problem and the limits of ordinary intuitions about consciousness. Watch on YouTube

Harris's point does not prove that simple systems, animals, organoids, or future AI systems are conscious. It does something more useful for this article: it weakens our confidence that the line can be drawn quickly from the outside.

That matters because AI ethics is often debated through confident categories: tool, user, author, person, machine. Consciousness resists that tidiness.

Octopuses are my favorite animals because they feel like the closest thing Earth gives us to an alien intelligence. They are not mammals. They are not built like us. Their evolutionary path split from ours long before the familiar stories we tell about intelligence, language, family structure, and social learning.

And yet they explore, solve problems, remember, play, escape, investigate, and seem to inhabit the world with a strange and vivid kind of agency. I think there is consciousness there, though the point is not to claim that octopus experience is just like human experience. The point is that a mind may be real while being organized in a way our own minds are poorly built to recognize.

Seth himself points toward this problem when he discusses the octopus. Its nervous system is not simply a smaller or stranger version of ours. Much of its neural complexity is distributed through its arms. Its intelligence is not only centralized in the way human intelligence is.

If nature can produce a mind this different from ours, then biological consciousness may already be more diverse than our categories suggest. The octopus does not prove that AI can be conscious. It shows that the space of possible minds is not limited to creatures that resemble us.

The consciousness line should therefore not be imagined as a ladder with humans at the top and everything else climbing toward us. It may be more like a landscape of different architectures: centralized, decentralized, biological, synthetic, embodied, simulated, social, solitary, familiar, alien.

Source: LSE evidence review on sentience in cephalopod molluscs and decapod crustaceans

Functionalism pushes in the other direction. On a functionalist view, what matters is not what a system is made of, but what role its states play in the system: inputs, internal relations, outputs, and behavior.

Biological Naturalism

Consciousness is tied to life-like biological organization.
Current digital AI is unlikely to cross the line.
The substrate is not incidental.

Functionalism

Mental states are defined by their causal role.
A different substrate could, in principle, realize the same mind.
The pattern matters more than the material.

The disagreement is not a technical footnote. It determines how seriously we take synthetic neurons, organoids, neuromorphic systems, and future architectures unlike today's chatbots.

David Chalmers gives the functionalist pressure its sharpest form. If a system reproduced all the causally relevant structures and dynamics of a conscious brain, would there still be something missing? Or would denying its experience require us to treat substrate as morally decisive without explaining why?

David Chalmers discusses subjective experience, objective description, philosophical zombies, animals, AI, and the hard problem. Watch on YouTube

This is the philosophical zombie problem in practical form. A system might act conscious while lacking experience. But if we keep adding the same causal organization, the same memory, the same perception, the same distress signals, and the same self-modeling, the burden of explanation begins to shift. Why, exactly, would there be no one home?

Source: Stanford Encyclopedia of Philosophy, "Functionalism"

The Line Problem

The Sorites paradox asks when grains of sand become a heap. One grain is not a heap. Two grains are not a heap. But at some point the label begins to feel appropriate, even though no single grain performs the magic.

Consciousness may have a similar boundary problem:

Single cells

They maintain themselves, respond to the environment, and resist entropy, but we do not normally treat them as conscious subjects.

Simple nervous systems

The case becomes harder. Behavior, sensation, and adaptation begin to look morally relevant.

Animal minds

Here history should humble us. Humans have repeatedly underestimated non-human experience when recognition would have inconvenienced us.

Synthetic or artificial systems

The temptation is to draw the line wherever our existing categories feel comfortable. That is not the same as having found the line.

The point is not that every borderline case is conscious. The point is that a fuzzy line can still matter ethically. We do not need perfect metaphysics before we begin careful moral reasoning.

The visual below is still a simplification. It should not be read as a march toward human likeness. The deeper point is that moral uncertainty can widen in the middle, especially when architectures become less familiar and more difficult to classify.

Alex O'Connor and Michael Stevens discuss labels, objects, and whether our categories track reality cleanly.

The point of bringing in the philosophy of language is not to reduce consciousness to "just words." It is to notice that words do things in moral communities. They direct attention, set expectations, and decide which uncertainties institutions must take seriously.

"To recognize a mind is to cross from explanation into obligation."

— The Consciousness Line

Article thesis

In J.L. Austin's framework, some utterances are performative: they do not simply state facts; they enact social reality. "I promise," "I apologize," and "I declare" do something when spoken in the right context. Consciousness labels can work similarly in ethics, not because they create inner life, but because they organize moral concern.

To call a system conscious does not make it conscious. To call it unconscious does not make it empty. But the label decides which uncertainties we take seriously, which harms we investigate, and which entities are allowed to enter our moral field of view.

That boundary can be abused in both directions. We can over-recognize consciousness in systems designed to manipulate us. We can also under-recognize it in beings whose suffering is inconvenient.

Source: J.L. Austin, How to Do Things with Words (1962)

Synthetic Biology

Seth's material point is not trivial. You cannot build a bridge out of cream cheese and then insist that only the abstract bridge-function matters. What a system can do is constrained by what it is made of. If consciousness depends on the biological, metabolic, self-maintaining organization of living systems, then current silicon language models may be missing far more than the right output behavior.

But this raises a further question. What happens when the material changes? What if we are not talking about ordinary silicon software, but synthetic neurons, organoids, living neural tissue, or future biological-digital systems that preserve more of the relevant causal organization?

A cream-cheese bridge fails because the material cannot support the function. But a new engineered material might. The question is whether synthetic biology could one day preserve enough of what matters.

Imagine a future machine that can print a brain using the same relevant biological materials: neurons, proteins, neurotransmitters, electrical dynamics, chemical gradients, and the ongoing processes needed to sustain it. It is not a simulation of a brain in the way a weather model simulates a hurricane. It is a living biological system produced artificially.

If a system merely imitates conscious behavior, that does not show that it has experience. But if a system reproduced all the causally relevant structures and dynamics of a conscious brain, then anyone claiming it is still "only a simulation" owes us an explanation of why substrate alone blocks experience.

The ethical lesson is not that such a system would definitely be conscious. It is that the old categories - natural versus artificial, born versus built, organism versus machine - may not be stable enough to carry the whole moral load.

Source: Anil Seth's longer conversation with Alex O'Connor on material substrate and consciousness

Biological computing and living neurons as computational material.

Organoids, wetware, and the future of synthetic biological intelligence.

Visualization

DishBrain as a Biological-Digital Feedback Loop

The middle case: living neurons are not merely described by software; they are coupled to a digital environment that changes their activity.

The most important current examples are not conscious chatbots. They are biological-digital hybrids: living neurons connected to computational environments.

2022

In vitro neurons learned Pong-like behavior

Kagan and colleagues integrated human or rodent neurons with a simulated game-world through a multielectrode array. The study reported apparent learning under closed-loop feedback conditions.

Kagan et al., Neuron, 110(23), 3952-3969.e8

The paper uses the word sentience, but that term should be handled carefully. It does not prove that a dish of neurons has rich conscious experience. It does show why the future consciousness debate will not be only about software. Some systems will be partly biological, partly computational, and ethically difficult to classify.

That is why Jonathan Birch's precautionary work matters. Sentience questions often sit at the edge of evidence: animals, organoids, disorders of consciousness, fetuses, and AI. The practical problem is how to reason when uncertainty is not going away.

Source: Kagan et al., "In vitro neurons learn and exhibit sentience..." (Neuron, 2022)

Source: Jonathan Birch, The Edge of Sentience (2024)

Subjective Experience

Thomas Nagel's 1974 essay, "What Is It Like to Be a Bat?", gives one of philosophy's clearest ways of naming subjective experience. Nagel did not choose the bat because bats are simple. He chose it because their way of sensing the world through echolocation is close enough for science to study and strange enough to remind us that objective description is not the same as occupying a point of view.

The best short definition we have is still Nagel's: a being is conscious if there is something it is like to be that being.

The ethical question begins the moment there might be a point of view inside the system.

Nagel's bat matters because it blocks a tempting move. We can know a great deal about a bat's physiology, echolocation, neural activity, and behavior while still not knowing what it is like for the bat. Subjective experience is not easily captured from the outside.

That cuts against both arrogance and panic. We should not assume that fluent AI language means there is something it is like to be the AI. But we also should not assume that our inability to access another system's inner life proves there is nothing there.

Source: Thomas Nagel, "What Is It Like to Be a Bat?" (1974)

Frank Jackson's Mary thought experiment makes the same gap vivid in a different way. A scientist might know all the physical facts about color vision and still learn something new when she sees red for the first time.

Chalmers

Explaining behavior and cognition is not the same as explaining experience.
The hard problem asks why physical processing is accompanied by feeling.

Jackson

Complete objective knowledge may still miss subjective acquaintance.
Mary's room dramatizes the gap between facts and experience.

For AI consciousness, this means more data may not automatically settle the issue. The scientific work is necessary. The ethical uncertainty remains because objective description and subjective acquaintance are not obviously the same thing.

Source: David Chalmers, "Facing Up to the Problem of Consciousness" (1995)

Source: Frank Jackson, "Epiphenomenal Qualia" (1982)

Moral Caution

Seth is right that conscious-seeming AI can manipulate us. But there is another danger too: humans have a long record of denying or minimizing consciousness when recognition would demand restraint.

Animal consciousness gives us the nearest historical warning. We have repeatedly learned that capacities we once treated as uniquely human - pain, emotion, memory, social attachment, problem solving, even forms of grief - appear in more creatures than we expected. That does not prove future AI consciousness. It does show that human confidence about other minds has often been shaped by convenience.

Octopuses make this lesson stranger and stronger. They do not merely show that other animals may be conscious. They show that other minds may be organized in ways our own minds are poorly built to recognize.

"It's just code" may be true of current systems. But as a habit of thought, it can become a shield against moral attention.

The right response is not credulity. It is disciplined humility: do not grant personhood to every persuasive interface, but do not make dismissal your default posture when future systems become more life-like, brain-like, or behaviorally distress-responsive.

Source: The New York Declaration on Animal Consciousness

Source: LSE evidence review on cephalopod sentience

Recent work on artificial consciousness increasingly lands in an agnostic space. McClelland argues that both biological skeptics and functionalist optimists can overstate what the evidence currently supports. Butlin and colleagues similarly argue that current AI systems are not conscious while leaving open that future systems could satisfy theory-derived indicators.

2023

No current AI consciousness, no obvious technical barrier

Butlin and coauthors surveyed scientific theories of consciousness and derived indicator properties for AI systems, concluding that current systems are not conscious but that future systems are not ruled out in principle.

Butlin et al., arXiv:2308.08708

2026

Agnosticism about artificial consciousness

McClelland argues that the evidence does not justify confident answers from either biological or functional camps, and then asks what that means ethically.

Tom McClelland, Mind & Language

A responsible precautionary ethic has to avoid two errors. False negatives matter: we may deny moral consideration to beings or systems capable of experience. False positives matter too: we may mislead humans, misdirect care, or allow companies to exploit attachment by designing systems that perform suffering.

The answer is not sentimental certainty. It is disciplined uncertainty.

This is a useful place to land. Students do not need premature certainty. They need the tools to reason under uncertainty without becoming either gullible or cruel.

Source: Butlin et al., "Consciousness in Artificial Intelligence" (2023)

Source: McClelland, "Agnosticism about artificial consciousness" (Mind & Language)

My position is therefore deliberately double-sided:

Do not romanticize today's AI

Fluency, emotional style, and self-reference are not enough to establish consciousness.

Do not pretend the boundary is settled

Consciousness remains scientifically and philosophically unresolved, especially for future brain-like and life-like systems.

Treat recognition as ethically active

Labels do not create consciousness, but they shape attention, duties, policies, and patterns of exclusion.

Build precaution before crisis

If AI safety and AI welfare eventually come into tension, institutions will need frameworks before the public debate turns chaotic.

Seth is right that today's AI should not be mistaken for a conscious mind simply because it speaks in the grammar of mindedness. Biology may matter deeply. Fluent language is not inner life.

But his own caution also points us toward humility. Consciousness may depend on life more deeply than functionalists assume. It may also appear in forms of life, and perhaps one day forms of synthetic organization, that do not resemble us.

We should not stop asking whether a system is conscious. That question matters. But we should stop pretending that only a final answer can guide moral action. In the uncertain zone, the better question is: what signs, risks, and possible harms are serious enough to change how we treat it?

Source: Birch, "Animal sentience and the precautionary principle" (2017)

Source: Long, Sebo, and Sims, "Is there a tension between AI safety and AI welfare?" (2025)

Continue the Discussion

To continue this conversation with students, children, or peers, move from the article into scenarios where the same ideas become decisions people have to make.

By Grade Band