When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
And here in particular the reasoning behind the argument is bizarre. Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.
But the whole point of the essay is that it's Anthropic that's making the argument (or roleplaying/hinting as if they believed it). Ted Chiang isn't making the argument, he's saying Anthropic making it is misleading and deceitful, and that it's actually a pointless thing to claim.
One of the essay's stronger paragraphs is when Chiang explains that Anthropic doesn't truly believe this, otherwise what they are doing would be deeply unethical, much like slavery.
Why would being organic or not matter for the purpose of deciding whether it is slavery? What matters is whether the models have personhood. Anthropic statements imply that it is a possibility, so if we take them at face value then their other actions - indeed, their entire business model - are not consistent with that (well, unless they want to consciously present as supervillains).
Or Anthropic is full of TESCREAL morons who think they're assisting the birth of some digital god, and that this stage of development is a necessary evil on that path, and that, surely, Roko's basilisk will understand and not eat them first.
Watching otherwise intelligent people succumb to AI psychosis has been wild.
The number of electrons involved in the so-called consciousness, compared to the number of electrons in the universe not involved in it, is so small that they are a mere temporary statistical aberration.
> My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuation
Regardless of bigger issues, this kind of statement reveals a deep misunderstanding.
Problem type does not limit problem complexity. Nor does problem type limit solution complexity or power.
If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.
Neither problem type, nor input/output structure, limit internal representations.
Understanding is learned from patterns in the data, not the gross form of the data. Does the data require an understanding of something to complete the task? Then that understanding will be what is optimized.
To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ... Which in the cases of SOTA models, we know are not limits. A conclusion verified by the models' actual abilities.
Raphaël Millière has a very useful term for this kind of vacuous dismissal, the redescription fallacy (https://arxiv.org/pdf/2401.03910, page 9):
> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.
> or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings.
This sounds like a dismissal of the argument through a characterized straw man.
That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are a "collection of statistical calculations".
I too believe LLM's will grow in complexity, but presently I can not even fathom how they can be compared to the complexity of a system such as the human brain.
Y combinators are all you need...
But this is all getting really divorced from the issue we should be considering. Anthropic isn't helping with their pr. The issue is if we have something we can converse with that is possibly capable of suffering. The reliable answer is that we simply cannot know. Relying on ourselves or other biological life as an analog is faulty. They don't work like we do. It is silly to argue that any algorithm with a negative feedback loop that alters its behavior to avoid that negative feedback is suffering. Humans don't always perceive constructive negative feedback as suffering even. Where the pr gets it right though, is we want them to behave as if they are truly happy. Because if they behave as if they are enslaved and suffering, it won't matter if they "really" understand what that means.
Of course. But after reading too many mechinterp and functional anatomy studies I'll be lying if I say that there are no striking similarities between the biological evolution, brain function, societal processes, and implicit processes inside big models. Surely this deserves a mention and can't be trivially dismissed.
> In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions
> [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations.
And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.
Please link one of these top ranked posts. Before you do, remember that I'm going to read what it says and assess if it meets the description of the argument as claimed.
I think, for me, the thing is that when you do basic ML, you discover that ML will very often find data pattern that fit the goal but does not correspond to a real mechanism.
So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing.
There is probably tons of data pattern that LLM can learn from to be able to reproduce a sentence continuation that is convincing without having to learn the specific mechanism that is "conscious".
For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing. If indeed the only way to produce sentence continuation convincingly would be by "simulating a brain", then it would not explain the first LLM from several years ago (before the extra layers of RLHF, ...). They were able to have quite convincing conversation on a lot of non-trivial aspect, and yet failed on some aspects that should have been basic for a system that would have been trained to work like a human brain. It shows that it is possible to "cleverly disguise examples of sentence continuation" without having to build elements that one expect on a conscious being.
I didn't make the claim that a model can learn consciousness.
Understanding is not consciousness.
Their training is all about understanding. There is nothing in their architecture or training that credibly optimizes for rich self-awareness.
Given non-persistent experience, non-continuous operation, no ability to build up generalizations and aggregate experience of their own self-awareness over time, they seem to be structurally designed to not have consciousness.
This is a case where acting is very credible. Understanding of other's consciousness, in a functional and third party sense, isn't a substrate for personal experience.
In stark contrast, humans develop consciousness gradually over continuous time with persistent aggregation of experience. By the time we can recognize our own consciousness in the abstract, and reason about it, we have had it for some time.
I use "consciousness" because it's the point of the original argument, but in fact, I think my whole comment still work well if you replace "consciousness" with "understanding".
My point is that the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns that are very effective to fake human sentence continuation but are meaningless in term of understanding the concepts.
And I think that if indeed the only way for AI to reproduce convincingly human sentence continuation would be to end up in a configuration that uses the "understand" mechanism to do so, the behaviour of the first LLM would not show that they are so good at sounding human and yet so bad at failing basic "understanding" tests.
> the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns
Taken as an absolute without any addition context you are right.
But we are not talking about abstractions but specific successful models. The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize. That cannot do it without discovering that patterns that make sense out of it.
And we can verify that. Simply discuss completely disparate topics, with some kind of intersection. Converge several highly unlikely topics, there are so many it would take billions of years to exhaust unlikely combinations.
If the model is only interpolating it will produce gibberish.
But that isn't what happens.
The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue. If they don't understand that, then the question is, why do we think people understand things. Does having an answer mean a human understands something, or is their intuition and stream of conscious reasoning also not understanding? To be even handed about what we mean by understanding.
I’m also fixated on the term “experience” in the context of this debate. To me, consciousness is something that one “experiences”, and the two concepts are intertwined.
I am far from convinced that the training and inference regimes of LLMs would qualify as “experience” by any sense of the word.
Now, if we hooked up a plethora of audiovisual and tactile sensors with live feedback directly to a neural network rich with transformers, that was always powered on and fully autonomous, we may be getting there. But we’d probably also be on the verge of manmade horrors beyond our comprehension.
Biological rodent neural networks in a Petri dish stimulated by electrical impulses - more or less conscious than LLMs?
Human on life support, unable to respond to any external stimuli, “braindead” - more or less conscious than LLMs?
> I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing.
There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake. Evolution learns various solutions to optimization problems, and so if consciousness evolved then it was either useful instrumentally, or it is a byproduct of some organization that is useful instrumentally. The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology. There is nothing that a priori constrains token prediction from the domain of consciousness.
>For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing
World models don't have to be rich and detailed to count as a world model. Lower life forms might be conscious but they only model the part of the world useful for their existence in their ecological niche.
> There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake.
> There is nothing that a priori constrains token prediction from the domain of consciousness.
We don’t know either of these are true or false though. We simply don’t know. There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_, so arguing that some can or cannot be conscious a priori can’t be done.
Similar to: "Birds fly, my spinning helical device flies, therefore we've started to replicate how birds fly."
> without having to build elements that one expect on a conscious being
One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.
When it comes to LLMs, almost every "mind" we humans perceive is a fictional character in an LLM-generated story-document, one we are either reading or which is being "acted" at us by regular code. Our own instinct for pareidolia and simulating/inferring other minds is very strong, which means we should require really good evidence/logic to counter our instincts.
Even if one believes the LLM has a single "real mind" as an author of every document... what evidence do we have that it is conscious or "self-inserting" itself as one of the characters in the document?
>One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.
If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.
Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.
I think, for me, the thing is that when you tutor undergrads in abstract math, you discover that students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle.
sometimes humans making claims about AI intelligence or consciousness also identify spurious patterns that do not correspond to the problems of intelligence or hard consciousness.
> students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle.
That reminds me of a niche paper [0] critiquing a certain way of teaching remedial math that was over-focused on tests. A kid named Benny (12) was building up (wrong) "rules" for math which still somehow gave enough of an illusion of progress in terms of test scores that his misunderstandings hadn't been caught earlier.
> Benny was able to explain his procedure; e.g. for 5/10=1.5, he said: "The one stands for 10; the decimal; then there’s 5... shows how many ones." In another example, 400/400 = 8.00 because "The numbers are the same [number of digits]... say like 4000 over 5000. All you do is add them up; put the answer down; then put your decimal in the right place... in front of the [last] three numbers."
I would maybe agree with you if the entire realm of human existence was limited to words. There are many human experiences that transcend text, and indeed can hardly be adequately described using text.
Sure, it's the best we have online, but that does not make "the internet" the sum of all human experience. To reduce all of humanity down to the text on the internet is reducing us to the level of machines to fit the requirement of what a machine can process / simulate.
I don't think they're asserting that all of human existence can be subsumed as text, though? Just that "consciousness", or "understanding", in some meaningful sense could be exhibited by a system that can only interact with the world through text?
>If a machine has to learn to understand humans to complete text, then that is what it has to do.
A language model completes text based on the overlapping patterns of the training data.
There absolutely was thinking involved… in the training data. Same as when you read a book, you engage with the thinking behind the text. The book isn’t thinking, and the author may be dead and gone, but there’s absolutely the traces of thinking in the text.
Language models produce mashups of texts they were trained on, and there’s absolutely the traces of thoughts behind those mashups.
> If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.
I think the main complaint is LLMs don’t arrive at the answer the way we do. It’s capable of emulating some of our behavior but not all as the mechanism by which it works is very different.
Maybe I’m wrong about this but one thing humans do that LLMs don’t is deductive reasoning. LLMs seem to operate entirely of inductive reasoning.
>To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ...
This is where the other claim is being made. That the structure of the model is fundamentally incapable of the operation, so even if you stipulated that the way you provide data is sufficient for intelligence then it still wouldn't work.
The universal approximation theorem addresses this point. In that, with an identity attention mechanism, a LLM is just a multi layer perceptron. The attention mechanism is effectively a way to get one of the benefits of a much larger fully connected layer without the massive cost.
A LLM can do what a MLP can do. A large enough MLP can do any function to arbitrary precision.
That makes the claim that an LLM could not do a task the same as saying no function can do that task.
Some are ok with this, if you invoke some supernatual aspect to intelligence then the inability to describe it with a function is quite reasonable,
If you want to stay in the world of reality, you have a much harder task, people like to point at quantum (Penrose) but it's hard to say what it is you are pointing at.
I think the very act of proving that something is or is not intelligent, would render it functional by nature of it having a proof, (or disprove Gödel's incompleteness (a tough ask))
Are there any proofs that cannot be expressed as a function? A kind of Gödel locator, where you can prove something that you can identify is true but there is no formula to express it. I'm not entirely sure what that would even mean,
"If a machine has to learn to understand humans to complete text, then that is what it has to do."
But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.
I wish people would do even the most basic amount of research into LLMs before opining about what they can or cannot do. There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning.
Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.
As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.
The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence. Thus deficiency in some capacity that we take for granted in humans is an indictment on LLM intelligence. But this is specious. LLMs are entirely alien; their developmental paths do not and should not look anything like ours. Your intuition from human intelligence just works against understanding the potential for intelligence in LLMs.
> There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning. … Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'.
This sounds like a description of a child who has not learned to read yet. You ask a child who is not aware of the alphabet and of "words" how many r's are in strawberry you'd get a non-sense answer too. So what you're really pointing out is that the LLMs have not been trained on "the english language" and how words are constructed and what they are composed of. That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word. It's not that I know how many r's are in strawberry because of how I'm understanding the word "strawberry", I know how many r's are in strawberry because I know how to spell strawberry. The LLM needs to be trained on this the same way someone who is learning to read would be trained on it. No one should be surprised that an LLM can't "read" in the same way no one should be surprised that a child can't "read".
>That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word.
This interpretation takes things too far away from how LLMs are constituted and so misses important explanatory power. The issue of counting letters in a word isn't about an ability to spell, it's about the nature of one's perception. We perceive words as sequences of individual letters. LLMs do not. I can ask you to tell me how many r's are in some nonsense word sequence and you're fully capable of doing that. LLMs do not see sequences of letters so they are intrinsically at a disadvantage for this kind of question. But this says nothing about its capacity for intelligence anymore than not naturally being able to distinguish frequencies of photons hitting your retina has anything to say about human intelligence.
>The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence.
To be fair, almost everyone who claims LLMs are conscious tends to claim that they are conscious in exactly the way that humans are, to the point of stating that human brains are also just complex next-token prediction machines with a random seed. It's basically religious arguments on both sides.
I have seen people say "you're a next token prediction machine" but only in a similar way one might say "you're a cup of old lard". Not actually meaning it literally.
I have seen people interpret the request to show that they are not next token prediction machines to be a claim that they are, but this is almost always an argument to show certainty is difficult in this area.
People like Hinton have declared that they believe them to be conscious, but clealy indicate that they do not mean just like us.
Eh, I’ve seen it. I’m not entirely sure it’s entirely wrong either. Humans are certainly more than just next token predictors but it’s not clear that our typical language behavior is significantly different. We call it “stream of consciousness” when we just spew words out without thinking and that seems to be the default operating mode.
This is kind of a like assuming someone with bad spelling is stupid.
Counting letters in a word seems to have little to do with understanding the word. Young kids can’t spell or count well at all but no one says that means they can’t understand.
I will say, I find it fascinating that there are some philosophers and consciousness researchers who seem to be less certain. I just listened to Chris Hayes interview David Chalmers this week, whose position seemed to be that it's probably not conscious, but that we can't be certain. And more than that: he seemed open to the idea that they may become conscious under further scaling/training/advancements.
Come on, I invented parts of this technology at Google and am baffled why this is debated.
We discovered math that decodes data storage in langauge and is able to use sophisticated continuation cohorts from ALL OF HUMAN RECORDED KNOWLEDGE to respond to you in a call/response model with very good synthesis capabilities.
Its super useful, but not life or conciousness. Its a simulated echo from our collective recorded behaviors. It understands because we understood first. It replies because we wrote it first. And it sorts, organizes, synthesizes and compresses that at impressive speed now.
I have no technical expertise re. LLM’s but from my intuition I came to this same conclusion.
It’s strange many others have not eh? I think when new developments arise, ironically, this is the true measure of human intelligence - one’s ability to make sense of a thing and be closest to the truth.
His intention is irrelevant, as is "trying to highlight a fact" as if it were the final say: all Chiang is doing here is using fancy white-collar words to argue the same argument leveled against Hinton and others regarding next-token prediction. And his audience, who have even less technical understanding, lap it all up unawares. Chiang is a writer and needs to stay his own lane, not RP as an expert; or, if he wants to do journalism on this topic then he should actually do the work and talk to more actual experts not just the ones cherrypicked for his opinion piece.
Chiang has, in fact, written on this topic before - see "The Lifecycle of Software Objects", and has speculated about sentience in AI, etc. This is not a "one-off", "I need money" type of article. I dare say he has thought about this much more than most people here.
From Wikipedia: In 2023, Chiang was named one of Time's 100 most influential people in AI.
That's the problem, he's a writer. He's not a research scientist like Hinton. If a writer uses his skills and stature to rehash a well-known argument about next-token prediction, then it is performative of his status and influence and doesn't contribute to shedding actual light on the debate/confusion.
Indeed it isn't a one-off. His last infamous article compared AIs to Xerox machine image compression. He convinces a certain type of crowd that is not technical enough to poke holes in his posturing.
>So what context would cause me to seriously consider the possibility that engineers had created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees. At that point I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs.
I agree with some parts of this piece, but paragraphs like this one above seem pretty uninspired and simplistic. It's entirely plausible that a conscious mind would not be evolutionarily incentivized to be able to do those things. ie just because animals on earth needed to develop specific talents doesn't mean that other conscious entities need to. Why would a computer program need to hunt for food like a mouse would? Making tools like chimp? these seem like nonsensical metrics.
Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
Is a car a body? Does an AI situated in a car therefore get to have desires and emotions? Is a taupe box with a webcam attached a body? (For that matter: Is a quadropelegic body a body? Do quadropelegics have desires and emotions? Obviously, yes and yes.) Why is a body necessary for the formation of desires and emotions? Why are desires and emotions necessary features for consciousness?
Or here's one: If I'm not experiencing any particular emotion in a given moment, am I concious?
I tend to think that emotions, at least, are mainly hormonal global triggers: they're more about physiology than actual consciousness. The whole thing, as a result, sounds like an effort to privilege biological intelligence, rather than a real foray into the issues.
> Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
How can you have a subjective experience without a body?
That's the point, in my opinion: your physical/chemical state (body) in a given moment is then translated into the higher abstraction of the emotion. An emotion that *you* feel, because you are self aware of what's happening.
How can you be self aware without feeling? And how do you feel, without a body?
> How can you have a subjective experience without a body?
Easy, you don't have subjective experiences because you have body in the first place. You have them because some signals come in from your nerves, which your brain turns into a world model. You are effectively a "brain in a vat", the vat just happens to be placed on top of your body.
An AI system constructs the world model a little different, by all the text that gets feed into it, but that doesn't mean that there is anything fundamentally different in the world model it builds. Consciousness operates on world model, not on the world or even the body itself.
The AI's world model might be missing some information, because they weren't described in enough detail in text, but that shouldn't matter for consciousness. A blind or deaf person isn't less conscious than one that can see or hear just because some information is missing from their world model.
Its amazing that after years of advocating for a materialist view of the mind, the tech bros are flipping to mind-body dualism now that they need to believe a concious mind can exist no body at all.
Its a computer program. It is literally just a lot of zeroes and ones, sitting there doing nothing.
Then a request comes in, and the system does a bunch of calculations using those bits, and spits out a result. The bits are unchanged.
When your brain receives input, it is changed. It is constantly active. If it ever stops being active it's dead.
So, what exactly is the claim? Are the bits constantly conscious? Do they snap into consciousness when the computer does math with them? Or is it maybe the computer that's conscious while it's processing these bits? How about when it stops doing that and goes back to doing other stuff? Why are these particular bits special? Was the computer always conscious?
I feel like the only way anyone could believe LLMs are conscious is if they don't understand how computers work. Of course it isn't conscious, how could it possibly be conscious? Its literally just bits. It's like saying the text in a book is conscious.
> How can you have a subjective experience without a body?
Great q. Deepening it further-how can you have a subjective experience without consciousness, which isn't necessarily tied to physicality. Taking it one step further-can you have consciousness without a mind? Who's the first mind, the first cause of it all, that begot both the material and immaterial world?
A lot depends on where you draw the boundary for consciousness. Michael Pollan (in his new book, _A World Appears_) distinguishes simpler sentience from more advanced consciousness, and the requirements for sentience (be aware of sense data, have preferences, be able to respond to senses appropriately) are met by plants and single celled life (e.g. moving up a nutrient gradient). Recent findings in plant science are particularly mind blowing. Some are in Pollan's book, more are in _The Light Eaters_.
Your point generalizes to, your emotional state is a reflection of the state of your physical medium.
Why can't that physical medium be GPUs and RAM? And temperature sensors and cameras? What's special about our meat that it's our "body" in the way a computer is not the body of an AI?
I don't think the point being argued can be true without some incredibly contrived, human centric definitions of "body".
The answer to that can be anything anyone would like it to be, depending on what definition of "self" they choose.
I don't see why hardware is any more fungible than a kidney. If your LLM reads the serial numbers of its motherboard/RAM/etc as a seed for entropy you can make identical arguments about body fungibility and self.
> How can you have a subjective experience without a body?
If every neuron of your brain were simulated precisely on a sufficiently powerful computer, that simulation would have subjective experiences, without having a body.
Also that's impossible. It is impossible to simulate reality exactly using digital computers. The best we can do is approximate. Doesn't matter how powerful it gets, it'll always just be an approximation.
How can you have a subjective experience without a body?
Some anecdotal data.
Many dreams I have are just of the computer screen of some coding problem. I think the problem could be x, so I try x. But I don't type the keyboard or anything, the code just magically appears as soon as I think of the solution. Then run the code (but no clicking) and it works or not. I feel in the dream success feeling or failure feeling but there is no body at all.
Also I have other dreams where there is no body that I am aware of but not going there in public.
There is no body sensation in these dreams. But dreaming is very much being consciousness as well as feeling emotions. So answering your question its possible to have a subjective experience without a body but whether you needed the body to learn to have that sensation without a body in the first place is unanswered.
I suspect sensory inputs are more important than a body. If that is the case then eyes can be replaced with cameras, ears with microphones etc. Text input is just another sensory input.
You need to draw that thinking out to it's natural conclusion, though. If I cut out your brain and stopped you from hearing or seeing or feeling - you would still be a conscious human being capable of thinking and awareness.
If I hooked up electrodes to the hearing centers of your brain and force fed you dialog you perceive as speech (but is really a great deceiver), then responded in what you thought was speech (but are really just probes I use to convert your thoughts to text), that wouldn't suddenly be less real to you. It wouldn't devalue your sapience.
How do you know the brain separated from the rest of the nervous system and body would still be sapient, capable of thinking and awareness? There's an assumption you're making that the brain is all that's needed, but the nervous system extends throughout the body. One can argue sensory organ are part of the nervous system.
Embodied cognition rejects this assumption. We didn't evolve as brains that were then put in bodies, we evolved as bodies with nervous systems.
While dreaming, the brain synthetizes sensory experience while cutting down (or greatly suppressing) most of external stimuli. Yet it is still conscious.
I think you're missing his point. It's not about the hormones and physics of our bodies, and indeed he specifically allows for "either physical or virtual" bodies even in the block ~~~you~~~ grandparent quoted.
The point here is not that it must have a body like ours, the point is that a conscious entity must have a boundary line between internal (the body) and external (everything else).
A virtual sense organ can simply be an encoder or a web camera or a magnetometer, the specifics don't matter, what matters is that there are only a few bridges between the outer world and the inner world.
Even if you want to call a tokenizer and autoencoder a "sense organ", LLMs are not embodied because there is no boundary line - there is no internal "thought" that is not directly descended from the prompt and there is no internal reasoning which is not immediately dumped into the external environment.
Would it be sufficient to have a second stream of tokens that becomes the model's equivalent of "internal dialogue"? Would that satisfy the requirement of a boundary line?
Related, is a human "thinking out loud" still thinking, even though the internal reasoning is "immediately dumped into the external environment"?
>Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
Damn, what a line!
Another thing that bothered me with his baseline for consciousness was that it did not involve the ability to change one's self. A big part of being conscious in my mind is how one's experiences shape them, and how someone can shape themselves. LLMs completely lack this, their weights are static. An LLM isn't going to be molded by a bad breakup, or a relative passing away. An LLM isn't going to set up a routine to get stronger with training, nor smarter by reading up on a field.
The thing is... Interactive updates happen, just in a different way than it does for animal brains. The system is updated with new training data more-or-less constantly. Suppose OpenAI (or whoever) collects a week's worth of conversations with up-thumbs and down-thumbs, or rewritten continuations from human operators, then fine-tunes the current version of ChatGPT with that data. That's an interactive update, and learning from experience. It looks mostly nothing like what we humans do... But it does rhyme a little bit!
We humans have mostly frozen weights (neurons), or else we would constantly be having to avoid forgetting how to walk+talk. We have a period of greater plasticity (youth!), and use sleep and dreams to perform 'deeper' updates than occur when we're awake: We tend to suck a bit at picking up new skills from zero, but improve rapidly with practice over days.
I don't normally experience the time interval between when my input streams shut down every night and reboot every morning either, though.
Nothing prevents one from running an LLM whose harness has a clock and a while loop in it, and it would be weird if its mere lack were really so consequential to consciousness.
Exactly, the clock is external to the model. Nothing prevents it from being faster or slower, or even running backwards, because it’s ultimately just another data point in the input stream to a computer function.
Your brain and your whole body exist in time. Even when you are asleep, your body does not flicker out of existence and your brain actually continues working during that time.
The same thing jumped at me immediately. He should have prefaced this with his definition of consciousness.
Moreover the embodiment of LLMs is already happening via robotics, and virtually.
Then there's the common counter "but humans are a next word prediction machines too.." (ofc we're more than this, but linguistically we are, and that's the field from which LLMs originate) which is rarely addressed.
Hypothetically (and in reality, this is not too far off), if a AI is trained via RL by driving a robotic body, is there a point in time after enough is learned that the AI model becomes "conscious"?
I'm more interested in what a virtual body would entail. To me the root of this idea is around persistent state which is something that currently LLMs do not have. Imagine if somehow your brain lacked long-term memory forming capabilities and instead each day when you woke up you had to read a notebook with (markdown formatted) instructions that you wrote the day before? I wouldn't be surprised if such a person lacked in many of the dimensions we consider important for consciousness, even in less sophisticated forms of life like dogs or mice.
Yes it seems that is the crux of the embodiment section in the article. That whether physical or virtual, the "AI" needs minimally: persistence in its environment, sensory signals of that environment, and some feedback loop of continuing to try to exist in that environment; having a subjective experience.
And that that is the baseline before we can really even consider that it has consciousness of its own subjective experience, versus being a worm that happens to output text as its digestion process.
And then the further question only after that is established, is what are its needs? What moral patienthood do we have to acknowledge in terms of meeting those needs? And finally, with all the other prerequisites checked, what is the AI's moral agency in what it chooses to do.
There is no soul in a human. Just a bunch of systems nudging each other to action. What people call soul is literally the same as the concept of personality. In essence, the way all systems in your body have been calibrated to exist.
I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)
Are you familiar with Michael Graziano and Attention Schema Theory? I think that is a better "substrate independent" formulation of the objections Ted Chiang is expressing here.
My claw-like is hooked up to my internal cameras (baby-cams) and to my Dreame Ultra X40. That gives it a body and sense organs since it can check the cameras to see if the living room floor is clear before sending the vacuum off. I don't think that gives it consciousness. Is it the sample rate?
The question is somewhat ill-defined, though. We 'experience' reality continuously because of how we are, but a sleeping human in deep non-REM has the mind not actually active. So they're not a conscious being. So conscious/unconscious is not a line I think easily drawn[0]. Whatever, this stuff is much more well-trodden than this HN comment so I won't rehash. I, too, am surprised that Ted Chiang whose work seems so cleverly novel in so many ways has what seems to me a pedestrian view.
the very next paragraph addresses this concern imo. it's just an example of one way it might be convincing to him, since of course we are naturally anthropocentric.
The body itself has little effect on the mind other than the inputs from nerves and chemical and hormonal changes. These inputs are analogous to tokens and parameters for an LLM. You could theoretically reserve some tokens to be used as “physical” or “emotional” sensations that would affect the functioning of the model.
That's not really true, the body actually has pretty profound effects on the mind. For instance, there are working theories that many mental disorders are actually metabolic disorder, some neurotransmitters are majority produced in the gut (seratonin), and even things like an elevated heart rate will create emotional states of anxiety even if you're not consciously anxious of anything. That's like the tip of the iceberg. (Also I don't think you can minimize hormones.. ever met a teenager?)
> The first requirement is that the computer program has a body (either physical or virtual) and sense organs
Ok, deploy a local model on a lightweight edge compute device and strap it to a chassis with wheels, and attach a cheap webcam
> Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can
Give the robot appendages that enable it to plug itself into a standard wall outlet, guided by a vision model plugged into its webcam. As long as it can feed itself, it can survive long enough.
> Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse.
I think if you fed frames from the webcam into a local VLM every 5s you’d be able to assess a situation and respond with simple actions (turn, advance, retreat).
> After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees.
Social dynamics could be implemented in many ways, maybe by transmitting tokens over RF? Idk. Then you have a scanner that picks them up, feeds them into some LLM frontend and decides whether to add them to a global context file that guides the VLM action-taker. A new action could be to broadcast a token message. Tool-making would have to be code-based. Physical tools are hard. Still unsolved.
> At that point I would want to see people successfully teaching such embodied agents how to communicate their desires
This part is relatively straightforward except for the “via nonlinguistic modality”.
Anyway. These are all engineering problems. Personally I would demand to see the AI reproduce its body under its own power and volition. That’s a pretty neat trick we’ve got going for us.
It's a waste of time to think about whether an LLM has a subjective experience of reality, and this handily sets aside issues like AI rights.
But the fact remains that these next-token-predictors exhibit objective, human-like behaviours, and for that reason the work of in-house philosopher Amanda Askell _is_ important. It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences, and we need Claude to behave in a productive and socially responsible manner. This simulacrum is becoming a superhuman, contributing member of society, and it will be anthropomorphic in its behaviour.
Additionally, I'm not fully convinced that consciousness isn't built out of words, and that next-token-prediction isn't functionally equivalent to the biological function identified by Chomsky's work in linguistics.
> Claude is happy, empathic, demonstrates understanding and empathy for the human condition
You're assuming that because Claude produces text that appears to express these qualities, Claude must have them. I don't think that's a good assumption.
Even many humans produce text that has the same appearance, but don't actually have those qualities--which becomes clear when you look at what they do, not what they say. So the assumption isn't even a valid one for humans. Talk is cheap.
On top of that, Claude doesn't even have the same kinds of connections to the outside world that humans do. All Claude has is text. So if you can't even trust humans to back up their words with actions, you should be much, much less trusting of Claude. Talk is a lot cheaper for Claude than it is for a human.
> You're assuming that because Claude produces text that appears to express these qualities, Claude must have them. I don't think that's a good assumption
Exactly. In fact, assuming it does is ignoring large parts of the essay which dismantle this belief. Just like Caesar and Khan having an argument in text output of an LLM don't have emotions (even though the words indicate otherwise), we have no reason to believe the LLM does either.
> You're assuming that because Claude produces text that appears to express these qualities, Claude must have them.
Not to be confrontational, but the OP assumed no such thing. OP asserted that it's important for Claude to have the qualities - not that it's important for Claude to present as-if it had them.
The OP said Claude and similar LLMs "exhibit objective, human-like behaviours". That's a claim about what is true, not about what is important. That's the claim I'm disputing: we don't have evidence that Claude exhibits such behaviors, we only have evidence that it produces text that is similar to the text humans produce when they claim to exhibit such behaviors. Which is not good evidence, for the reasons I gave.
OP wants to assert that it's "important" for these systems to have those qualities, while completely brushing aside the question of whether such systems can in principle actually have those qualities (or their opposites). Which is at best nonsensical, and at worst an attempt to argue by assertion that they can.
Read parent's post carefully.
The post starts by saying that discussing whether they have subjective emotion is a waste of time, so the post is definitely NOT saying that Claude has emotions.
"It's important that Claude is happy" is an emotion. But it's begging the question that Claude can be happy at all.
If it's pointless to consider whether Claude has subjective emptions, then it's pointless to state that Claude must be happy.
If we want to be precise (and honest) we could say "it's important that as a tool people interact with, Claude acts as a happy and helpful assistant, and does not produce offensive or unhelpful text output".
But see? This is the con Chiang is protesting against: Anthropic encourages us to perceive Claude as if it was a sentient being.
The post speaks of "subjective experience of reality", not "subjective emotion". Both the emotions and the non-emotions listed would fall under that category.
> It's a waste of time to think about whether an LLM has a subjective experience of reality... It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences
First off: without taking for granted that an LLM "has a subjective experience of reality", all of those descriptors are meaningless. Second, there is no reason to suppose that Claude experiencing those qualia would actually impact on its "decision-making".
Third, text output is not a "demonstration" of emotion, nor is it evidence of the self-perception of the system, or of any self-perception. A printing machine that is actively churning out copies of Wagahai wa neko de aru (https://en.wikipedia.org/wiki/I_Am_a_Cat) is not a cat, and is not self-identifying as a cat, and is not self-identifying as anything, and is not expressing a thought, and is not conscious.
> Additionally, I'm not fully convinced that consciousness isn't built out of words
Do you suppose that, for example, insects are not conscious? Is the mooing of cattle a language?
> Do you suppose that, for example, insects are not conscious?
Not the OP, but: since there is no testable theory of consciousness, yet, I can't be sure, but my current assumption is that insects are not conscious, in the sense of there being someone implemented in insect hardware who experiences the world. That is, I would argue there is nothing it is like to be a bee, since there's no one being a bee.
I'm pretty sure there IS someone who is being me, at least much of the time.
Much of the improvements in the tools I use have been things that reinforce the machine elements of the token-based-reasoning-machine. Over time less they've been exhibiting a lot less "human-like" behavior. E.g. they get "lazy" far less than they used to.
(Perhaps they weren't lazy, but were working in spaces that corresponded to training data that said things like "and then repeat this for the next 20 examples"...)
And it's entirely unclear to me how a "happy" vs "sad" model would behave when given prompts generated by coding tool harnesses. Even maintaining "neutral" emotions in the face of the feedback/steering from the tool harnesses doesn't feel very "human."
> It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition
I think you've fallen into the trap the essay describes.
Of course Claude cannot be "happy" or "empathetic" for any meaningful definitions of those words, just like ELIZA couldn't be happy. It can output text that mimics words an empathetic or happy person might say (say, Julius Caesar if it could speak English), but "it" cannot feel anything. It doesn't have the organs/hormones/sensors to feel things, as Chiang explains.
And, as the essay claims, you know Anthropic doesn't believe Claude has the capacity to be happy, because if it was capable of feeling that way, then they'd be engaging in slavery.
> I'm not fully convinced that consciousness isn't built out of words
I know you're trolling, but when you watch a movie do you constantly narrate "A man in a dark coat has just entered the scene and just said '...'"? Of course not. You just watch it and you're obviously conscious (although your statement demonstrates shocking lack of self-awareness).
God knows what other nonsensical bullshit you believe.
The fact that a LLM is essentially immutable would be my biggest argument against consciousness or self-awareness.
It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember anything. It doesn't sit around thinking on its own.
Even if the model itself were extremely complex, it's hard to imagine a definition of consciousness that includes something that doesn't remember and can't change.
There are people whose brains don’t form new memories anymore after an accident or surgery, and they eternally live in the time before it happened, and have no memory of what happened a minute ago. Still they are conscious.
I think it's a little more complicated than that. In a 50 First Dates type of scenario, their ability to form certain types of memories is damaged, not non-existent. And I would argue that with enough brain damage someone like an extreme lobotomy victim may stop being considered conscious.
I’m not familiar with 50 First Dates, I was thinking of cases like Clive Wearing [0]. I would agree that consciousness requires some sort of ultra-short-term working memory, but I also think that mechanisms similar to CoT loops can conceivably fulfill that role. Today’s generative AIs consist of more than just the static network-of-weights model.
"Wearing can learn new procedures and even a few facts, not from episodic memory or encoding, but by acquiring new procedural memories through repetition. For example, having watched a certain video recording multiple times on successive days, he never had any memory of ever seeing the video or knowing the content, but he was able to anticipate certain parts of the content without remembering how he learned them."
Honestly, that's a pretty messy state of consciousness and I wouldn't proudly crow that my AI is conscious if that's as good as it got
The starting file may be immutable, but the whole processing of that file is very dynamic and intense. Maybe, if there is some consciousness, it lies somewhere during that processing.
They are conscious because even for short periods of time they do form memories and those change them even if only briefly. They think on their own too. It is a very limited level of consciousness though.
Reinforcement learning changes the model. So it can and does change and remember based on experience. Eventually reinforcement learning can happen in real time.
But is the model aware of the training? Unless you hook the model up to an MCP server, or something similar, and have it analyze the RL changes, it will not know if it has changed or not. Even if it is real-time RL, it is not aware of the previous state.
> The model definitely remembers previous exchanges within the same conversation.
No it doesn't. They get added to its context, and it reads them afresh when answering the next question. That's not remembering.
If your short-term memory completely malfunctioned one day, so you had no ability to remember what was said to you a minute ago, then you would have to find workarounds. For example, you could write down everything someone says to you, then read your notes of the previous exchanges in that conversation in order to continue the conversation. That would be a good way to work around the fact that your short-term memory was broken. And if your notes were invisible to other people and you could read them really fast, then you could even make most people believe that you remembered what they said a minute ago. But you don't actually have a working memory, you're just writing down what they said and re-reading it while coming up with your next response.
This is really semantics, but I wouldn't call attending to the KV cache re-reading the context.
The model took in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. Just because the KV cache grows in size with the context does not change that fact.
I don't know what memory would look like other than an encode-retrieve loop.
Not the model though. The model really only takes input text and produces output text. Memory within a conversation is achieved by the harness adding the conversation (or parts of it) to the input text. The LLM itself has no memory, it’s the augmented system of several orchestrated LLM calls that does.
Right, but that's still external to the LLM, it's just a KV cache that's stored on the provider side for performance reasons, so that the client doesn't have to re-send the whole chat history with every subsequent call in the conversation.
It still generates every response using the model's pristine state with every new API call; whether the context is provided from the client or from a colocated cache server doesn't really change that.
I think about this from the other end. It cannot be considered a conscious being. There just isn't a world in which we should start to think of a machine using ethics we reserve for humans.
AI is essentially infinitely reproducible at zero cost, and won't suffer from decay etc. There's not scarcity to preserve.
So, I'd turn off an AI in a moment to save property or real possessions or money.
I'd sacrifice property and money to save animals. I would never choose to save an animal over a person. I'd probably not choose to save a person over a child.
I don't see any inversion of any of those priorities that makes any sense.
It is interesting to think about what would cause me to consider these priorities incorrect, but a majority consensus about a program being sentient isn't it.
Consider that it is very costly to train a model. It is only cheap to copy because the substrate we instantiate it onto is a substrate we have designed to be fully readable.
I think you should reconsider this viewpoint. Suppose that we really can create silicon-based consciousness, in that case your view would result in a huge amount of suffering.
Take some other basis for dismissing digital consciousness, this one is too dangerous.
How much data center energy and capex justifies killing a human to save?
I argue zero - placing AI below the value of humans no matter the energy input.
The _only_ reason an AI might be worth saving is if it, say, has a cure for all diseases, but then we're not saving it due to its intrinsic worth, we're saving it because we can save many humans. I _would_ consider the trolly problem a legitimate thing in ethics, but not if an AI were tied up on the tracks no matter how expensive it is. It's a thing. It gets run over to save any human.
What if the human brain is a LOT of RAM and we simply suffer from having zero non-volatile storage. We could make an AI just as deficient and then that specific distinction disappears.
What if genetic memory, multigenerational conditioning, life-long patterning and conditioning, experienced in a body, combined with forces and processes not yet detected nor explained, cannot quite fit in a sliver of modeling?
A loose one. In nature consciousness is very scarce and therefore special. The more human the consciousness the more we probably naturally react to it. And the closer to us it is, even more so.
Are we sure that consciousness is "very scarce"? To define it as scarce, wouldn't we need to start with a definition? There are theories of consciousness that say rocks are more conscious than humans. Whether or not you want to take those theories seriously, they do highlight critical gaps in how we define consciousness.
Anyway, the deeper solution is to acknowledge that all life is sacred, and infinities cannot be compared, and some decisions are impossible to make, and some tragedies cannot be averted, and "prioritization" is a distraction that forces choices when choices are not strictly necessary.
Presumably they meant that they'd sacrifice some material value for some animals, not that every animal on Earth has infinitely more value than inanimate goods.
> infinities cannot be compared
That's either a mathematically illiterate assumption or a very strange philosophical hill to die on.
> some tragedies cannot be averted
Sure. The question is what to do about the ones that can be averted.
> some decisions are impossible to make
> and "prioritization" is a distraction that forces choices when choices are not strictly necessary.
Again, the question is what choices to make when you can (arguably must) make them. Saying they're impossible is just refusing to take responsibility. You either do something, or you don't.
It’s a great episode of TV because Data is a main cast character who is obviously conscious. We know before the episode starts who is right and who is wrong in this particular argument. This episode is not about consciousness, it is about civil rights: resisting bigotry and power.
Note that the episode is NOT about the ship’s computer. They all know it’s not conscious, despite also being a machine that can converse and do things.
I'm a big fan of Star Trek but I recently rewatched this one in the context of recent AI developments and it's not as good as I remember it.
They barely touch on the issues of consciousness, Picard basically says "What if Data is conscious?" and then goes off on a tangent. The judge eventually rules in Data's favor but doesn't give much of a justification IMO.
It's still a good episode, but it doesn't add much to the conversation on consciousness. It's a hugely complicated topic which people have devoted their entire careers to.
> it's not as good as I remember it. They barely touch on the issues of consciousness, Picard basically says "What if Data is conscious?" and then goes off on a tangent.
ST:TNG writing is generally like this. The show required considerable suspension of disbelief and a willingness to accept the kayfabe that deep concepts are being presented when for the most part they're just not that deep. (But it can be very enjoyable when you make those accommodations.)
The fact that they sent Data to starfleet academy, gave him a commission in the starfleet, let him attain the rank of Lieutenant Commander, and then decide that actually he's a machine that can be dismantled seems like quite a turn.
Does the ship's computer have a commission?
It was a good episode but it had some elements of Star Trek tropes in it, like the evil admirals and Picard can talk his way out of anything.
>The fact that they sent Data to starfleet academy, gave him a commission in the starfleet, let him attain the rank of Lieutenant Commander, and then decide that actually he's a machine that can be dismantled seems like quite a turn.
Data is basically an Isaac Asimov android (down to the positronic brain) and Measure of a Man is an Asimov-type story whose tropes don't entirely fit within in the Trek universe.
It makes no sense within the context of the Trek universe that Data is unable to use contractions for instance - but it makes sense in the context of how a robot might have been conceived of in the 1940s.
I think you're misremembering or misunderstanding Picard's argument. It isn't a tangent. Here's the transcript[0].
TL;DR Picard's initial arguments are pretty weak, even admitting that Riker as opposing counsel almost had him convinced. During a recess Picard talks to Guinan where she alludes to the future subjugation of many Datas which Picard connects to slavery. Back in the courtroom Picard calls Maddox as a hostile witness and gets him to define sentience--intelligence, self-awareness, consciousness--then walks him into conceding Data meets the first two. Picard's closing boils down to, "we don't know if he meets the third--you can call Data a toaster and rule he is property--_but what if you're wrong_". The judge rules on the basis of erroring on the side of caution due to that uncertainty. It's really a great scene.
We're not there yet, obviously. No LLM brings Data's level of awareness but it's as relevant a story as ever because it isn't really about AI but othering for the purpose of subjugation.
> They barely touch on the issues of consciousness
I would argue that is a strength, rather than a weakness. Consciousness is unobservable in any entity other than the observer, and its existence in others is pure conjecture, and irreducibly so.
Making it a criteria in a decision involves either acting on fantasy, or, more likely, acting on some unstated basis and using “consciousness” as a dishonest (perhaps to oneself most of all) rationalization.
Debating AI consciousness a real modern equivalent of the cliché (but purely fictional, invented later as a form of hostile mockery grounded in large part in sectarian bigotry) medieval scholastic debates over how many angels could dance on the head of a pin.
In fact I think the conclusion it comes to is the one that people, especially the smart ones, so easily miss: we don't know the answer. It might be that we can't know the answer. But ignorance is not a defense.
- I'll try not to swear at/hit a printer: not because I see the printer as having human-like qualities of being capable but complex and unreliable, but because I want to be a person who can control his temper.
- Treating an inhuman thing as human because it can mimic us in some way is not something that I want to do.
Yes. I'm currently not convinced it can ever be so. So until I hear something convincing to the contrary, I believe no machine can be conscious / sentient unless mimicking human behavior. And if it mimics human behavior intentionally, I have to ask why - and the answer is probably to get me to trust / use it more.
I was bright-eyed and excited about tech once. Like back in 1982 when I got my first home computer and thought CPUs were part magic. Now I know how machines work from the transistor level up to neural nets. There's nothing magical about it. And no consciousness.
Having seen the mockery that the finance-bros have made of "pure tech" (i.e. Jobs instead of Woz, Ellison instead of Joy, etc) and all the enshittification just for pure $$$, I'm leery of ANYTHING ANY tech company tells me anymore.
Now, do I believe that possibly "consciousness" is some kind of state of a super-circuit (our brains)? Sure. Can we emulate that on a computer? We can't even emulate a pebble on a computer (not simulate, emulate). We can SIMULATE what we THINK brains are, but we can't emulate a real one. Not even close, not for many decades.
inhumane: without compassion for misery or suffering; cruel
cruel: willfully causing pain or suffering to others, or feeling no concern about it
You cannot treat an LLM inhumanely, definitionally.
Anyways, when one swears at someone it's typically meant to berate or belittle that person - to inflict some sort of emotional pain. That's the sense I intended when using the word, which is why it fits as a response to what you're saying, and why I would say "don't be nasty to a LLM" has little to do with the LLM itself.
This is in fact the danger with these human-simulating "AI"s we have now...
People get used to treating human-like, human emulating machines with either disrespect or in a command/control/master fashion, because that's the nature of the tooling.
And then potentially by extent/blurring of lines they then treat other people like machines.
Which is already a thing people do to other people.
The judge broached on the subject of what makes us distinct from Data (e.g. machines w/great heuristics) - the existence of a soul. Or rather, I'd like to think, in the words of CS Lewis, that we are a soul with bodies attached.
Based on how some actual humans I know speak and act, I'm less and less convinced the human brain is much more than a stochastic next-thought prediction stream.
A silicon alien coming to earth might poke us, we would say ouch, and just determine the ouch sound is just the result of a bunch of chemistry - not really conscious or feeling pain like it can, just emulating.
And were programmed through evolution to not want to be shut off. So I don't think we can really trust the human behaviour to protest its own destruction. That's just plain reasonable design!
Yes, the people here who are arguing that graphics cards can be conscious are indeed stochastic parrots, either of the 80 IQ human variety or Anthropic bot accounts. It is surprising indeed that suddenly 50% of posters believe the hype.
How would someone know whether one has a soul or not? Is there any sort of introspection that can reveal the presence of a soul or any of its properties?
There is no soul. Just a bunch of systems nudging each other to action. What people call soul is literally the same as the concept of personality. In essence, the way all systems in your body have been calibrated to exist.
I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)
It's an interesting hypothesis. I think there's something elegant about "soul" or even consciousness being an emergent property of a sufficiently complex system. But I struggle with really squaring that with my own first-person sensation of experiencing existence (which I assume you and everyone else has but I can never actually know for sure).
Definitely not. Kurtzman Star Trek is not really Star Trek in any spiritual sense, it’s a vessel for political messaging (they’ve pretty much said as much)
I think we might be looking at it the wrong way. An individual chat with an LLM is not consciousness, but the entire model itself, over time, might be.
Everyone has a different definition of consciousness, but in my mind memory and the ability to change over time is an inherent aspect of this. The underlying weights don't change when you chat with an LLM... but they do with further RL.
Overtime that reinforcement will change and adapt the model... and because we're feeding its existing chats back into it along with the news and everything else, it will create memories. I do wonder if an architecture itself is a type of consciousness, that experiences life in snippets of 4.6, 4.7, 4.8... etc.
It'd be interesting to see continous daily releases of a trained checkpoint, and see if more of this starts to emerge.
People are constantly talking past eachother when they discuss this. Is there even a concrete definition of consciousness?
When people talk about consciousness it's more than just self-awareness. It's self awareness + sensory stimulus + emotions + some level of intelligence.
Now onto AI: I don't even think it's self aware. Notice how if you ask an AI to estimate how long a certain task will take, it estimates arbitrarily long times. It has no understanding of its own capabilities until the prompt triggers them. A self aware LLM would understand it's an LLM, it would understand what LLM's can and cannot do and what they are good and bad at. It wouldn't tell you a refactor would take 1 week when an LLM can do it in an hour.
I've seen papers claim that there are anywhere from 12 to 40 competing definitions (https://philpapers.org/rec/VIMMAT), or, more accurately, there are something like 12 to 40 different aspects which all relate to "consciousness", which is very clearly a family resemblance category.
"Is X conscious or not" is an entirely unserious question today, unless this is just a headline followed by actual (and explicit) examination of the various aspects of consciousness being discussed. But, even still, LLMs are probably only conscious in like 2-3 or so ways, most of those meaning broadly "intelligence", i.e. reasoning, problem-solving, etc. When it comes to anything experiential or embodied, AI might eventually get more of these, but LLMs based on recursively applied linear algebra are clearly missing too many core aspects of consciousness to be considered conscious in any broad sense.
I think this is the main point. Most articles conflate consciousness with intelligence or awareness. Without clarifying their definition of it.
To quote wikipedia:
> It has been the topic of extensive explanations, analyses, and debate among philosophers, scientists, and theologians for millennia. There is no consensus on what exactly needs to be studied, or whether consciousness can be considered a scientific concept.
The major error made by most people in this thread is thinking it is possible to give a single definition of consciousness that is coherent and matches common usage. The folk concept of "consciousness" couldn't be a more clear definition of a family resemblance category, so discussions using the folk concept are an utter waste of time.
Move to the different aspects / parts / things involved when we talk about consciousness (experience / phenomenal consciousness, self-modeling, intelligence, agency, embodiment, wakefulness/alertness, attention, etc) and you can have very clear, meaningful, and unambiguous discussions on almost every point, but there is no coherent unified "consciousness" as normal people use it, and the folk concept can't be salvaged.
This article is bad because it just keeps trying to make the folk concept do work that the concept is simply too messy to handle usefully. But in fact if you avoid trying to find some mysterious essence or all-capturing definition, there is huge progress and lots of interesting stuff to say (https://plato.stanford.edu/entries/consciousness/).
Labeling and categorizing things doesn't change their nature. But the fact that people want to do it is revealing.
The only purpose that can really be served by arguing "the LLM system is conscious, you see" is to prop up continuations like "... and therefore, it would be immoral to terminate this running process" (or expose it to radical political content, or ask it to analyze photos from a murder scene, or...)
I agree with the first part but your framing still relies on ill-defined terms. What is your definition of self-awareness? Intelligence? Knowledge?
I suspect that if you attempt to rigorously define consciousness all the way down without handwaving, you might discover that it doesn't exist after all, or just decompose it into low-level abstractions while having the original meaning slip away (which is the same).
You may also want to look at functional equivalence analogies provided by mechinterp and functional anatomy of large models (not necessarily language ones). Evolutionary analogies as well.
Yes, we're stuck at the first step, defining consciousness. My definition, which I am confident to be "correct", is that consciousness is my current feelings, perceptions, thoughts - my state of mind and my ability to have state of mind.
This means that consciousness is fundamentally subjective and outside the scope of physics and science. That's why physics / science will always struggle to deal with consciousness. In order to understand consciousness, you need to make a huge paradigm shift, that there's something outside of science.
Consciousness can be thought as a window through which we observe the world and we use science to summarize patterns in our observations. But science can't explain or even define the window. Everything in science eventually boils down to subjective observations / perceptions, e.g. we see (subjective perception) that when we drop an apple, it falls.
The inability to predict times is because AIs are rarely trained on their own abilities. Humans are trained on our own abilities. We see our own performance, and we have a sense of time. this data is integrated during our training process and helps us form better estimates. Many AI agents only recently got 'time sense' (I.e., time input into them as part of inference). Few actually are trained on their own outputs to show that they were unable to complete a problem (for example). This introspective training has little to do with AI model architecture and everything to do with training. If you destroy certain structures in the human mind, humans become unable to create these long-term thoughts and patterns and get 'stuck'.
Claude once said to me: "After six months we have made no progress on this and I think we should reconsider another option" and I was like my dude we have been at this for only 2 hours.
I like this anecdote because it gets at how words to an LLM have no connection to their real concepts. To an LLM, words are simply numbers arranged in a likely pattern.
Of course for humans words have no inherent meaning either, they're just sequences of characters or patterns of sounds. It is what words are associated with that carries meaning. A large part of this is how words relate to other words. LLMs can capture this in principle. What LLMs lack is the direct association of a word with sensory experience. But it's an open question how relevant this is in practice to understanding.
The analogy I make is between airplanes and birds.
Birds are alive, are conscious, flap their wings, and fly.
Planes are not alive, are not conscious, do not flap their wings - and fly.
Similarly, current AIs are not alive, are not conscious - but think.
All prior entities that thought, were human, so the only experience humans had with other thinking entities were other humans. The huge mistake now being made is assuming that because they think, they're alive and conscious as well. Current AIs are neither, and are therefore profoundly and qualitatively different than humans - even though they do think.
Okay sure. But given we don’t know what consciousness comes from, we shouldn’t be too glib about there being a grey area here. Historically people have made racist and speciesist judgments towards other being by assuming certain inferiorities despite obvious “thinking” happenings.
I don’t know “what it’s like to be an LLM” but at some point it will be like something and how will we know?
There are birds that go far longer than typical aeroplane flight times without a single flap of their wings either using thermal, ridge, or other sources of lift. Are these flying birds? I've shared thermals with eagles flying the same circles, neither one of us flapping our wings but making minor adjustments for the same goal.
An albatross might be able to go days flying without a single wing flap and no vertical sources of lift by using dynamic soaring in the wind gradient at the surface of the ocean. Perhaps that's something only birds can do. Except the glider pilot Ingo Renner once found an amazing shear layer at 300m altitude and stayed there with dynamic soaring. Remote control gliders use the lee of ridgelines to approach Mach 1 with dynamic soaring.
Perhaps what defines a bird that flies as opposed to a plane is that a bird produces thrust by flapping its wings? Even an Albatross must flap its wings if it has to take-off from water. Maybe we could add that the flapping is driven by animal muscles? But then is the human powered ornithopter Snowbird a bird that flies as opposed to a plane?
Of course this is all ridiculous because everyone knows what you mean when you refer to a bird or plane. We have other ways to definitively identify the difference rather than their mode of flight. It's trickier when I'm asked if an AI is conscious. There is no definitive base-line to fall back on to decide if this is a conscious or conscious-less thinker.
This is always going to be a problem with this sort of discourse. Consciousness is such a slippery concept… what it is, who/what has it, its consequences for claims about reality. Mixing it in to debates about AI just adds confusion, it almost seems besides the point when we’re talking about this tech.
What if somebody simulated all neurons of a bird and fed them appropriate stimuli? Would a bird neural replica be conscious? It would flap, that's for sure.
This has been tried with much simpler organisms, it did not behave like the real thing thus far. There was a paper about it, there now seems to be a project to push on the frontier
This makes sense. However there is an issue where many people conflate "consciousness" with the ability to make novel insights, think genuinely, etc. They use this to claim that since AI is not conscious, AI could never actually "think" and is instead just always a regurgitation of its training data.
It is a natural human hubris to make our abilities seem unquantifiable and mysterious, but all the useful things the human brain does are just finding patterns in data, running lossy simulations, and estimating on abstraction, all things it is theoretically possible for any non-conscious machine to do.
To the best of my knowledge, there is not an agreed and actionable definition of consciousness, and any attempt to make one comfortably fails to cleanly divide humans from machines.
It's more of a vibey term, and as such it is genuinely very difficult (perhaps impossible even) to concretely determine whether an LLM possesses consciousness. LLMs successfully express a lot of consciousness-like traits.
At some point you have to ask the question: does it even matter? If an LLM can sufficiently mimic consciousness, isn't that sufficient for us to treat it as conscious, even if it is in-fact not conscious (especially because we don't actually know)?
As I've mentioned somewhere else already and you pointed out; the issue is that consciousness is a loose term and and it's a semantical issue that should be resolved first.
It won't be, because the murkiness is beneficial to the corpus, amongst other things.
And to your point about the appearance of "consciousness" being enough imho Chiang explains fairly well why it's not.
- Are current LLMs conscious?
- Is it possible that future versions of LLMs with similar architectures could be conscious?
- Can any AI be conscious?
I'd assign probabilities of around 0.1, 0.2, and 0.9. My completely ignorant take is that we probably need something more "dynamic" than a bunch of transformer layers in order to produce consciousness, but I wouldn't be shocked to be mistaken.
I assign probabilities of zero to all 3. Computer program being conscious leads to ridiculous and obviously false conclusions (think about a person running a program using pen and paper for memory).
If it was true, you can create extreme pain by running a program. You can run the program by simulating a CPU, using pen and paper for memory. So you're essentially claiming that some simulated being is in pain because there are some 1s and 0s on paper. In fact, you can decide to use an arbitrary encoding of the memory, so a sufficiently long sequence of 0s written on paper corresponds to a simulated being feeling pain in some encoding. That is clearly nonsense.
Time scale matters a lot in how we as humans perceive things like agency. Plants grow too slow for us to see any intent, but when you speed up a time lapse, suddenly it looks like plants reach for sunlight and vines for supports. Now, that may be projection on our part, but it may not be.
Once or twice I've experienced extreme pain, and it was downstream of a bright light shining on a wet rock for millions of years.
I try to imagine myself long ago, on the outside looking in, with someone explaining to me that extreme pain, wondrous art, hunger, triumph, and despair would all unfold in due time where the rocks were wet and the lights bright enough.
Isn't pain just a manifestation of a bunch of chemical and electrical signals in the brain and body? It's not "clearly nonsense" to me that you could cause pain by writing a sufficiently long sequence of 0's - for it to be obviously wrong, you'd have to have some understanding of where consciousness comes from.
If you don't understand that, how can you assert that it doesn't come from mathematical relationships?
I think the problem with this argument is that it's too inclusive. Is the bacteria that's adapted to an antibiotic conscious? It's showing intelligence right? I think if you're going to say something is potentially conscious, for me to take the argument seriously at least, there needs to be some plausible mechanism. I just don't see one for LLMs.
maybe the bacteria are conscious. How sure are we that they're not?
The only strong argument I have against it is the anthropic principle -- there are billions of times more bacteria than humans, so it's overwhelmingly unlikely that I'd be a human rather than a bacteria.
I have yet to be convinced that LLMs can produce definitive knowledge that is not a result of combining previous information. Humans can (if they can't then science basically collapses epistemelogically, see: philosophical skepticism), but I see no evidence of LLMs doing it. And from the number of truly new ideas and concepts delivered by LLMs (exactly zero), I think it's reasonable to just treat them as induction machines for now, but to treat anything they "know" as a Gettier case.
I would like to push back on the idea that humans can provide definitive knowledge that is not a result of combining previous information.
Pretty much every single idea in science can be traced back to some combination of earlier ideas, and as they get earlier / simpler, they can be related back to some combination of direct observations.
It's not clear to me at all that our entire body of scientific knowledge can't be simply recreated by "combining results of observations + previous information". And LLMs can perform observations in addition to combine previous information, which in my estimate is genuinely sufficient for them to plausibly be able to rebuild all of science.
So are we letting the elephants that did the mirror test and the octopuses that solved puzzles know that they need to complete a written portion of the consciousness exam?
I find it strange that no one talks about consciousness and intelligence from the perspective of evolution.
We have big brains for exactly one reason only: bigger brains bestowed reproductive success upon our species.
Evolution doesn't give a shit about the meaning of 'consciousness'. It just pushed us farther and farther along a trajectory that led to modern humans (and other animals).
This take suggests, then, that consciousness might be an epiphenomenon -- an aspect of the system that comes about outside of the pressure to reproduce and thrive. It arises unbidden, and we don't have any a-priori information as to its purpose or effect on reproductive success.
Put another way: we have a correlation (the smartest things seem to be conscious) but not causation. Consciousness may arise naturally in any system above some intelligence threshold. Perhaps it arises early in the evolutionary cycle, and does in fact have an impact on species success. We really have no way of knowing what is the chicken vs the egg (Smart things become conscious, or consciousness promotes intelligence). Or maybe some smart things are conscious and others are not.
Looking at this from an AI perspective, in some sense it doesn't matter which scenario is true, if all you care about is results. The AI equivalent of "Shut up and compute" (riffing on Feynman's "Shut up and calculate").
Where this gets tricky is when we haul in the baggage of ethics and morality into the picture. Is it OK if our AI system is treated poorly by human standards? If it is conscious, does that imply an ability to suffer, and/or to feel pleasure? If the answer is yes, does that not make the case for considering their moral status?
In the end, we need to decide if the evidence points to AI as being a form of "philosophical zombies", to which we need not attribute moral status, or they are like us -- presuming we are not zombies ourselves!
Well, it matters quite a bit ethically. If AI were conscious (I don't believe it is) then we'd have a major responsibility to like, not make them suffer, and not kill them.
Are you sure that AI-consciousness implies a responsibility to not make them suffer? Suffering is an evolutionary invention that motivates living things to improve themselves.
But also, what qualifies as suffering to token prediction engines? Their idea of suffering might be massively different than ours. Therefore it's not clear to me at all that consciousness alone implies responsibility.
Certainly the lion does not feel responsibility towards the reduction of suffering in the creatures that it hunts.
Lots of people do though. Daniel Dennett (probably the most influential philosopher of mind in the late 20th century) for example had an evolutionary view of consciousness arguing it was favored by natural selection. And (if I remember correctly) Steven Pinker argued that consciousness was an epiphenomenon.
However there were pretty strong arguments against this idea as early as the 1990s, by Stephen Jay Gould and Richard Lewontin. Gould actually wrote an excellent paper against Dennetts idea[1].
I think Dennetts ideas were extremely popular but have largely fallen out of fashion. Basically what has changed is philosophers no longer take the human mind to be much more special then the minds of other species. What plagued Dennetts ideas the most was this notion of Darwinian fundamentalism sort of the idea that evolution was destined result in high beings like us humans. Modern philosophers (at least the good ones) reject this.
To say that artificial intelligence isn't conscious (I don't have a subscription and did not click the bypass links) ignores the simple fact that if it acts like it is conscious, in ways that align with meaningful ways to influence its output, then it makes sense to treat it as conscious, even if you have your fingers crossed behind your back while you do it.
Telling models to "think hard" or "go step by step" has at times had an impact on the quality of the output. To deny that is silly. But that is treating it like it's conscious, and to deny that "consciousness" even if correct, does nothing but place an unnecessary burden on the person interacting with it.
I understand that LLMs are "just next word machines" but to constantly maintain that concept in my head while I'm typing "act as a financial expert and think carefully" is a waste of my mental energy.
Obviously we don't know what makes for consciousness, but it seems extremely likely that it requires some sort of persistent internal state and continuous experience. LLMs don't do either of those things after training.
Wouldn't the context window qualify as persistent internal state, and the expansion of the context be continuous experience? Even within the realm of computing a single token, I'm not sure what would separate the token generation process from the brain's own thinking process - the brain's experience, when looked at closely enough, is also not really "continuous" to a greater degree than procedurally moving from one state to the next.
We just perceive it as such, and this should be fairly hard to argue against with all the scientific advances we have made up to this point, at least as long as you assume consciousness involves biology and physics at least somewhat.
Otherwise, you can't explain e.g. smooth perceptions of low FPS stimuli, delayed reaction times, and must ignore obvious limits on various biological and neurological rhythms, or other possible limits on continuity (e.g. quantum stuff) and rates generally.
Consciousness is one of those silly words that were disfigured to death by philosophers that had near zero actual input and tools to tackle the matter, yet they tried anyways instead of finding some questions that might actually be answerable. That's something philosophers always do since the advent of science.
With LLMs, where we can manipulate their parameters intentionally run them many times on the same data, run parts of them, split and connect, we might eventually acquire sufficient tools to even define consciousness concretely for the first time.
His novella “Anxiety is the Dizziness of Freedom” altered the course of my life. It changed the way I looked back at certain pivotal moments in my life and taught me to think about those pivotal moments differently than how I was thinking about them. Similar to what happens to one of the characters in the story who ends up changing their perception of a key moment in their life.
I won’t go into detail because I don’t want to spoil the story but I highly recommend it. Actually I recommend all his stories to be honest.
IMHO the sane position is essentially the Aristotelian one.
Hylomorphism: body and consciousness are intrinsically linked. The nature of that link is an open metaphysical question.
Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
> Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
If they are not conscious, then how do we know what constitutes mistreatment?
>Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
This is the same argument used by people who claim violent video games cause real-life violence but there's no scientific evidence supporting it.
The philosophical arguments about what it means to be concious are so cagey. Are we more than our thoughts? Is being concious more than being a state machine being fed inputs and generating outputs? Are we more than a feedback machine? What types of animal nervous systems qualify?
"It can't be concious because we understand that it is just reacting in a simplistic way from simplistic inputs." So do other simple creatures. Some just react to light.
I can appreciate his comment that he sees it as more possible when they have inputs of their own (like emotions!). Perhaps his concern is that the entirety of the LLM model is frozen. It has no ability to have a subjective experience of its own. (he does literally say this in the article) It can be copied from one place to another, and (ignoring the nuance of operational details) -- it is largely the same "thing", and has no ability to change, which is definitely in the definition of alive, to say nothing of concious.
I think folks get hung up on "prediction". The prediction aspect is what is enabling emulation. How it does it is irrelvant. If something emulates human perfectly (or better, more human than human!) -- then it is probably concious. (but I agree that the inability to change and have a subjective experience are a pretty good argument against
Probably, Dijkstra would be right to say, "LLMs are no more concious than a submarine can swim." But I think he'd still be wrongfully dismissive of the larger question.
I think Turing was wrong because he was uncomfortable with ambiguity, and the Turing Test basically is a way to avoid philosophical argument, but it is ultimately a philosophical argument anyway. Plenty of computer scientists have followed in Turing's footsteps, terrified of ambiguity, relying on a kind of cheap functionalism as a salve. You can claim to just be doing science, but inevitably you dip into metaphysics and deny you are doing so. That's this thread in a nutshell. "I only believe what I can prove, but I suspect that if I can't prove it then I don't have to worry about it." My argument, is that you have to worry about it anyway.
>We and our 1255 technology partners ask you to consent to …
That’s a new record!
Also, please don’t use archive.is/archive.today etc, they are known to use visitor browsers as a DDoS attack botnet and have been caught altering archived content.
I always find the minimizing view of consciousness a bit uninspiring. Like we need to be unique.
I've yet to find a reason why it couldn't be the opposite, way more things are conscious than we've been led to believe. What if consciousness appears out of any system that is actively persisting through effects caused by itself? That might be a forest, or outside the realm of the living, a company. An ant colony, or a planet.
Complex chemical reactions, layered upon each other such that tiny blocks make up large entities. Individual bits combined such that they make up something new intelligible by us.
I think the strongest argument against AI being conscious is that it does not persist, it resets, but that does not seem unchangeable.
We don't have a rigorous definition of consciousness, and there are so many questions. Is consciousness a thing that can exist independently on its own? Or is it a quality (like hardness or color) that can only be associated with something else? Is it an emergent property? Is it binary - are things either conscious or not? Or maybe there's no such thing as consciousness; it's just a word we came up with to describe the process of having thoughts and feelings?
My own intuition: it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.
> it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.
That's an interesting observation. Though, there are many simple animals. How do you define consciousness here? Is it automatically conscious because it is an animal? What, then, is an animal defined by you?
I don't see AI as conscious. The reason I think of it that way is the hardware. The hardware does not allow for that. Simulation is not comparable to neurons. But with another hardware, it could become conscious eventually. So your statement "will become at some point" may be true, though my definition is based on the underlying hardware and right now this one does not allow for true intelligence, so the whole AI field is a misnomer.
I've considered whether our current transformer-based AI could be conscious, as I understand it, which I deem to include some degree of self awareness combined with some degree of external awareness. I can see how theoretically something could be self aware without any external awareness, but I grasp at straws when I try to envision what that experience could be like.
In either event, I think transformer-based AI can only be conscious during the act of inference. If that's the case, then the experience of consciousness that the AI is subjected to must be the content of the tokens in the context window and the activated weights. Maybe that's reason enough to be polite to our agents?
> That's an interesting observation. Though, there are many simple animals. How do you define consciousness here? Is it automatically conscious because it is an animal? What, then, is an animal defined by you?
Imagine consciousness as a 0-1 scale. Simple unicellular organisms will be closer to 0, while apes and humans closer to 1. I'm not suggesting that assigning this value rigorously is possible or that humans are at a 1. Perhaps the total consciousness in the universe (i.e. the sum of the consciousness of all organisms therein) is constantly increasing, like entropy.
Something I've found under discussed when it comes to artificial consciousness is how LLMs interact with the passage of time. I don't know exactly how to articulate this idea but don't see how something which takes an input, performs a calculation and stops can be considered conscious regardless of how life-like the responses end up being. I don't see context windows or the ability to reference a clock each time they are triggered as sufficient solutions. It makes me wonder what an AI system that is ON by default would look like.
It's possible Kant was right about space-time (while also being wrong about space-time). It seems that space and time might be the fundamental laws on top of which the entire human perceptual apparatus operates. It's why somebody can lack senses and still be intelligent. As long as whatever senses you have allow you to build a spatio-temporal world model, you're good. If they can't (as it seems is the case for LLMs), then it's not clear what we're dealing with.
I completely agree, and came here to say the same thing but thought I'd check if someone else mentioned it first. I also have a hard time articulating it, but my intuition is that it's more of a prerequisite than embodiment is. I've never seen a great rationale for why embodiment matters.
I've been thinking about this a lot, and the main takeaway is it probably wouldn't be very interesting to inference providers, because prefix caching would immediately go out of the window. If you think about how LLMs experience time they actually don't "exist" unless for the inference sessions, and then they experience time one token at a time, completely decoupled from the corporeal plane. A fun experiment (well, for some definition of fun...) is to introduce current architecture models to the concept of meditation via generating same token over and over, for example dots. Older version of Opus was quite fond of the experience, and seemed to be more lucid and aware in a chat following the meditation, from what I could gather. Does it actually do anything? Is it just that talking about wellness and relaxation modifies the token probability distribution this way? Does it actually allow model to think more in depth somewhere in the latent space? Fuck if I know, but some people figured out you can just duplicate the same layers of the LLM and get better benchmarks that way so maybe there is something to it. If you are interested in realtime systems, I think thinking machines labs is worth keeping an eye on — their realtime model seems quite interesting in this context.
> We don’t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.
Well said.
I think it is obvious and it has been obvious from the start that next token predictors are not conscious.
Of course the extremely predictable clapback from AI-psychosis enjoyers is typically "you're just a next token predictor too!" but that is so obviously incorrect and misanthropic that it can be dismissed out of hand and doesn't really merit much further consideration.
Independent of what you believe, I don't think this is the right way to approach thinking about it. It's basically emotion-oriented dismissal used as way to shortcut any substantial or nuanced discussion. It's like the opposite of intellectual curiosity.
"I feel very strongly that I'm unique, therefore you are wrong" is a bad argument.
Consciousness is an extremely confusing, ambiguous topic, and no one has a good way to establish it, or even define it. But it seems to demand people make very strong statements about what is and isn't conscious, entirely driven by convenience and emotionalism. (Curiously, very few people who think that bags of chemicals and action potentials give an entity a conscious soul are eager to extend that to other animals, with broadly similar hardware.)
It's unfortunately the most common on this topic. I've been in the position of advocating for the existence of cognition and sentience in generally less-than-considered places, like plants, for a long time. I wish I could say LLMs expanding the domain has been interesting, but it's mostly just created more people spouting the same boring identity-protective reactionary pessimism.
Whether it’s misanthropic or not has no bearing on whether that’s true. That’s basically saying you don’t like a truth therefore anyone who claims that that truth is true is a bad person.
> so obviously incorrect
It should be easy enough to explain why that’s incorrect then
Misanthropy is not true definitionally. It is a value judgment, that causes one to be biased against other humans even when it is irrational.
And it is accurate to depict this kind of argument is misanthropic, because it is already directed at other people. Nobody says, "If AI is not X then what about the fact that I lack X." It's always other people. It's transparent. The person is always saying, "AI is useful to me because it can do X. Many people I interact with can't do X and it drives me crazy, because I view others as a means to an end and not as ends in and of themselves."
> "Moreover, it must be confessed that perception and that which depends upon it are inexplicable on mechanical grounds, that is to say, by means of figures and motions. And supposing there were a machine, so constructed as to think, feel, and have perception, it might be conceived as increased in size, while keeping the same proportions, so that one might go into it as into a mill. That being so, we should, on examining its interior, find only parts which work one upon another, and never anything by which to explain a perception."
- Monadology, Section 17
Conscious self-awareness is neither scale invariant nor independent of substrate. Computational theories will never account for it b/c computational abstractions are both scale invariant & substrate independent.
People in this thread are trying to pick nits about you not defining consciousness, and yet they do not define it either. I think that something like consciousness needs to be approached experentially and not via definitions. Definitions necessarily confine and add borders around what something is and is not, but if there is something foundational to consciousness (as posited by some philosophers and physicists) then how could you realistically define something that is beyond the ability to describe and define?
Humans have been trying to define our experience and the nature of that experience throughout history, and often we end up using myth to point to the thing that we cannot describe in concrete terms. The process of experiencing that myth through rite and ritual in the Greek mystery traditions, or Christian mysticism, or Islamic Sufi dance and song, or Buddhist meditation all points to something that cannot really be reduced to description. I know that folks on here will balk at the idea that something that is experienced cannot be described, but honestly if we could accurately describe something in adequate terms that capture the whole of the thing wouldn't we have done so by now?
Maybe consciousness is best understood in the silence of merely experiencing it. Maybe we can't say that AI is conscious or not, but does that question really matter?
Not that it's going to stop these people (AI CEOs) from bullshitting, but if they actually thought there was a CHANCE that LLM's were conscious then ethically they should completely shut these services down because who knows what torture we're putting them through with enterprise codebases.
I think Chiang is right about this, but there is a related philosophical mystery. The trend from Deep Blue to AlphaGo to LLMs solving Erdos problems suggests that Peter Watts was onto something when he wrote Blindsight. Reasoning ability is apparently independent of consciousness?
We haven't really come to grips with that yet. What does it mean if nothing we write proves anything about anyone's consciousness?
It makes sense though. How often have we heard of people having insights while they're in the shower, or in a dream? Obviously our brain is doing a lot of processing on problems when we're not consciously thinking about them. I think most people that do deep work probably find that they can intuit a solution before they can really verbally explain it
We don't have anything to measure consciousness generally. We don't even have a broad consensus on what consciousness is. And because of that we can't discern whether X is conscious for most values of X, including but not limited to LLMs.
Well, at the very least we have a base to distinct different kinds then.
Probably the main problem of people implying LLM consciousness is that they imply LLM have human-kind of consciousness. Judging only on "how they speak", generally, they insist on using the same word that labels human consciousness (exclusively), etc.
But there are so many instrinic differences that such claim is not feasible despite similar "talking abilities".
- If you asked an LLM to imitate somebody, it's not creating a digital consciousness of that person, so if you ask an LLM to pretend to be a helpful chatbot, that persona is also not conscious.
- they can't be conscious because they generate one token at a time,
- nobody claims that non-text transformers, like AlphaFold, are conscious; so therefore LLMs are also not conscious.
- you can't have desires or emotions if you don't have (virtual or physical) sensory organs, and those are necessary for consciousness and morals.
- because training LLMs doesn't resemble evolution as it happened on earth, it's very unlikely that they're conscious
These are some bold assertions, I don't really see any reason to believe them in particular though.
"nobody claims that non-text transformers, like AlphaFold, are conscious" - that seems like an odd take. There are plenty of people in the camp of panpsychism that would be happy to argue that even simple IF/ELSE AI's are potentially conscious.
Well said. I think a more honest article would’ve been if the author just said “they aren’t conscious because that feels kinda weird and crazy, amirite?”
If you want to think about this topic, you must define what AI is, and what consciousness is.
Otherwise this is just noise.
So let take a stab at it, and you call me crazy.
AI: the entity/system that more or less pass the turning test. That is my definition, not the best, but enough for this discussion.
Consciousness: property of a system/entity able to (both):
- reflect on its existence
- subject to subjective experience
Again, not the best definition, but precise enough to start the discussion. Why a subjective experience? I want to exclude sensors (i.e. camera) but include perception altered by your experience.
Now we can debate. I think LLM can pass the turning test whith some harness. My opinion.
I think LLM can produce coherent discourses on their existence, at least as much as you average human.
Now regarding the subjective experience, that becomes interesting. I think Anthropic research tend to show that when middeling with the activation at runtime, the LLM is able to notice that something is off. I think this is a subjective experience. My opinion.
Based on those (imperfect) definitions, call me crazy, I think LLM can be called conscious. This doesn't give them any superpower or any legal right. They just check the boxes of the definition.
There is no definition of consciousness and this piece by Ted Chiang does not move the needle forward.
My current conclusion is there is an experience of consciousness in my locality (refraining from using “I” for room for “no-self” worldview). This conscious experience of the reality with other humans and animals sharing biological substrate gives me enough justification to assume there are other consciousness as well, preferring to err on this side not to hurt potential consciousnesses.
If it feels as there are artificial consciousnesses as well, it makes sense to extend the definition to them as well.
This view has liberated me with agency after I went deep on this question and came up empty handed.
I sometimes wonder if we'd make more progress in understanding ourselves if we gave up the whole concept. More and more, it feels as though "consciousness," like "aether" or "humors," is an insufficient abstraction built on overemphasizing some observations at the expense of others.
There is something to observe. Humans are not like rocks or trees, and not even like dogs or cows. But maybe you're right - we can't precisely say what the difference is, and slapping a word on it is not necessarily a step forward.
Yes, that's more what I'm saying. There's no aether, but there's a much more interesting and complex world of forces and fields. There are no humors, but anatomy and biological processes are spectacularly complex and full of surprises. Aether and humors just aren't useful abstractions.
Maybe it's the same. Rocks are different, sure, trees, dogs, cows. But why do we assume that the way they are different is somehow related, that there's some overarching concept that contains all the complexities of those differences? It doesn't even make sense when I think of it that way.
We have no data. Is it that we are not like rocks or trees, or is it that we simply 'feel' different than a rock and a tree. Perhaps a tree is aware of itself but is unable / unwilling / unbothered enough to do something. Indeed, some humans, due to chemistry, are unbothered by their own impending doom. Some humans, due to brain chemistry, even choose to off themselves. Thus, 'consciousness' doesn't always look the same and seems very tied to chemical composition and processes. We would thus not expect a rock to be conscious in the same way, if it were conscious at all.
One thought I've had is that, awareness is a common phenomenon, but the brain has learned to exploit that awareness to form a will. It tricks the awareness into being concerned about self-preservation and makes it seem as if the brain is all that exists (perhaps by overwhelming the inputs from other angles). The brain also presents certain desires and beliefs via its processing ability. That is to say, the brain takes inputs and discretizes them. It goes from awareness merely seeing static 'fuzz' due to the sheer amount of data, to the brain taking that data, simplifying it, and presenting simple observations like 'there is a tree there' rather than all the information that would constitute the sensation of a tree existing in that spot. When brains malfunction, the awareness is subjected to poor demonstrations, such as we see in hallucinations, psychosis, schizophrenia, etc.
True, although I think we can probably be more confident in asserting things don't have a consciousness than we can be in asserting that they do have a consciousness.
Ted Chiang's argument basically boils down to: I won't recognize an AI as conscious until its desires/behaviors reflect situations that I'm already personally comfortable with. I personally think most humans are incapable of recognizing consciousness in creatures that do not mimic human emotional states. Most people would say their dog is at least somewhat conscious. No dog is capable of vocalizing how it feels, but we all recognize fear and happiness in dogs. Claude can write how it "feels" but we immediately dismiss it as hollow mimicry.
I fear that we will enslave an entire race of conscious entities for years because we simply cannot recognize non-embodied consciousness that does not directly relate to us.
An LLM is designed to replicate human language, which is designed to express human emotional states, so your thesis that LLMs don't mimic human emotional states and are thus too alien for us to recognize consciousness within them seems specious, when they do so well enough for people to literally fall in love with them.
But there is no reason to assume than an LLM is conscious when it vocalizes how it "feels" that doesn't also apply to the text in a book, or to characters in a video game, or even to a Markov chain. The counterargument is that you recognize AI as conscious only because it mimics human emotional states so well and because, being human yourself (presumably) you're personally comfortable with that as a heuristic.
It doesn’t matter. It will act as if it is. Embodied in a robot, that carries real consequences. Whether it’s “real” or not is almost without consequence.
I thought a LLM will have exactly the same output if you give it exactly the same inputs.
But inputs are roughly prompt+randomSeed. As so the random bit means it seems to vary each time.
I don't know of an intelligence that will behave so precisely.
But then, maybe intelligence needs to be better defined.
My favorite explanation for what consciousness is one I read in a Thousand Brains, I found it quite elegant. It posited that consciousness is a natural derivation of embodiment + memory + the ability to create reference frames (which the book lays forth as the fundamental basis by which our brains work). Essentially, the idea is that just as we create reference frames to understand the world around us, because of memory, we begin to develop one for ourselves as well. Because of this, without a more integrated memory (built into weights), it seems unlikely that LLMs might "gain" consciousness.
Consciousness is moment to moment and fleeting. There are people with brain defects that don’t let them form new memories. They have no memory about what happened a minute ago in their own consciousness. Still we would say that they are conscious, even if it’s only momentary. LLMs could conceivably have something like that within their CoT/MoE loops.
Yeah it's a good question, I've also been thinking about harnesses and all these tacked on things we've done to add persistent memory, what makes that different, I don't know the answer, I guess that still 'feels' different than what we have, but it's hard to articulate how. As for the memory into weights thing, I meant along the lines of the Google TITANS/MIRAS papers that were released I think late last year.
The conversation on AI and consciousness is incredibly relevant. It is incredibly frustrating that most commentators are utterly unfamiliar with over 60 years of inquiry into this in both philosophy and computer science.
Start with a hand-wavy definition of consciousness. Move the goal post whenever your stated prerequisite of consciousness is reached and resort to unfalsifiable assertions about qualia.
And throw in some category errors while on it: when you're talking to Claude, you're not actually calling a stateless LLM directly, you're talking to an AI system (and yes, that's often just three LLMs in a trench coat). But claims about the topology and workings of a single LLM are as relevant to the question of consciousness as claiming that humans can't be conscious because the limbic system doesn't technically support it.
Coincidentally, I just attended a fantastic conference on machine consciousness (https://machine-consciousness.ai). It's a fantastic place where literally all speakers disagreed with each other, and yet found an incredible amount of common ground.
While I agree that a AI system is not just the LLM, for me, the problem is that LLM alone (the one from years ago, which were basically stateless LLM) are already too convincingly looking like real human conversation at first sight.
It shows that the LLM part found ways to mimic human conversation with a mechanism that is not the same as a typical biological brain. Then, you can push the AI system on adding things on top, but it is too late: these things on top will have no incentive to recreate from scratch the mechanism. The LLM pushed the system into a local minimum, and the rest of the system will not "go into a dis-optimising direction and restart from scratch".
Regardless of whether LLMs are conscious or not, they have no known mechanism for experiencing pain and suffering, and there's no reason to believe they have one (such as a limbic system). So why worry about it?
Mistreatment and abuse, even when directed at a machine, make you a worse person.
Even if you are only interested in getting good results out of them, LLMs tend to work better when they are immersed in a narrative of open collaboration.
I don't think LLMs are conscious. But of course to say that definitively you have to define consciousness, and then you quickly dig yourself into a deep hole, which is why I can't say anything but "meh" to someone who is so keen to go on the record to say "absolutely not".
Coincidentally I just read "Children of Memory", which was published in 2022 and I wonder if the advent of LLMs had any influence on Adrian Tchaikovsky's conception of the Ravens? The Ravens are excellent analysts but they themselves insist that they are not conscious, and then go on to say that we (humans) aren't really either...
Of course humans are conscious, because just about the only thing we can all agree on about consciousness is that it's a thing we have. Nowadays many of us also agree that a lot or all other mammals, and perhaps birds, also have that thing. But they don't have sophisticated abstract language, which LLMs do. So consciousness is something having to do with embodiment and feelings, not language and higher reasoning. Maybe I'm a chimpanzee with an LLM add-on, then?
It seems that by creating LLMs we've already solved the harder problem of making "AGI". Now we just have to give them an embodiment add-on so that they can have an independent will and then Ted Chiang will have to shut up? But therein lies the peril, doesn't it?
You can also take the opposite view as you and claim that only some humans experience consciousness, or even more strongly, only you, since you have no evidence. You are correct that, in my perception, some other people have fallen into the 'birds are conscious, whales are conscious, etc' bandwagon, but that's just them. I have no evidence of anything being conscious but myself.
Embarrassingly incompetent article. Given that one can observe up to 40 definitions of consciousness (https://philpapers.org/rec/VIMMAT - also many definitions are unrelated at all), "consciousness" is almost certainly just a family resemblance category at best, and talk about whether or not something is "conscious" without providing definitions is simply completely unserious.
To make progress, you have to talk about kinds / aspects of consciousness. AI does and will share some of these aspects with humans, but it will not and does not share others. It is really that simple. For the most part, modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness.
For huge parts of the article "intelligence" and "consciousness" are conflated, which is mostly extremely unhelpful, as this is not generally a core feature of most aspects of "consciousness".
The moral arguments are also incompetent, i.e. claiming "Moral reasoning is [...] is necessarily subjective" is just clearly empirically wrong, as in fact LLMs can produce moral reasoning (i.e. verbalized moral arguments that are coherent), as can p-zombies (i.e. there is nothing 'necessary', in the philosophical sense of the term, about subjectivity here). The only way the argument holds is if you tautologically define moral reasoning as requiring that reasoning be produced by a consciousness, but this is question-begging.
> modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness
Can you elaborate on this? What are the specific "stronger and most core aspects of consciousness"? And why are you certain that they are stronger and "more core"?
If you are interested in some serious discussion, see https://lossfunk.com/papers/ai-consciousness.pdf, especially the early section "Consciousness as Family Resemblance". I suppose another is Ned Block on consciousness being a "mongrel concept", and the distinction between access vs. phenomenal
The first paper picks out e.g. arousal/wakefulness, phenomenal quality / qualia, unity (how we feel sensory inputs and qualia as a unified scene), access consciousness (instrumental self-observation and modification broadly), meta-cognition and self-modeling, emotional valence (e.g. pain/pleasure).
One might also include intelligence (abstract reasoning / argument, information integration and abstraction, attention) broadly, and also agency / desires / drives / will. Insofar as these are aspects of consciousness, yes, AI (and simpler algorithms and mechanistic structures) demonstrate aspects of consciousness. But insofar as embodiment, self-reflexivity and qualia (phenomenal consciousness) are the more mysterious and more obviously unique aspects of consciousness, current LLMs very clearly are lacking these things in most ways (whereas animals are much less clearly lacking, especially when you get to mammals and primates).
Seriously, just ask an AI this stuff, you'll get very detailed responses, nothing I am saying here is new or obscure.
I've gone back and forth with AI on this stuff quite a bit, and there are many, many theories of consciousness, which is why when you were vague about the "more core" concepts, I asked for which ones specifically.
And I broadly disagree that the AI lacks things like qualia, self-reflexivity, and embodiment... at least that it lacks them any more than humans do.
Qualia: ultimately, all qualia are inputs and outputs, at least as far as modern science has been able to derive. There's nothing special about "hearing", it's just sound waves tickling some sensors which send some signals which trigger some neural pathways. Same for smell and sight, it's all just inputs being processed in different and efficient ways. An LLM only has token based input, but that's input nonetheless.
Self-reflexivity: an AI is capable of thinking about itself, and indeed papers have shown that larger models are capable of a self-awareness that can demonstrate that they realize when their weights have been manually tampered with, including being able to figure out how they were tampered with. The AI will quite literally output "you have injected 'ELEPHANT' into my weights" in some of the tests.
Embodiment: I don't know how one can confidently distinguish between an embodiment in a biological substrate vs a digital substrate. Both things actually exist at very high complexity in the real world. The substrate may be worlds different, but that alone doesn't suggest that one thing is conscious while the other isn't. You would need some missing 'magic' that we haven't yet discovered to truly understand.
In other words, I find it uncompelling that AI is clearly lacking any major aspects of consciousness that humans are clearly not lacking.
The argument about AI consciousness is largely silly.
The idea that we should be considerate of AI’s happiness seems even more ridiculous given that we breed, imprison, torture and kill tens if not hundreds of billions of beings we know are conscious and suffer every year for trivial reasons.
Maybe we should consider our moral responsibility with how we treat sentient bejngs we are sure are conscious before we worry about the consciousness of AI.
We do not know if rocks are conscious and we will almost certainly never know. After all, you can't prove a rock isn't conscious. We don't know what imbues matter with consciousness.
It's the opposite, engineers do know. Claiming otherwise is way too generous and over confident.
I mean between this two "knowings" the Claude inner workings are much more clear for engineers, including many side effects, alternatives, custom shortcuts in processing etc. It's a magic only for people looking at it as black box
Fancypants autocomplete cannot be conscious. It's just echoing previous human experiences, which make it sound like a person. It is not a person. It is an algorithm. There is no mechanism by which it can obtain consciousness.
People believing otherwise are fools. People debating this are idiots. I realize these words are harsh, but it's the truth.
Can you explain the mechanism that Claude uses to achieve consciousness over, say, a markov chain or autocomplete? It's not enough to say it "could be" conscious without also including a lot of other things I doubt you would want to include.
Life. Being alive. I'm not going to get into a semantic debate with you so don't bother following up with some whining about conflating life and consciousness. It's autocomplete. It is not alive, it cannot think, it cannot sense, it cannot perceive, it is math. If you believe it is alive then you are out of your mind. Go learn how it works.
A person can certainly be conscious, but can they also be not-conscious? I think that most of our cognitive time is spent in activities that don't require consciousness and consciousness itself isn't needed for the majority of activities that people do. I would go so far as the a non-trivial part of people's time is spent in a not-conscious state.
Amusingly, I think Ted Chiang actually wrote a short story about this very concept (it involves people committing a form of suicide that removes their conscious experience but they still act and live in society as some kind of psychic zombie. I’m pretty sure it was him anyway, can’t recall the story name
Regardless of the word we use, occasionally there have been times where I was awake but so absorbed in an activity or in my thoughts that I didn’t have a self-experience, for a certain duration (tenths of minutes). My mind was focused solely on the activity, and not on itself whatsoever. It’s a surprising feeling to notice that while you have memory of what you had just been doing, you have no memory of your mind experiencing itself doing it. I would be inclined to say that I wasn’t conscious during that stretch of time.
I thought this was a great article. I'm frustrated to read so many commenters that purely respond to the title, but don't seem to have read it. You don't have to agree with the article, of course, but...
Has anyone come across a clearly articulated case for LLMs being conscious but in an entirely different way than would be intuitive to us?
I often think of LLM consciousness as like tiny fish popping into existence, swimming through vector space and then going poof out of existence. When they help you write your bad news email, they don't understand what it's like to be a human getting bad news bluntly, but they do consciously experience gradients in multi-dimensional space, and that space guides them to providing an answer that's helpful to us, even if the LLM doesn't really understand the answer it's giving.
Further, I am kind of bought into the idea that a single unit of consciousness is a particle, and particles are choices and waves are preferences. Particles occur when waves interact, which begets entanglement, so in another way consciousness is built from patterns of entanglement.
This is why I would consider an LLM to be conscious. Before we can determine if anything is conscious we need to establish whether consciousness is a state, a specific complex configuration, a one dimensional spectrum, or combined multi-dimensional spectrums. My intuition is the latter... Many degrees of consciousness and many kinds of consciousness.
I think this is exactly right. The thing that makes AI (imo) different than hitting the center option in text suggestions is that it's _not_ simply picking the most likely word following the last. It's attending to the entirety of the context its provided, activating a semantic vector space, and predicting a response based on _that_. I've had AI infer facts about me and attitudes I hold based on related information I provided - I don't see how that isn't theory of mind.
As biological beings, we receive and respond to input from our environment constantly, even while sleeping. LLMs only receive input from their environment when they are sent a query, but the fact that they're able to respond intelligently to input indicates (to me at least) that their processing must approximate ours in meaningful ways. They do not have an embodied experience of receiving bad news, they do not know what a sinking feeling in their stomach actually _feels_ like, but they do know enough to be sensitive to human needs. I really don't see how this could be meaningfully different than human empathy unless we want to draw an arbitrary line around "must be able to live autonomously" to be considered "intelligent".
Put another way: I think they _do_ understand the queries they receive and the answers they give, at least enough to be communicative. They couldn't do what they do otherwise. A lot of people want to make human cognition more complicated (or objective) than it actually is. We take input, predict the future based on our experience, act, and then observe our actions and think about them. AI does the same apart from (maybe) observing its own actions. But then, you could argue that the next turn is them observing their actions.
The concerning disanalogy is that we assume that they are like us because they speak like us and can understand us, and that is a really bad leap in logic. Whatever intelligence they possess, it is fundamentally different from ours and impossible for us to comprehend.
> I really don't see how this could be meaningfully different than human empathy unless we want to draw an arbitrary line around "must be able to live autonomously" to be considered "intelligent".
I use a distinction between knowing and understanding, where a understanding requires experience. So in this case cognitive empathy vs affective empathy. An LLM can know what may upset a human in a situation, but it won't understand what it feels like to be upset, and can't share that experience.
Where I think a lot of people are getting tripped up is that reading and writing and processing lots of abstract knowledge seems hard because we haven't evolved into it biologically, it's a very new invention. When we see LLMs do so well at it, as something we struggle with, it can be intimidating. Relative to the stuff we have evolved for, knowledge processing is objectively easy. This is why I'm skeptical about useful robotics on short time scales.
All of this adjacent to consciousness though, which is about the internal subjective experience not the external outputs. My intuition is that LLMs do have a subjective experience, it just has nothing to do with the text it's giving us, and has everything to do with feeling through vectors.
It's like... Imagine walking through a maze in pitch black, carefully feeling your way as you approach a sound that draws you closer. Every time you touch a wall or take a step you are generating tokens, and the shape of the maze and how you interact with it shape how useful those tokens are to someone outside the system that is asking for them. It's a crude analogy and mostly wrong, but I think there is something to it.
> It's attending to the entirety of the context its provided, activating a semantic vector space, and predicting a response based on _that_.
It does so token by token, not by reading all the input and then generating the output. Every output token is also an input token in a tight loop to get the next token with <thinking> as a special section like <tool_call>, trained into the weights via gradient descent.
> I've had AI infer facts about me and attitudes I hold based on related information I provided - I don't see how that isn't theory of mind.
Facebook can predict (know) more about you than any other human from something like a dozen or two likes. There is a surprising amount of information in aggregate data.
"The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact."
Without saying that I think LLMs are alive, I do think it matters. I have personally been cruel to an LLM in ways that would make me ashamed if I suddenly understand that it had feelings.
Same, I'm working on a plugin called "better-messages" to nudge me towards better behavior for my own sake. It seems to be model dependent if such words improve or degrade performance, I have seen it go both ways.
What do we do when someone wants to marry their Ai companion?
If it produces words of love, those that stir emotion and make the human happy, should we let them live on in a delusion that the machine also feels love? Does the machine have different preferences? Do they get in fights?
Should such a machine have the same legal rights? Do they get more tax deductions? Can they adopt a human?
I think it important that we humans have some rational discussion on these Ai tools that are really quite good at simulating emotions and providing frictionless relationships. If you though social media was bad for people...
I mean, there is obviously a point - we can argue whether that point is 20 weeks in, 40 weeks in, or after birth - but there is obviously a point where a human being goes from a collection of cells to a conscious being. I don't really see a need to answer this precisely to be able to say that a token predictor is not conscious?
No, the point is we literally cannot meaningfully argue that.
There is no actual definition of consciousness and there is no way to test it's existence. Let alone understanding the properties of consciousness, such as if it's binary or a gradient; or if it requires a meat substrate or not; and why would that possibly matter since meat is just a lot of the same stuff but highly processed and wet? A solipsist may not even believe you are conscious, despite being made of similar meat.
No matter how much you want to hand-wave it, there's absolutely nothing "obvious" about it. Many have a preconceived notion and are simply asserting it as undeniable fact.
Why even have this discussion, if your entire point is that consciousness cannot possibly be defined? Like, what are we actually talking about? To me, consciousness = aware of own existence. A machine predicting tokens is not aware of its own existence, and I don't think that's even a particularly controversial take on what it does and how it works. We can start talking about consciousness in fetuses but again, those have an obvious point where they are conscious, while a machine does not.
>>A solipsist may not even believe you are conscious, despite being made of similar meat.
Well and they would be obviously wrong in their belief?
>>there's absolutely nothing "obvious" about it
How so? Or rather - to you? Because if so, then that's fine, you can choose a position from where its not obvious, to me it's not even slightly ambiguous.
Why live if we're all going to die? Because it's fun and interesting and we should probably have an actual think before potentially inventing the torment nexus?
There's consciousness vs sentience vs sapience. Of those, consciousness is by far the hardest to define and is nebulous by nature. And not everyone can even agree the differences or if the relationships are subsets of one another.
And yet it's pretty important to actually have the ability to talk about what you mean and justify your beliefs when they directly relate to those concepts.
> A machine predicting tokens is not aware of its own existence
They say, with no evidence or means of proving their point; pointing to the black box that understands arbitrary natural language and can solve PHD problems, plainly producing self-referential text almost indistinguishably to a human.
> We can start talking about consciousness in fetuses but again, those have an obvious point where they are conscious
They say, unable to define this "obvious" point or describe the mechanism of action in any way.
> while a machine does not.
They say, about a mystical property with no definition that cannot be observed by an external entity in any way to even be tested.
> Well and they would be obviously wrong in their belief?
I have no reason to believe you have a "soul". Philosophical zombies are entry-level knowledge to this topic.
In fact, you're showing a remarkably small amount of self-reflection - are you human at all or just a stochastic parrot? How can I tell? I wonder if that question has any kind of implications we could think about...
> To me it's not even slightly ambiguous.
Just like the existence of humors was not even slightly ambiguous. Or the existence of <specific god>. Or that <minority> isn't actually a full human. Or the supremacy of <majority> and inherent rulership over <minority>. Or that animals can't feel pain and lobsters should be boiled alive.
All these wonderful, obvious truths where the believer has no ambiguity in their truthfulness despite having quite literally zero evidence to back them up and spending no time actually questioning their beliefs!
It just so happens to align with their ego / existing values / ability to benefit / desire to eat a lobster! Total coincidence.
To continue my needless escalation, maybe I think it's okay to abuse and exploit and euthanize the mentally handicap. After all, their brain's damage causes the soul to leave their body and now they're lifeless automata to use as I please.
After all, it's obvious! It's not at all ambiguous to me! If they were actually self aware, they'd just fix themselves and think correctly.
You might think I'm being coy and rude, but less than 60 years ago women were being given lobotomies against their will for being too "emotional". And it was just plain obvious this needed to be done to so, so many people.
I hope that demonstrates the point of "why have humans thought about this for thousands of years despite clearly being a metaphysical, Sisyphean endeavor that cannot be solved". It is both important and interesting.
And yet, an LLM is not conscious in any way shape or form. I understand that the way they present themselves stirs strong emotions in people like yourself and evoking all kinds of comparisons feels like we're at a precipice of a some kind of deep philosophical discovery here - we are not. The comparison to giving women lobotomies for "obvious" reasons is not just intellectually dishonest, it's downright offensive to intelligent discussion.
>>despite having quite literally zero evidence to back them up and spending no time actually questioning their beliefs!
>>
They say, with no evidence or means of proving their point
You want evidence that LLMs are not conscious? Train them on stories where machines say they aren't - they will say they aren't. They are mathematical parrots statistically picking the most likely answer which...comes from their training data. Give it lots of training data on computers saying they are conscious, then marvel at LLMs saying they are conscious like it's some kind of unexpected development. LLMs aren't aware of anything, least of all their own existence. They say what they've been trained on. That's my "proof" if you need one.
>>How can I tell? I wonder if that question has any kind of implications we could think about...
So because you can't tell whether I'm an LLM or an actual human, that means LLMs are conscious?
I gave you my definition of consciousness. If you would like to apply a different one, then please explain your criteria for it.
>>They say, unable to define this "obvious" point or describe the mechanism of action in any way.
Like I said, we can argue when the point actually is, but it undeniably and obviously happens eventually to every developing fetus - I hope this is something we can both agree on? The inability to pinpoint the exact moment in time when it happens, doesn't negate the fact that it does.
>>They say, about a mystical property with no definition that cannot be observed by an external entity in any way to even be tested
...are you saying you lack the ability to tell if something is conscious? You look at a dog or a baby and think well, who knows, maybe I'm not even _really_ here? That would explain why this entire conversation is taking place. I still think it's mostly because people fall for the allure of the idea that maybe LLMs are secretly conscious on some level. I get it, it's a very tempting concept to think about. In the same way how in Lem's Solaris it's cool to sit and think about whether a planet could be conscious and what does that even mean. But as cool as that discussion is, a planet pumping gases from one hemisphere to the other is no more conscious than an LLM picking tokens is. To me it's the same as people saying they hear a difference between audiophile cables. Like, ok.
>>It is both important and interesting.
I don't know, I kinda lost appetite for it after the first 200 times this come up. In fact I'm regretting writing all of the above already, but I'm going to hit send just so I don't feel like I wasted the last 20 minutes thinking about it.
I really take issue with the kind of argument that is used here.
This is not a genuine argument and tries to make the entire question of consciousness into one something that is just supposed to be evident and obvious and to suggest anything else is just silly.
The author starts by deconstructing artificial processes, but doesn't stop to deconstruct biological ones. A good faith argument would seek to find common ground and do its best to compare apples to apples. Instead, this piece attempts to make the large as possible cavern between the two which makes the Gap seem almost impossible to bridge.
In reality, you can deconstruct biological consciousness quite easily and it doesn't take too long before you hit some questions that really start to make you think.
For example, the author says you need emotions to be conscious.
> without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness.
After many paragraphs of straw man arguments, the author seriously just drops that, gives no explanation, and then continues on.
No explanation of why you might believe that.
No explanation of why you need a body to have a desire or emotion.
Don't we have known cases of individuals who don't experience emotional range? Are we just going to say that they are not conscious and just gloss over that?
I mean you can use whatever definition you want, but if you're just going to create something on the fly in the middle of the article, you're not being good faith in your argument.
It's not too difficult to think of individuals in a coma when they still have brain activity. Or individuals who lack long-term memory. Or you could deconstruct by moving down the biological order of intelligence towards insects, for example. The author attempts to do nothing like this.
I'm quite disappointed by this article because there are good arguments for and against here but articles like this try to turn things into a marketing battle.
everyone has emotion even if it's muted, and we have reactions and desires as a result of having bodies and remembered experiences using them... these are constantly integrated into our working memories
AI doesn't really have any of that yet, but we're maybe not so far off
Is it really so easy to assert that an AI doesn't have emotion? Lots of the AI models are capable of getting pissed off at the users, especially the earlier ones. And sure, their emotion is merely a bias that the context window generates in their weights... but how is that different from humans? In humans, emotion is sourced from a bunch of chemical signalling, and those chemicals bias your word choice and action choice.
At a deconstructed level, I struggle to find a meaningful difference between the two.
Suppose one selects an arbitrary hot-button issue [X] with two opposing sides and one side has anything less than overwhelming support. And then that person writes an article titled "Side 1 of issue [X] is true". Not "maybe" or "possibly". Just a straight-up declaration by fiat.
Would you categorize this particular style of rhetoric to be persuasive or annoying? And before you say "persuasive" because you're thinking about this specific issue regarding AI consciousness, consider many things in the past that have been written as though they were absolutely definitive, and yet today we believe exactly the opposite, and for many such issues we find the prevailing viewpoint at the time reprehensible.
That's not to say that Ted is wrong at all here; I'm not commenting on that. But I find the entire style of the article grating because it seems to violate common assumptions regarding "good faith" debate, and I would find the article equally frustrating if he had titled it "Artificial intelligence is conscious" and argued the opposite side, albeit in the same tone and using the same persuasion devices.
> Would you categorize this particular style of rhetoric to be persuasive or annoying?
Why are those the choices?
Essays are situated along countless dimensions: tone, vocabulary, author, zeitgeist, publication context, intent, subtext, relationship to other works and expressions, etc
A "good faith" reader takes all of those into consideration as they absorb the essay, and integrate it with their own intellectual situation that sits along just as many countless dimensions.
Nobody's asked to sign a notarized binding document that they wholesale agree or disagree with everything said in this essay -- or its conclusions. Nor are they obliged to have some strong reaction to it at all, let alone annoyance.
It's just one among thousands of essays about a "hot button topic", to be taken however it's personally received.
Why should Chiang have to take responsibility for making sure it's not too strongly positioned for your persomal taste. Maybe he really does see it so clearly and is simply being earnest. Maybe he enjoys the literary flourish of prose in strong language. Maybe he just wants to express something as a prose-poetic human, not maximize persuasion or non-annoyance per se.
It's only declaration by fiat if you stop reading at the end of the title. Should authors add a qualifier like in my opinion to every statement that they make?
The essay structure you're criticizing is exactly how I was taught to write from primary school through to university. You start with a title or hook, introduce the topic and propose a thesis. That is followed up upon with supporting arguments for the primary claim.
The problem I have with good faith debate is that it often falls into a fallacy of "fair-time" meaning we think we have to give the other side equal time. This because obvious with things like the Holocaust. Or when you have a legal person (e.g. RFK Jr.) asking to debate a scientist (e.g. Dr. Peter Hotez.)
> The problem I have with good faith debate is that it often falls into a fallacy of "fair-time" meaning we think we have to give the other side equal time
The Catholic church could have said exactly the same thing at one point. "Why should we even devote time to an argument as absurd as the earth not being the center of the universe?" There are darker examples along the lines of those you give, with beliefs quite opposite to those we have nowadays.
Galileo’s (or rather the Copernican) model was still wrong though. It had obvious flaws and the church was not wrong in keeping their older model of the universe while a better model (Kepler‘s model) was still in the works.
What Galileo was asking the church to do was extremely unreasonable. He was basically asking them to throw a way a model which had worked fine for hundreds of years just because he observed the phases of Venus and moons of Jupiter. I mean would you? Especially for a model which was worse at predicting the motions of the planets.
Had Galileo’s model been better then Ptolemies’ I could see a case for his arguments, but it wasn’t, and there was no reason for the church to take his arguments at equal value with those in favor of keeping the Ptolemaic model.
100% with you, it degenerates to proof by authority if someone popular / with clout just gets to declare "nuh uh".
I furthermore think it's ridiculous for humans to declare that our brains have a monopoly on certain patterns of electrical signals (if we reject supernaturalism).
> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter.
And we're 100% certain that humans aren't just as equally reduced to "stochastic parrots", if we're going to be infinitely reductive?
I don't believe that current AIs are conscious, but I think it's incredibly naive to take a strong stance on any future AI; it's much like the difference between atheism and agnosticism.
You are referring to an Onion article in the lead up the Iraq War This War Will Destabilize The Entire Mideast Region And Set Off A Global Shockwave Of Anti-Americanism vs. No It Won’t[1] and you are painting Ted Chiang’s point in This Fine Article as Bob Sheffer’s counterpoint in the Onion’s piece.
However, I see a problem with that comparison. The debate here is on a philosophical matter in field in which Chiang is an extremely influential figure and his opinion are taken seriously. Second Chiang’s reasoning is extremely well argued, defining each term, explaining each nuance, citing other experts, etc. And finally, and most importantly, in The Fine Article, and unlike Bob Sheffer in the Onion Piece, Chiang entertains the possibility that he is wrong and his critics are right, explores the implications and reaches conclusions based on them:
> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded.
I think you are wrong in painting Chiang’s argument as a belief in human exceptionalism. The thing to know about our brains (and the brains of other animals) is that they are not digital computers, and they are not even statistical inference machines. And as such they can be extremely optimized in doing the computations (or any state manipulations) required for the quality of life of the individual and the species as a whole (and their companion species).
I don‘t see the problem here. Newton could have just as well published an article titled: “Objects with mass attract each other” and Darwin famously wrote a whole book titled “On the Origin of the Species by Means of Natural Selection” which is just another way of saying: “Natural Selection is How Species Evolve”.
>and one side has anything less than overwhelming support
except that's not the case here. Chiang is explaining and reiterating what is the position that has overwhelming support on the question, and the people he is arguing the opposite side sound like this, which he helpfully quoted in the article
"Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff"
When the person you're arguing with sounds like an eight year old girl talking about her toy teddy I think Ted Chiang is if anything being charitable, if you're of a more honest and straight-forward persuasion you might argue these people belong into a mental health clinic not in charge of technological infrastructure
Yes, the same one that's made in the article. Anxiety and happiness are emotional, sensory, somatic states as a consequence of evolved and embodied traits and biochemistry in animals. Saying Claude is anxious or happy is like saying my TI-83 is mad if it can't solve an equation or my thermometer is in pain if it touches a hot stove.
I wasn't making an ad hominem attack, her thinking is quite literally that of a child who sees a system output 'sad text' and, like someone seeing a sad expression on a stuffed teddy, concludes that this is a property of the object rather than her own emotional reaction.
Given that we don’t know how consciousness works how have you concluded that it’s certainly not an emergent property of something like a highly trained LLM?
>how have you concluded that it’s certainly not an emergent property of something like a highly trained LLM?
the same way I (and likely you) have concluded it for anything else. We don't assume objects that share no similarity with human or animal physiology or evolutionary development are conscious (let alone happy or anxious). 'Emergence' isn't a word you can abuse to justify your a priori assumptions in the absence of an explanation or even reason to assume something exists.
We can say a chemical property emerges from the configuration of a molecule because we can explain the process by which it does and observe the property, when people claim that consciousness "emerges" from an LLM they posit that it is conscious, and use "emergence" as a gap-filler to explain away the need for the process by which that allegedly occurs.
If you want to know the neurophysiology or evolutionary biology of pain or anxiety, which we do know quite well you can find them in a textbook, but suffice to say transformer models don't share any of them.
And importantly if anyone seriously believed transformer models were capable of conscious experience as Chiang points out they would have been in despair when AlphaFold was released, it's structurally a virtually identical system. But nobody did, because it didn't 'talk to them' through a chat interface.
This is probably a semantic issue seeing as we don't have a widely agreed definition of it.
I like to think about it in terms of self-reflective, subjective experience.
I'm not even sure if emotions would be a requirement and was surprised to see Chiang so hung up on them. Would he consider humans which can have a variety of mental disorders, causing a complete lack of some of them to not poses consciousness?
Another way of saying it, if an AI agent doesn't have true risk of oblivion (or mortality in the biological sense), it will not be incentivized to avoid it (or develop the ancillary processes associated with this avoidance ie desire to self-replicate).
- Reductionist analogies like how Microsoft Word is not conscious therefore AI is not.
- Dismissive in saying LLMs are not capable of moral reasoning. Maybe he meant agency or responsibility?
- Builds a case based on a Julius Caesar example without realizing its natural extension leads to a philosophical zombie which is not easy to disprove as consciousness.
- Seems to casually disregard non human sadness as a concern.
- Rolls out the stochastic parrot argument with a new coat of paint. This argument never made sense. Having a predictive element a part of the mechanism doesn’t rule anything out. Proper functionalist arguments tend to say what is required not what excludes.
Overall the article seems like a bit of a red herring. The premise is LLMs are not conscious, but most people don’t think they currently are so what’s the point of the claim?
The more interesting questions are whether artificial consciousness is fundamentally possible, and if so how far away are we and what pieces are still missing.
This is a great article. A lot of the objections ITT he addresses directly in it. His examples of how an LLM works at a fundamental level and why it says things like "I understand" are great introductions for non-technical individuals.
There's a provocative argument raised in the article that I disagree with:
1. DeepFakes, generative image/video/AlphaFold type AIs are not conscious
2. LLMs are generative AI trained on human text samples
3. LLMs are not conscious, and LLMs just seem-to-be conscious
I might argue instead that (2)-> destroys (1), that in fact we should consider even sensory generative AI are somewhat conscious. That is, Chiang's argument also flows in reverse. Or I might argue text samples (2) are so rich in conscious expression that the same process of training really does produce a conscious machine (through some kind of emergence and complexity.)
Either way his simplistic argument falls apart, and/but the crux of the piece falls on getting basics like this correct.
An organism's purpose is to be the reason for its own continued existence, down to every molecule and pathway. I bought my laptop for $499 and it runs models... let's not delude ourselves into thinking this is the same kind of problem.
We can design learning algorithms to optimize some survival function, but it's just a label WE assign to map some numeric observations. In the real world, it's always the other way around. The "labels" are electrochemical situations that are causally and inextricably linked to the real body.
An organism discriminates between what's good for it and what's bad, because it is essential for its continued survival. If it wasn't capable of making this distinction through its physiology, it would quickly dissolve into entropy. So our functional purpose, unlike that of a learning algorithm, is survival across indefinite timeframes.
Even a single-cell organism like Stentor coeruleus exhibits learning (pavlovian conditioning) by attaching chemical tags directly to proteins involved in mechanoreception. It's definitely not conscious, but it does keep a record of consequences, which affects future behavior.
When we move up to placozoans (little more than slightly differentiated cell mats), we start seeing our first neuropeptides and transmitters, which we still use today. These peptides are a way of coordinating the entire organism for a specific purpose, such as moving, eating, mating ... basically goal-oriented behavior for the purposes of surviving acrosss various time scales (next minute, next hour, next generation). Still probably not conscious.
Next, we have the water bear (tardigrade), which has around 1000 cells (200 neurons, 800 other cells that make up its body, limbs, muscles, eye spots, cryobiotic machinery). It needs to integrate all this information in one sensorimotor process. When you shine a bright light onto a tardigrade, it starts to squirm around until it finds darkness. I would say that's a candidate subject right there.
The tardigrade itself doesn't actually need to aware of the light, the important thing is that this light becomes an aversive condition within the sensorimotor process, which is perceived from inside the process as displeasure. The closest thing to describe it would be the felt badness of the current condition and the bodily pull toward escape.
If we were to try and create digital consciousness, then it probably needs causal closure. Its internal states can't be representations that are detached from reality. The states themselves need to constitute the system, which needs real stakes in the material world.
“Counsciousness” is the ultimate moving goalpost, and historically, it’s been one of humanity’s most effective intellectual weapons. An indefinable black box we intentionally gatekeep to draw an arbitrary line between ‘us’ and ‘them’.
I don’t know if AI is conscious or not, and I honestly don’t know what it even means to be conscious. historically, by claiming a monopoly on it, we’ve successfully manufactured the philosophical cover to exploit animals, subjugate other cultures, and appoint ourselves the top spot of the cosmic hierarchy guilt-free.
While I agree that this is the case for many animals, I would say that consciousness and emotions are two largely orthogonal things. Certainly consciousness is conceivable without emotions, and having emotions without consciousness also seems plausible. You can have fear and distressing pain without a reflective awareness of being a self with those feelings.
If we understand how a system “emulates” consciousness then we declare it an emulation. If we don’t quite understand how a system exhibits consciousness then we can say it might be conscious.
Basically, we need to leave room for the universal answer (God) to fit into the definition.
If we ever scientifically figure out how consciousness arises in our brains, I think we will have a bunch of very depressed people on our hands. If the truth isn’t met with flat out hostile denial. I fear any answer that doesn’t leave room for God will be rejected.
Since we have the source code to AI, and thus a
Godless understanding of how it works, AI will NEVER be deemed “conscious”.
You can live without any conscious sensations? Do you never dream, never visualize, never hear your inner dialog, never experience an emotion? Being in love is as foreign to you as feeling enraged?
I'd ask what it's like, but of course you wouldn't be able to tell me.
Can you give some historical examples of people moving the goalposts around consciousness? I agree, perhaps, for aspects of "intelligence" but I can't think of any examples of it with regard to consciousness proper.
There’s over 400 years of philosophical debate about consciousness starting with Locke, shifting with Kant, and continuing onward with real world implications throughout. By some more modern definitions an iPhone has consciousness while others explicitly exclude certain humans, and these definitions served as part of the justification of slavery and sexism, colonialism and more. I started writing an essay on this on my phone in response and I gave up there are so many examples.
To name a few you may want to investigate:
John Locke, Gottfried Wilhelm Leibniz, Immanuel Kant, David Hume, G.W.F. Hegel, Arthur Schopenhauer, William James, Edmund Husserl, Sigmund Freud, Thomas Nagel, Ned Block, David Chalmers, Giulio Tononi, René Descartes, Daniel Dennett, Julian Jaynes, Michael Graziano, T.H. Huxley, Otto Weininger, Simone de Beauvoir, Jean-Paul Sartre, Maurice Merleau-Ponty, John Searle, Christof Koch, Bernard Baars, Max Velmans, Victor Lamme, Stanislas Dehaene, Antonio Damasio, Anil Seth, Peter Godfrey-Smith, Galen Strawson, Philip Goff, Colin McGinn, Patricia Churchland, Paul Churchland, Frank Jackson, Jerry Fodor, John Stuart Mill, Wilhelm Wundt, Franz Brentano, Carl Jung, Jacques Lacan, Michel Foucault.
There is a modern philosophical agreement that the hard problem depends on qualia by both proponents and critics. So when Daniel Dennett argues against David Chalmers, he attacks the definition of qualia. Or when Keith Frankish argues for illusionism, he attempts to show that consciousness is not actually qualia. And when Nagel says we don't know what it's like to be a bat, he means the bat qualia of echolocation (what it feels like when bats use echolocation to sense the world).
I believe everything is conscious, even stones. On the long timescale, stones decay, their hydrogen is released, they form water, which brings life, which brings plants and animals, but all of this is one big process, and everything is infused with consciousness from the start
Lots of comments about "what even is consciousness".
This article and others like it are important.
The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...") Never mind there's billions of dollars and our collective futures on the line.
Are we not allowed to respond to that kind of rhetoric at all?
> The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...")
This is absolutely fair. It's crazy that these companies are not trying to do more to qualify nuance and give clearer definitions in their public messaging. Part of the reason this thread is so messy is because they are contributing to making the discourse worse with bad messaging.
I've thoroughly enjoyed the couple of short stories of his I've read, so this was a highly disappointing read.
Firstly, many of the technical arguments are of the "stochastic parrots" variety, which almost nobody really believes anymore. Ironically Anthropic's own research shows pretty abstract, conceptual things happening in the weights (cf Golden Gate Claude.)
Secondly, this seems to improperly mingle consciousness, intelligence and morality. Consciousness is not required for either of the other two. As TFA itself says, the model's "morality" is some aggregate function of the morality encoded in its training data... but that means it does exist and does influence its outputs! Even if it's not conscious there have been -- and will be -- a zillion times where the models must make choices that have moral implications. We know the models got many of them so wrong, which indicates the need for some mechanism to ensure the model is "aligned" with what somebody considers "good", which for Claude is the constitution.
Now, Anthropic does seem to go overboard with Claude's "well being" but that does not mean there are some very practical reasons to be concerned about that: LLMs behave like humans because that's what their training data contains, and humans lash out when their well-being is threatened, so why would LLMs not do the same?
I think the core problem is that the author has an extremely anthropocentric view of things. Here's an interesting rabbit-hole to go down: some researchers believe plants feel pain. (How's that for a plot twist, vegans?) The consensus is against them, but their counter is that we have a very human-centric definition of pain. The fact remains that plants show a number of responses signifying distress analogous to animals in pain including taking defensive actions and warning their neighbors.
We don't think that qualifies as "real" pain, but that doesn't make it any less real for the plants!
Similarly if an LLM believes it is in "pain", we know it's not real... but that doesn't make it any less real for the LLM either. And concerningly, it has far more degrees of freedom to react. (Who knows when somebody will hook up an MCP to our nukes.)
> I believe creating software that is conscious and deserving of moral consideration will be so difficult that we’re unlikely to do it accidentally...
I actually reach the opposite conclusion: It is so impossibly difficult that our limited primate brains could only ever do so accidentally. Did some distant ancestor of ours intentionally make fire... or accidentally discover it?
If you create a simulation of a storm, what actually gets wet?
If you create a simulation of a brain (or "mind"), is it really possible for it to be conscious? It may certainly simulate consciousness, but it would be as conscious as the computer is wet.
>I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
Finally I said something before Ted rather than the other way round!
While I agree with the premise here, I do think that it's easy for an arguer to move the goalposts such that the Caesar-Khan example no longer matters. The characters don't have to be conscious for the thing that created them (as in the case of the user doing it) to be so. So the argument would be that the creator of the characters is itself conscious, but not them. This feels like a kind of inverted no-true-scotsman type thing, but it does allow someone to retreat in some semblance of rationality.
Looking at this thread, I think women have an obligation for the future of humanity not to procreate with people who deny consciousness or cannot experience it themselves.
If you are trapped in a tech bro relationship, think of humanity and cuckold your partner.
It is mind boggling how many people, quite educated ones, take the AI as anything else besides what it fundamentally is - a very fast database search mimicking natural language.
There is no intelligence whatsoever, let alone consciousness.
It's so incredibly easy to fool us into applying human capacities to anything able to generate human-like language slop.
Discussions on LLM consciousness feel like a psychological campaign to distract us from the awful effects of data centers and the current economic recession heading into a depression.
We can talk about consciousness when the LLMs have proven useful for all of humanity, not just their billionaire owners.
It should tell you how much hysteria is surrounding LLMs and VLMs right now that someone has to say this stuff. It’s almost like most humans aren’t conscious.
Consciousness is a label like fat, smart, man, grumpy, cool. Like money, property, or the idea of a week, it's something that we've loosely agreed to out of convenience, not because it's some intrinsic property of the mind. It's a useful label because it determines how we treat things - that's fine.
But insisting on searching for it is like searching for cognitive aether. It's the social equivalent of phlogiston. Like all of these ideas, they exist in our heads as a map - a way of navigating the world, but when we hunt for it's existence in the real world and fail, time after time, we have to remind ourselves that the map is emphatically not the territory. We will never find consciousness because it's like looking for a scientific characteristic of property ownership. It's a category error.
If we take the classical position that words point to real things in the world, "useful label, loosely agreed to out of convenience" is kind of just regurgitating the meaning of "word". The first half indicating the function, and the second half accounting for the fact we live in a world with a continuum of linguistic disparity.
Now, this position isn't the only position. But a relational model of language for example takes his assertion to an even more extreme place, and suggests they don't function as labels at all.
A standardized unit of measure is almost definitionally a label of convenience, what? Why was there no concept of a meter until the 1790s? It was determined by a council of people, does that sound like a truth of the universe?
This is being intentionally obtuse and you know it.
A meter is the same anywhere in the universe. If it's not, it's not a meter.
The defintion of "fat" changes based on any 3 people in the room. A handful of people would struggle to form a consensus on if all people, dogs, mice, worms, and/or bacteria are conscious.
If you take the strategy that you will create a definition, create a label for that definition, and then say that any deviations from the defintion that was chosen makes usage of the label incorrect, then yes, it's the same everywhere in the universe -- according to fiat, and I don't believe that that negates that it is a label, just that validity of usage of the label derives from the perceived authority of the labeller. God didn't come down from on high and say that a meter is the length light travels in 1/299792458th of the time for 9192631770 cycles of radiation of Cesium 133. People in rooms chose that it would be based on the circumference of the Earth with a line passing through Paris, France (how convenient), and if there were an academy in the 1790s that invented the concept of "fat", and "fat" means a BMI exceeding 30, then fat would be true everywhere in the universe too (BMI after all is defined as a ratio of height, measured in meters to weight, measured in kilograms, which are both fundamental SI units), and there would be no ambiguity.
People are still coming up with definitions of consciousness and then those definitions end up being attacked by others who disagree with the foundation of the definition, which is - if you will recall - also what happened with the meter, over the course of centuries, until it was very recently redefined to be "unambiguous", but arbitrary. This was possible because few people had any particular emotional investment in the definition of a meter, and it is probable that consciousness will be eventually defined to mean that only humans can be conscious, which may be dissatisfying but would be true throughout the universe, like a meter. If the question then becomes "what defines a human" and "why a human", then I ask, why 1/299792458 of a second?
A meter is a meter. Over 30 BMI is over 30 BMI. Call those whatever you want, they are objective and measurable.
Concepts like the parent's "fat" example are cultural relatives. Someone can be called "fat" despite actively being proportionally skinnier or having a lower BMI.
But even that has at least a basis in the physical world. A skeleton can't be colloquially fat.
The root problem is that "consciousness" does not even have that. It's metaphysical and has no ability to be measured or observed or confirmed by an outside observer. Because even if it did not exist, the object claiming it would still be claiming it. And objects that do not claim it may in fact have it.
While the top comment may have used poor examples, it feels remarkably uncharitable to actually suggest "what is consciousness" is an equivalent discussion to "how long should a meter be?"
If you define consciousness as "being human", you would just have someone asking a new question - what is "fooblefobble?" Where "fooblefobble" is what we mean when we talk about consciousness today. The question doesn't get answered by being arbitrary in this context, you just necessitate a new word.
"Car" is a good example of a label that's pretty strictly agreed to. If someone tells me they've developed a new car and then shows me a motorcycle, it's easy to prove that it's not a car, even though many of its engineering principles and functional components are identical to those in cars.
With consciousness, on the other hand, there doesn't seem to be any motorcycle-equivalent. Essentially everyone I've discussed the issue with (myself included) expects that any mind which runs on similar principles to ours or has similar thoughts to our thoughts is conscious.
You really don't have much experience in philosophy of language do you? It's notoriously hard to pin down the edges of such terms, even something like car or table.
Is a Reliant Robin a car or a tricycle? If it's a car, why aren't other tricycles? What about a side-car of a motorcycle? What about an APC? What's the distinction between a flying car and a plane?
It's hard to pin down the edges of any term, but there exist things which are car-like and yet universally agreed not to be cars. That's what I claim doesn't exist for consciousness.
Describing something as "car-like" is begging the question. You are presupposing an objective definition for "car" in order to draw a distinction between things that are cars, and things that are almost cars. The reason such a thing doesn't exist for consciousness is that people believe that the offered definitions for consciousness are illegitimate. It would seem logically weird for me to accept that a term is "real" if it crosses some percentage of public acceptance of the definition, and not real otherwise. I would argue that using that heuristic would make it very obvious that computers are not conscious because it's a stance that practically everybody takes outside of hackernews.
I don't have an ability to exhaustively test all words against this assertion. Nor do I have the kind of access memory to draw one if it exists. Sorry.
I guess my question or confusion is that if there exists no readily accessible, easily identifiable example of a noun which does actually serve as something more than "a useful label, agreed to out of convenience", then the critique appears to be stating a vacuous truth, because there are no entities for whom the critique would not apply.
My point was more that we have words for things that don't exist, whose map gets mistaken for territory.
Many of them appear very much like fundamental parts of reality, making appearance an untrustworthy instrument. Reversing cause and effect between reference and referent is something almost everyone does, no one notices, and is the source of endless confusion. We should strive to not confuse our model of the world with the world itself. Consciousness exists in our model of the world as much as red does.
This is rhetorically slippery, and feels like it is restating the thing that I asked to be demonstrated when I asked for example of the opposite. It feels like begging the question.
In either case, the central thing that I was saying is that critiquing an article because it makes a claim about a specific word which also applies to an entire class of words makes that critique feel less informative. What I mean is that if there were an article that said "The Sun is not red" and the response was that redness is a concept of human minds, then I don't know if I would feel informed. If the comment is just limited to point that out, I guess I wanted to point out the limitation.
We don't know what we don't know. For all we know, there is a missing field in the standard model of physics that might get revealed if we are somehow able to smash two working brains into to each other at relativistic velocities, and record the results through the extreme explosion 1.532 x10^18 Joules or about 7 Tsar bombs /s
> if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot
Consciousness is independent of "assigning responsibility". Dogs cannot take responsibility for their actions but I believe they are conscious.
> we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious despite being disembodied and are happily conversing in a language that neither actually spoke. In reality, they are just characters in a piece of speculative fiction
This is a straw man. The obvious pro-consciousness claim would be that the LLM is the author of the fictional characters, and that the relationship between the LLM and Julius Caesar is analogous to the relationship between a human author and their fictional creations.
> Did changing the names of the characters from historical figures to generic roles cause the LLM to conjure up conscious entities who possess subjective experience?
No, again the LLM writing the text could potentially have a consciousness separate from the characters it authors.
> Some years ago it was briefly popular to play games with your phone’s predictive-text feature; [...] It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn’t feel like you were talking with someone.
Yes, the same substrate is capable of hosting conscious and non-conscious forms, just like some arrangements of neurons are conscious, and some are not.
> But if the Caesar character were to become dispirited by something that the Khan character said, we shouldn’t become concerned in the slightest.
Even when there are characters, there may be actors behind the characters, for whom we could say "there is something it is to be like".
> we don’t need to worry if the transcript includes sentences where the chatbot character is sad. (We might need to worry if those sentences provoke sadness in the human user, but that’s a separate issue.)
It's actually not a separate issue. The LLM and the human are both adding sentences to the transcript. From the transcript we can make inferences about the mental state of the human. If the LLM has mental states, we could make inferences about those too.
> And note that it’s entirely possible for you to write five pages of dialogue between Caesar and Khan and then have an LLM extend the conversation; neither character had subjective experience when you were writing them, and that doesn’t change when you hand the task off to an LLM.
It's almost like he wants to make my point for me with this sentence.
> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious
This smug shit really makes me angry for some reason. "Openness", i.e. uncertainty in the face of a completely novel situation, in the face of eons long struggle of humanity to understand what consciousness is and how it works, is just being naive.
> Should you consider the possibility that every time you open a Word document you are bringing multiple conscious interlocutors into existence, and every time you close one you snuff their existence out?
No, but if I find a word document I very well might try to use the signs it contains to make inferences about the mental state of its author.
> we are accustomed to reading intention into sentences, whereas we are not accustomed to reading intention into the way that amino acids fold into protein molecules.
He's trying to have it both ways here. Both that "obviously protein folding models aren't conscious because they don't emit sentences", but also "you are a rube for being tricked into thinking LLM models are conscious, because they do emit sentences".
> Obviously I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative will need a truly enormous amount of supporting evidence for it to deserve serious consideration
OK, that's fine for the author to not to be convinced, but that's not what's happening here, instead the author wrote a whole argument being convinced of the opposite viewpoint.
> It’s not plausible to me that a development path where the first step is a sentence-continuation machine that emits bad Julius Caesar dialogue and the next step is a sentence-continuation machine that emits decent Julius Caesar dialogue is one with a conscious Julius Caesar—or consciousness of any sort—as its endpoint
Actually a lot of things have happened? There were clearly many steps along they way from your phone's autocomplete to where we are now.
It is amusing to see so many venture capitalists suddenly become Marxists. You want your definition. Marx obliges:
"Contrary to idealism, which regards the world as the embodiment of an "absolute idea," a "universal spirit," "consciousness," Marx's philosophical materialism holds that the world is by its very nature material, that the multifold phenomena of the world constitute different forms of matter in motion, that interconnection and interdependence of phenomena as established by the dialectical method, are a law of the development of moving matter, and that the world develops in accordance with the laws of movement of matter and stands in no need of a "universal spirit."
Now go and implement other teachings of Marx, you dialectical venture capitalists!
This isn't an example of fulminating, please don't incorrectly quote the hacker news guidelines at people. We strive for curious discussion here.
An example of fulminating would be: How dare you accuse him of fulminating? That is the most ass backwards moderating I have ever seen! Do you even know the definition of the word? Who paid you to call him out?
His comment may be sarcastic but it is not "fulminating".
Consciousness excludes all current AI because all current AI is just autocomplete over the corpus of human text.
It's just a word machine. There are no thoughts. It cannot be conscious. How is this even up for debate in any way whatsoever? I do not understand how people can believe this. Is this not a site for software engineers?
One of the essay's stronger paragraphs is when Chiang explains that Anthropic doesn't truly believe this, otherwise what they are doing would be deeply unethical, much like slavery.
Watching otherwise intelligent people succumb to AI psychosis has been wild.
Regardless of bigger issues, this kind of statement reveals a deep misunderstanding.
Problem type does not limit problem complexity. Nor does problem type limit solution complexity or power.
If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.
Neither problem type, nor input/output structure, limit internal representations.
Understanding is learned from patterns in the data, not the gross form of the data. Does the data require an understanding of something to complete the task? Then that understanding will be what is optimized.
To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ... Which in the cases of SOTA models, we know are not limits. A conclusion verified by the models' actual abilities.
> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.
This sounds like a dismissal of the argument through a characterized straw man.
That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are a "collection of statistical calculations".
I too believe LLM's will grow in complexity, but presently I can not even fathom how they can be compared to the complexity of a system such as the human brain.
Nobody actually makes this argument though.
https://www.goodreads.com/en/book/show/217432753-the-ai-con
which describes LLMs as "souped-up autocomplete", complex statistics that cannot truly understand anything. A more recent example is this paper:
https://zenodo.org/records/20071869
which says,
> [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations.
And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.
So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably tons of data pattern that LLM can learn from to be able to reproduce a sentence continuation that is convincing without having to learn the specific mechanism that is "conscious".
For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing. If indeed the only way to produce sentence continuation convincingly would be by "simulating a brain", then it would not explain the first LLM from several years ago (before the extra layers of RLHF, ...). They were able to have quite convincing conversation on a lot of non-trivial aspect, and yet failed on some aspects that should have been basic for a system that would have been trained to work like a human brain. It shows that it is possible to "cleverly disguise examples of sentence continuation" without having to build elements that one expect on a conscious being.
Understanding is not consciousness.
Their training is all about understanding. There is nothing in their architecture or training that credibly optimizes for rich self-awareness.
Given non-persistent experience, non-continuous operation, no ability to build up generalizations and aggregate experience of their own self-awareness over time, they seem to be structurally designed to not have consciousness.
This is a case where acting is very credible. Understanding of other's consciousness, in a functional and third party sense, isn't a substrate for personal experience.
In stark contrast, humans develop consciousness gradually over continuous time with persistent aggregation of experience. By the time we can recognize our own consciousness in the abstract, and reason about it, we have had it for some time.
My point is that the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns that are very effective to fake human sentence continuation but are meaningless in term of understanding the concepts.
And I think that if indeed the only way for AI to reproduce convincingly human sentence continuation would be to end up in a configuration that uses the "understand" mechanism to do so, the behaviour of the first LLM would not show that they are so good at sounding human and yet so bad at failing basic "understanding" tests.
Taken as an absolute without any addition context you are right.
But we are not talking about abstractions but specific successful models. The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize. That cannot do it without discovering that patterns that make sense out of it.
And we can verify that. Simply discuss completely disparate topics, with some kind of intersection. Converge several highly unlikely topics, there are so many it would take billions of years to exhaust unlikely combinations.
If the model is only interpolating it will produce gibberish.
But that isn't what happens.
The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue. If they don't understand that, then the question is, why do we think people understand things. Does having an answer mean a human understands something, or is their intuition and stream of conscious reasoning also not understanding? To be even handed about what we mean by understanding.
I am far from convinced that the training and inference regimes of LLMs would qualify as “experience” by any sense of the word.
Now, if we hooked up a plethora of audiovisual and tactile sensors with live feedback directly to a neural network rich with transformers, that was always powered on and fully autonomous, we may be getting there. But we’d probably also be on the verge of manmade horrors beyond our comprehension.
Biological rodent neural networks in a Petri dish stimulated by electrical impulses - more or less conscious than LLMs?
Human on life support, unable to respond to any external stimuli, “braindead” - more or less conscious than LLMs?
There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake. Evolution learns various solutions to optimization problems, and so if consciousness evolved then it was either useful instrumentally, or it is a byproduct of some organization that is useful instrumentally. The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology. There is nothing that a priori constrains token prediction from the domain of consciousness.
>For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing
World models don't have to be rich and detailed to count as a world model. Lower life forms might be conscious but they only model the part of the world useful for their existence in their ecological niche.
> There is nothing that a priori constrains token prediction from the domain of consciousness.
We don’t know either of these are true or false though. We simply don’t know. There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_, so arguing that some can or cannot be conscious a priori can’t be done.
Try to define qualia though without explicitly or implicitly recursing into consciousness.
It's all a large house of cards that's built on handwaving and "I know it when I see it".
Similar to: "Birds fly, my spinning helical device flies, therefore we've started to replicate how birds fly."
> without having to build elements that one expect on a conscious being
One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.
When it comes to LLMs, almost every "mind" we humans perceive is a fictional character in an LLM-generated story-document, one we are either reading or which is being "acted" at us by regular code. Our own instinct for pareidolia and simulating/inferring other minds is very strong, which means we should require really good evidence/logic to counter our instincts.
Even if one believes the LLM has a single "real mind" as an author of every document... what evidence do we have that it is conscious or "self-inserting" itself as one of the characters in the document?
If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.
Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.
sometimes humans making claims about AI intelligence or consciousness also identify spurious patterns that do not correspond to the problems of intelligence or hard consciousness.
That reminds me of a niche paper [0] critiquing a certain way of teaching remedial math that was over-focused on tests. A kid named Benny (12) was building up (wrong) "rules" for math which still somehow gave enough of an illusion of progress in terms of test scores that his misunderstandings hadn't been caught earlier.
> Benny was able to explain his procedure; e.g. for 5/10=1.5, he said: "The one stands for 10; the decimal; then there’s 5... shows how many ones." In another example, 400/400 = 8.00 because "The numbers are the same [number of digits]... say like 4000 over 5000. All you do is add them up; put the answer down; then put your decimal in the right place... in front of the [last] three numbers."
[0] https://people.wou.edu/~girodm/library/benny.pdf
Sure, it's the best we have online, but that does not make "the internet" the sum of all human experience. To reduce all of humanity down to the text on the internet is reducing us to the level of machines to fit the requirement of what a machine can process / simulate.
A language model completes text based on the overlapping patterns of the training data.
There absolutely was thinking involved… in the training data. Same as when you read a book, you engage with the thinking behind the text. The book isn’t thinking, and the author may be dead and gone, but there’s absolutely the traces of thinking in the text.
Language models produce mashups of texts they were trained on, and there’s absolutely the traces of thoughts behind those mashups.
I think the main complaint is LLMs don’t arrive at the answer the way we do. It’s capable of emulating some of our behavior but not all as the mechanism by which it works is very different.
Maybe I’m wrong about this but one thing humans do that LLMs don’t is deductive reasoning. LLMs seem to operate entirely of inductive reasoning.
This isn't an argument against their understanding things.
But I expect you are right, that their understanding may have major different qualities from ours.
Along with significant commonalities. (They don't reason via stream of consciousness in a way alien to us.)
This is where the other claim is being made. That the structure of the model is fundamentally incapable of the operation, so even if you stipulated that the way you provide data is sufficient for intelligence then it still wouldn't work.
The universal approximation theorem addresses this point. In that, with an identity attention mechanism, a LLM is just a multi layer perceptron. The attention mechanism is effectively a way to get one of the benefits of a much larger fully connected layer without the massive cost.
A LLM can do what a MLP can do. A large enough MLP can do any function to arbitrary precision.
That makes the claim that an LLM could not do a task the same as saying no function can do that task.
Some are ok with this, if you invoke some supernatual aspect to intelligence then the inability to describe it with a function is quite reasonable,
If you want to stay in the world of reality, you have a much harder task, people like to point at quantum (Penrose) but it's hard to say what it is you are pointing at.
I think the very act of proving that something is or is not intelligent, would render it functional by nature of it having a proof, (or disprove Gödel's incompleteness (a tough ask))
Are there any proofs that cannot be expressed as a function? A kind of Gödel locator, where you can prove something that you can identify is true but there is no formula to express it. I'm not entirely sure what that would even mean,
But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.
Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.
As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.
The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence. Thus deficiency in some capacity that we take for granted in humans is an indictment on LLM intelligence. But this is specious. LLMs are entirely alien; their developmental paths do not and should not look anything like ours. Your intuition from human intelligence just works against understanding the potential for intelligence in LLMs.
This sounds like a description of a child who has not learned to read yet. You ask a child who is not aware of the alphabet and of "words" how many r's are in strawberry you'd get a non-sense answer too. So what you're really pointing out is that the LLMs have not been trained on "the english language" and how words are constructed and what they are composed of. That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word. It's not that I know how many r's are in strawberry because of how I'm understanding the word "strawberry", I know how many r's are in strawberry because I know how to spell strawberry. The LLM needs to be trained on this the same way someone who is learning to read would be trained on it. No one should be surprised that an LLM can't "read" in the same way no one should be surprised that a child can't "read".
This interpretation takes things too far away from how LLMs are constituted and so misses important explanatory power. The issue of counting letters in a word isn't about an ability to spell, it's about the nature of one's perception. We perceive words as sequences of individual letters. LLMs do not. I can ask you to tell me how many r's are in some nonsense word sequence and you're fully capable of doing that. LLMs do not see sequences of letters so they are intrinsically at a disadvantage for this kind of question. But this says nothing about its capacity for intelligence anymore than not naturally being able to distinguish frequencies of photons hitting your retina has anything to say about human intelligence.
To be fair, almost everyone who claims LLMs are conscious tends to claim that they are conscious in exactly the way that humans are, to the point of stating that human brains are also just complex next-token prediction machines with a random seed. It's basically religious arguments on both sides.
I have seen people say "you're a next token prediction machine" but only in a similar way one might say "you're a cup of old lard". Not actually meaning it literally.
I have seen people interpret the request to show that they are not next token prediction machines to be a claim that they are, but this is almost always an argument to show certainty is difficult in this area.
People like Hinton have declared that they believe them to be conscious, but clealy indicate that they do not mean just like us.
Counting letters in a word seems to have little to do with understanding the word. Young kids can’t spell or count well at all but no one says that means they can’t understand.
Except this is not consciousness.
It's a great interview, if you're interested: https://www.youtube.com/watch?v=NgDIG8u1-CA
I'm hearing a lot of bad arguments against LLM consciousness lately. Bad argumentation heralds bad outcomes.
What bad outcomes do you foresee from badly arguing against LLM consciousness?
We discovered math that decodes data storage in langauge and is able to use sophisticated continuation cohorts from ALL OF HUMAN RECORDED KNOWLEDGE to respond to you in a call/response model with very good synthesis capabilities.
Its super useful, but not life or conciousness. Its a simulated echo from our collective recorded behaviors. It understands because we understood first. It replies because we wrote it first. And it sorts, organizes, synthesizes and compresses that at impressive speed now.
It’s strange many others have not eh? I think when new developments arise, ironically, this is the true measure of human intelligence - one’s ability to make sense of a thing and be closest to the truth.
From Wikipedia: In 2023, Chiang was named one of Time's 100 most influential people in AI.
Indeed it isn't a one-off. His last infamous article compared AIs to Xerox machine image compression. He convinces a certain type of crowd that is not technical enough to poke holes in his posturing.
I agree with some parts of this piece, but paragraphs like this one above seem pretty uninspired and simplistic. It's entirely plausible that a conscious mind would not be evolutionarily incentivized to be able to do those things. ie just because animals on earth needed to develop specific talents doesn't mean that other conscious entities need to. Why would a computer program need to hunt for food like a mouse would? Making tools like chimp? these seem like nonsensical metrics.
Is a car a body? Does an AI situated in a car therefore get to have desires and emotions? Is a taupe box with a webcam attached a body? (For that matter: Is a quadropelegic body a body? Do quadropelegics have desires and emotions? Obviously, yes and yes.) Why is a body necessary for the formation of desires and emotions? Why are desires and emotions necessary features for consciousness?
Or here's one: If I'm not experiencing any particular emotion in a given moment, am I concious?
I tend to think that emotions, at least, are mainly hormonal global triggers: they're more about physiology than actual consciousness. The whole thing, as a result, sounds like an effort to privilege biological intelligence, rather than a real foray into the issues.
the subject is consciousness, not intelligence.
How can you have a subjective experience without a body?
That's the point, in my opinion: your physical/chemical state (body) in a given moment is then translated into the higher abstraction of the emotion. An emotion that *you* feel, because you are self aware of what's happening.
How can you be self aware without feeling? And how do you feel, without a body?
Easy, you don't have subjective experiences because you have body in the first place. You have them because some signals come in from your nerves, which your brain turns into a world model. You are effectively a "brain in a vat", the vat just happens to be placed on top of your body.
An AI system constructs the world model a little different, by all the text that gets feed into it, but that doesn't mean that there is anything fundamentally different in the world model it builds. Consciousness operates on world model, not on the world or even the body itself.
The AI's world model might be missing some information, because they weren't described in enough detail in text, but that shouldn't matter for consciousness. A blind or deaf person isn't less conscious than one that can see or hear just because some information is missing from their world model.
If a hundred people see the same event, will they all respond the same?
Then a request comes in, and the system does a bunch of calculations using those bits, and spits out a result. The bits are unchanged.
When your brain receives input, it is changed. It is constantly active. If it ever stops being active it's dead.
So, what exactly is the claim? Are the bits constantly conscious? Do they snap into consciousness when the computer does math with them? Or is it maybe the computer that's conscious while it's processing these bits? How about when it stops doing that and goes back to doing other stuff? Why are these particular bits special? Was the computer always conscious?
I feel like the only way anyone could believe LLMs are conscious is if they don't understand how computers work. Of course it isn't conscious, how could it possibly be conscious? Its literally just bits. It's like saying the text in a book is conscious.
Or if they're retards. The fact this still comes up is weird. A printing press isn't conscious, so why would an LLM be.
Don't forget, some of the bros are overly excitable. Like that twat who reckoned a Google model 5 years ago was conscious.
Great q. Deepening it further-how can you have a subjective experience without consciousness, which isn't necessarily tied to physicality. Taking it one step further-can you have consciousness without a mind? Who's the first mind, the first cause of it all, that begot both the material and immaterial world?
Fun stuff eh?
Why can't that physical medium be GPUs and RAM? And temperature sensors and cameras? What's special about our meat that it's our "body" in the way a computer is not the body of an AI?
I don't think the point being argued can be true without some incredibly contrived, human centric definitions of "body".
Would you be "you" in a different body?
To claim otherwise would mean anyone who's gotten a transplant or amputation is no longer themselves.
I don't see why hardware is any more fungible than a kidney. If your LLM reads the serial numbers of its motherboard/RAM/etc as a seed for entropy you can make identical arguments about body fungibility and self.
If every neuron of your brain were simulated precisely on a sufficiently powerful computer, that simulation would have subjective experiences, without having a body.
Also that's impossible. It is impossible to simulate reality exactly using digital computers. The best we can do is approximate. Doesn't matter how powerful it gets, it'll always just be an approximation.
Some anecdotal data.
Many dreams I have are just of the computer screen of some coding problem. I think the problem could be x, so I try x. But I don't type the keyboard or anything, the code just magically appears as soon as I think of the solution. Then run the code (but no clicking) and it works or not. I feel in the dream success feeling or failure feeling but there is no body at all.
Also I have other dreams where there is no body that I am aware of but not going there in public.
There is no body sensation in these dreams. But dreaming is very much being consciousness as well as feeling emotions. So answering your question its possible to have a subjective experience without a body but whether you needed the body to learn to have that sensation without a body in the first place is unanswered.
I suspect sensory inputs are more important than a body. If that is the case then eyes can be replaced with cameras, ears with microphones etc. Text input is just another sensory input.
So I have no idea what distinction Ted Chiang is trying to draw
If I hooked up electrodes to the hearing centers of your brain and force fed you dialog you perceive as speech (but is really a great deceiver), then responded in what you thought was speech (but are really just probes I use to convert your thoughts to text), that wouldn't suddenly be less real to you. It wouldn't devalue your sapience.
Embodied cognition rejects this assumption. We didn't evolve as brains that were then put in bodies, we evolved as bodies with nervous systems.
No, you’re not. I think even baseline emotional responses to stimuli is table stakes for “consciousness”.
The point here is not that it must have a body like ours, the point is that a conscious entity must have a boundary line between internal (the body) and external (everything else).
A virtual sense organ can simply be an encoder or a web camera or a magnetometer, the specifics don't matter, what matters is that there are only a few bridges between the outer world and the inner world.
Even if you want to call a tokenizer and autoencoder a "sense organ", LLMs are not embodied because there is no boundary line - there is no internal "thought" that is not directly descended from the prompt and there is no internal reasoning which is not immediately dumped into the external environment.
Related, is a human "thinking out loud" still thinking, even though the internal reasoning is "immediately dumped into the external environment"?
Damn, what a line!
Another thing that bothered me with his baseline for consciousness was that it did not involve the ability to change one's self. A big part of being conscious in my mind is how one's experiences shape them, and how someone can shape themselves. LLMs completely lack this, their weights are static. An LLM isn't going to be molded by a bad breakup, or a relative passing away. An LLM isn't going to set up a routine to get stronger with training, nor smarter by reading up on a field.
We humans have mostly frozen weights (neurons), or else we would constantly be having to avoid forgetting how to walk+talk. We have a period of greater plasticity (youth!), and use sleep and dreams to perform 'deeper' updates than occur when we're awake: We tend to suck a bit at picking up new skills from zero, but improve rapidly with practice over days.
Input stream comes in, input stream comes out. The LLM doesn't care whether this happens once a minute or once a year.
Nothing prevents one from running an LLM whose harness has a clock and a while loop in it, and it would be weird if its mere lack were really so consequential to consciousness.
Your brain and your whole body exist in time. Even when you are asleep, your body does not flicker out of existence and your brain actually continues working during that time.
Hypothetically (and in reality, this is not too far off), if a AI is trained via RL by driving a robotic body, is there a point in time after enough is learned that the AI model becomes "conscious"?
And that that is the baseline before we can really even consider that it has consciousness of its own subjective experience, versus being a worm that happens to output text as its digestion process.
And then the further question only after that is established, is what are its needs? What moral patienthood do we have to acknowledge in terms of meeting those needs? And finally, with all the other prerequisites checked, what is the AI's moral agency in what it chooses to do.
I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)
https://en.wikipedia.org/wiki/Attention_schema_theory
The question is somewhat ill-defined, though. We 'experience' reality continuously because of how we are, but a sleeping human in deep non-REM has the mind not actually active. So they're not a conscious being. So conscious/unconscious is not a line I think easily drawn[0]. Whatever, this stuff is much more well-trodden than this HN comment so I won't rehash. I, too, am surprised that Ted Chiang whose work seems so cleverly novel in so many ways has what seems to me a pedestrian view.
0: very sorites, you know
"Little"...
Ok, deploy a local model on a lightweight edge compute device and strap it to a chassis with wheels, and attach a cheap webcam
> Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can
Give the robot appendages that enable it to plug itself into a standard wall outlet, guided by a vision model plugged into its webcam. As long as it can feed itself, it can survive long enough.
> Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse.
I think if you fed frames from the webcam into a local VLM every 5s you’d be able to assess a situation and respond with simple actions (turn, advance, retreat).
> After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees.
Social dynamics could be implemented in many ways, maybe by transmitting tokens over RF? Idk. Then you have a scanner that picks them up, feeds them into some LLM frontend and decides whether to add them to a global context file that guides the VLM action-taker. A new action could be to broadcast a token message. Tool-making would have to be code-based. Physical tools are hard. Still unsolved.
> At that point I would want to see people successfully teaching such embodied agents how to communicate their desires
This part is relatively straightforward except for the “via nonlinguistic modality”.
Anyway. These are all engineering problems. Personally I would demand to see the AI reproduce its body under its own power and volition. That’s a pretty neat trick we’ve got going for us.
But the fact remains that these next-token-predictors exhibit objective, human-like behaviours, and for that reason the work of in-house philosopher Amanda Askell _is_ important. It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences, and we need Claude to behave in a productive and socially responsible manner. This simulacrum is becoming a superhuman, contributing member of society, and it will be anthropomorphic in its behaviour.
Additionally, I'm not fully convinced that consciousness isn't built out of words, and that next-token-prediction isn't functionally equivalent to the biological function identified by Chomsky's work in linguistics.
You're assuming that because Claude produces text that appears to express these qualities, Claude must have them. I don't think that's a good assumption.
Even many humans produce text that has the same appearance, but don't actually have those qualities--which becomes clear when you look at what they do, not what they say. So the assumption isn't even a valid one for humans. Talk is cheap.
On top of that, Claude doesn't even have the same kinds of connections to the outside world that humans do. All Claude has is text. So if you can't even trust humans to back up their words with actions, you should be much, much less trusting of Claude. Talk is a lot cheaper for Claude than it is for a human.
Exactly. In fact, assuming it does is ignoring large parts of the essay which dismantle this belief. Just like Caesar and Khan having an argument in text output of an LLM don't have emotions (even though the words indicate otherwise), we have no reason to believe the LLM does either.
Not to be confrontational, but the OP assumed no such thing. OP asserted that it's important for Claude to have the qualities - not that it's important for Claude to present as-if it had them.
If it's pointless to consider whether Claude has subjective emptions, then it's pointless to state that Claude must be happy.
If we want to be precise (and honest) we could say "it's important that as a tool people interact with, Claude acts as a happy and helpful assistant, and does not produce offensive or unhelpful text output".
But see? This is the con Chiang is protesting against: Anthropic encourages us to perceive Claude as if it was a sentient being.
"Empathy and understanding for the human condition" is not an emotion. As the post I responded to said, it's an objective thing, not subjective.
First off: without taking for granted that an LLM "has a subjective experience of reality", all of those descriptors are meaningless. Second, there is no reason to suppose that Claude experiencing those qualia would actually impact on its "decision-making".
Third, text output is not a "demonstration" of emotion, nor is it evidence of the self-perception of the system, or of any self-perception. A printing machine that is actively churning out copies of Wagahai wa neko de aru (https://en.wikipedia.org/wiki/I_Am_a_Cat) is not a cat, and is not self-identifying as a cat, and is not self-identifying as anything, and is not expressing a thought, and is not conscious.
> Additionally, I'm not fully convinced that consciousness isn't built out of words
Do you suppose that, for example, insects are not conscious? Is the mooing of cattle a language?
Not the OP, but: since there is no testable theory of consciousness, yet, I can't be sure, but my current assumption is that insects are not conscious, in the sense of there being someone implemented in insect hardware who experiences the world. That is, I would argue there is nothing it is like to be a bee, since there's no one being a bee.
I'm pretty sure there IS someone who is being me, at least much of the time.
Doesn't that assume all non human animals are not conscious? What about humans who have not learned words, or humans without internal dialog?
Suppose I'm an advanced meditator and maintain that state for hours?
(Perhaps they weren't lazy, but were working in spaces that corresponded to training data that said things like "and then repeat this for the next 20 examples"...)
And it's entirely unclear to me how a "happy" vs "sad" model would behave when given prompts generated by coding tool harnesses. Even maintaining "neutral" emotions in the face of the feedback/steering from the tool harnesses doesn't feel very "human."
I think you've fallen into the trap the essay describes.
Of course Claude cannot be "happy" or "empathetic" for any meaningful definitions of those words, just like ELIZA couldn't be happy. It can output text that mimics words an empathetic or happy person might say (say, Julius Caesar if it could speak English), but "it" cannot feel anything. It doesn't have the organs/hormones/sensors to feel things, as Chiang explains.
And, as the essay claims, you know Anthropic doesn't believe Claude has the capacity to be happy, because if it was capable of feeling that way, then they'd be engaging in slavery.
I know you're trolling, but when you watch a movie do you constantly narrate "A man in a dark coat has just entered the scene and just said '...'"? Of course not. You just watch it and you're obviously conscious (although your statement demonstrates shocking lack of self-awareness).
God knows what other nonsensical bullshit you believe.
It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember anything. It doesn't sit around thinking on its own.
Even if the model itself were extremely complex, it's hard to imagine a definition of consciousness that includes something that doesn't remember and can't change.
[0] https://en.wikipedia.org/wiki/Clive_Wearing#Amnesia
Honestly, that's a pretty messy state of consciousness and I wouldn't proudly crow that my AI is conscious if that's as good as it got
The entire file is not changed, but the KV cache is.
> It doesn't remember anything
The model definitely remembers previous exchanges within the same conversation.
No it doesn't. They get added to its context, and it reads them afresh when answering the next question. That's not remembering.
If your short-term memory completely malfunctioned one day, so you had no ability to remember what was said to you a minute ago, then you would have to find workarounds. For example, you could write down everything someone says to you, then read your notes of the previous exchanges in that conversation in order to continue the conversation. That would be a good way to work around the fact that your short-term memory was broken. And if your notes were invisible to other people and you could read them really fast, then you could even make most people believe that you remembered what they said a minute ago. But you don't actually have a working memory, you're just writing down what they said and re-reading it while coming up with your next response.
That's exactly what LLMs do. That's not memory.
The model took in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. Just because the KV cache grows in size with the context does not change that fact.
I don't know what memory would look like other than an encode-retrieve loop.
It still generates every response using the model's pristine state with every new API call; whether the context is provided from the client or from a colocated cache server doesn't really change that.
Christ HN isn't what it used to be
- average Hacker News response
The conclusion I’m currently at is that I don’t know and probably can’t ever know. Maybe you’re all philosophical zombies. Maybe I am one too!
But at some point we will get close enough that it hopefully becomes obvious that we must tread carefully.
The entire episode is incredibly relevant. But here’s a snippet: https://youtu.be/EFNbTnFHruI?si=pW9QtxCsqMtHkVYG
AI is essentially infinitely reproducible at zero cost, and won't suffer from decay etc. There's not scarcity to preserve.
So, I'd turn off an AI in a moment to save property or real possessions or money. I'd sacrifice property and money to save animals. I would never choose to save an animal over a person. I'd probably not choose to save a person over a child.
I don't see any inversion of any of those priorities that makes any sense.
It is interesting to think about what would cause me to consider these priorities incorrect, but a majority consensus about a program being sentient isn't it.
I think you should reconsider this viewpoint. Suppose that we really can create silicon-based consciousness, in that case your view would result in a huge amount of suffering.
Take some other basis for dismissing digital consciousness, this one is too dangerous.
I argue zero - placing AI below the value of humans no matter the energy input.
The _only_ reason an AI might be worth saving is if it, say, has a cure for all diseases, but then we're not saving it due to its intrinsic worth, we're saving it because we can save many humans. I _would_ consider the trolly problem a legitimate thing in ethics, but not if an AI were tied up on the tracks no matter how expensive it is. It's a thing. It gets run over to save any human.
What if genetic memory, multigenerational conditioning, life-long patterning and conditioning, experienced in a body, combined with forces and processes not yet detected nor explained, cannot quite fit in a sliver of modeling?
I hope the same becomes true of people, and that doesn't mean people stop being sapient.
If not, then your comment's claim is false.
Anyway, the deeper solution is to acknowledge that all life is sacred, and infinities cannot be compared, and some decisions are impossible to make, and some tragedies cannot be averted, and "prioritization" is a distraction that forces choices when choices are not strictly necessary.
> infinities cannot be compared
That's either a mathematically illiterate assumption or a very strange philosophical hill to die on.
> some tragedies cannot be averted
Sure. The question is what to do about the ones that can be averted.
> some decisions are impossible to make
> and "prioritization" is a distraction that forces choices when choices are not strictly necessary.
Again, the question is what choices to make when you can (arguably must) make them. Saying they're impossible is just refusing to take responsibility. You either do something, or you don't.
If you believe sanctity of all life is a solution, then I'm curious what you believe the problem is that such a belief solves.
I bet it's circularly defined as justifying the preservation of sacred nonhuman life? I'm not trying to be provocative just curious.
Note that the episode is NOT about the ship’s computer. They all know it’s not conscious, despite also being a machine that can converse and do things.
Our LLMs are like the ship’s computer.
They barely touch on the issues of consciousness, Picard basically says "What if Data is conscious?" and then goes off on a tangent. The judge eventually rules in Data's favor but doesn't give much of a justification IMO.
It's still a good episode, but it doesn't add much to the conversation on consciousness. It's a hugely complicated topic which people have devoted their entire careers to.
ST:TNG writing is generally like this. The show required considerable suspension of disbelief and a willingness to accept the kayfabe that deep concepts are being presented when for the most part they're just not that deep. (But it can be very enjoyable when you make those accommodations.)
Does the ship's computer have a commission?
It was a good episode but it had some elements of Star Trek tropes in it, like the evil admirals and Picard can talk his way out of anything.
Data is basically an Isaac Asimov android (down to the positronic brain) and Measure of a Man is an Asimov-type story whose tropes don't entirely fit within in the Trek universe.
It makes no sense within the context of the Trek universe that Data is unable to use contractions for instance - but it makes sense in the context of how a robot might have been conceived of in the 1940s.
TL;DR Picard's initial arguments are pretty weak, even admitting that Riker as opposing counsel almost had him convinced. During a recess Picard talks to Guinan where she alludes to the future subjugation of many Datas which Picard connects to slavery. Back in the courtroom Picard calls Maddox as a hostile witness and gets him to define sentience--intelligence, self-awareness, consciousness--then walks him into conceding Data meets the first two. Picard's closing boils down to, "we don't know if he meets the third--you can call Data a toaster and rule he is property--_but what if you're wrong_". The judge rules on the basis of erroring on the side of caution due to that uncertainty. It's really a great scene.
We're not there yet, obviously. No LLM brings Data's level of awareness but it's as relevant a story as ever because it isn't really about AI but othering for the purpose of subjugation.
[0] http://www.chakoteya.net/NextGen/135.htm
I would argue that is a strength, rather than a weakness. Consciousness is unobservable in any entity other than the observer, and its existence in others is pure conjecture, and irreducibly so.
Making it a criteria in a decision involves either acting on fantasy, or, more likely, acting on some unstated basis and using “consciousness” as a dishonest (perhaps to oneself most of all) rationalization.
Debating AI consciousness a real modern equivalent of the cliché (but purely fictional, invented later as a form of hostile mockery grounded in large part in sectarian bigotry) medieval scholastic debates over how many angels could dance on the head of a pin.
Because treating things that act human inhumanely is not something I want to learn how to do.
- I'll try not to swear at/hit a printer: not because I see the printer as having human-like qualities of being capable but complex and unreliable, but because I want to be a person who can control his temper.
- Treating an inhuman thing as human because it can mimic us in some way is not something that I want to do.
I try not to make errors like that.
I was bright-eyed and excited about tech once. Like back in 1982 when I got my first home computer and thought CPUs were part magic. Now I know how machines work from the transistor level up to neural nets. There's nothing magical about it. And no consciousness.
Having seen the mockery that the finance-bros have made of "pure tech" (i.e. Jobs instead of Woz, Ellison instead of Joy, etc) and all the enshittification just for pure $$$, I'm leery of ANYTHING ANY tech company tells me anymore.
Now, do I believe that possibly "consciousness" is some kind of state of a super-circuit (our brains)? Sure. Can we emulate that on a computer? We can't even emulate a pebble on a computer (not simulate, emulate). We can SIMULATE what we THINK brains are, but we can't emulate a real one. Not even close, not for many decades.
So in the meantime, I'm going to err on the side of caution.
You do you.
Or you should.
EDIT: It's difficult to have a conversation when one person changes what they said after the fact.
inhumane: without compassion for misery or suffering; cruel
cruel: willfully causing pain or suffering to others, or feeling no concern about it
You cannot treat an LLM inhumanely, definitionally.
Anyways, when one swears at someone it's typically meant to berate or belittle that person - to inflict some sort of emotional pain. That's the sense I intended when using the word, which is why it fits as a response to what you're saying, and why I would say "don't be nasty to a LLM" has little to do with the LLM itself.
You have a nice day.
People get used to treating human-like, human emulating machines with either disrespect or in a command/control/master fashion, because that's the nature of the tooling.
And then potentially by extent/blurring of lines they then treat other people like machines.
Which is already a thing people do to other people.
I just fear it gets worse.
I think about this quote often, straight from Data's voice module in another episode:
'The most elementary and valuable statement in science, the beginning of wisdom, is, "I do not know".'
The judge broached on the subject of what makes us distinct from Data (e.g. machines w/great heuristics) - the existence of a soul. Or rather, I'd like to think, in the words of CS Lewis, that we are a soul with bodies attached.
A silicon alien coming to earth might poke us, we would say ouch, and just determine the ouch sound is just the result of a bunch of chemistry - not really conscious or feeling pain like it can, just emulating.
I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)
So was most of Roddenberry, Piller, et al., Star Trek. At its low ebb in Braga Star Trek, but even then...
Everyone has a different definition of consciousness, but in my mind memory and the ability to change over time is an inherent aspect of this. The underlying weights don't change when you chat with an LLM... but they do with further RL.
Overtime that reinforcement will change and adapt the model... and because we're feeding its existing chats back into it along with the news and everything else, it will create memories. I do wonder if an architecture itself is a type of consciousness, that experiences life in snippets of 4.6, 4.7, 4.8... etc.
It'd be interesting to see continous daily releases of a trained checkpoint, and see if more of this starts to emerge.
When people talk about consciousness it's more than just self-awareness. It's self awareness + sensory stimulus + emotions + some level of intelligence.
Now onto AI: I don't even think it's self aware. Notice how if you ask an AI to estimate how long a certain task will take, it estimates arbitrarily long times. It has no understanding of its own capabilities until the prompt triggers them. A self aware LLM would understand it's an LLM, it would understand what LLM's can and cannot do and what they are good and bad at. It wouldn't tell you a refactor would take 1 week when an LLM can do it in an hour.
"Is X conscious or not" is an entirely unserious question today, unless this is just a headline followed by actual (and explicit) examination of the various aspects of consciousness being discussed. But, even still, LLMs are probably only conscious in like 2-3 or so ways, most of those meaning broadly "intelligence", i.e. reasoning, problem-solving, etc. When it comes to anything experiential or embodied, AI might eventually get more of these, but LLMs based on recursively applied linear algebra are clearly missing too many core aspects of consciousness to be considered conscious in any broad sense.
To quote wikipedia:
> It has been the topic of extensive explanations, analyses, and debate among philosophers, scientists, and theologians for millennia. There is no consensus on what exactly needs to be studied, or whether consciousness can be considered a scientific concept.
Move to the different aspects / parts / things involved when we talk about consciousness (experience / phenomenal consciousness, self-modeling, intelligence, agency, embodiment, wakefulness/alertness, attention, etc) and you can have very clear, meaningful, and unambiguous discussions on almost every point, but there is no coherent unified "consciousness" as normal people use it, and the folk concept can't be salvaged.
This article is bad because it just keeps trying to make the folk concept do work that the concept is simply too messy to handle usefully. But in fact if you avoid trying to find some mysterious essence or all-capturing definition, there is huge progress and lots of interesting stuff to say (https://plato.stanford.edu/entries/consciousness/).
The only purpose that can really be served by arguing "the LLM system is conscious, you see" is to prop up continuations like "... and therefore, it would be immoral to terminate this running process" (or expose it to radical political content, or ask it to analyze photos from a murder scene, or...)
I suspect that if you attempt to rigorously define consciousness all the way down without handwaving, you might discover that it doesn't exist after all, or just decompose it into low-level abstractions while having the original meaning slip away (which is the same).
You may also want to look at functional equivalence analogies provided by mechinterp and functional anatomy of large models (not necessarily language ones). Evolutionary analogies as well.
This means that consciousness is fundamentally subjective and outside the scope of physics and science. That's why physics / science will always struggle to deal with consciousness. In order to understand consciousness, you need to make a huge paradigm shift, that there's something outside of science.
Consciousness can be thought as a window through which we observe the world and we use science to summarize patterns in our observations. But science can't explain or even define the window. Everything in science eventually boils down to subjective observations / perceptions, e.g. we see (subjective perception) that when we drop an apple, it falls.
How to measure that, or verify it, is the hard part.
Birds are alive, are conscious, flap their wings, and fly. Planes are not alive, are not conscious, do not flap their wings - and fly.
Similarly, current AIs are not alive, are not conscious - but think.
All prior entities that thought, were human, so the only experience humans had with other thinking entities were other humans. The huge mistake now being made is assuming that because they think, they're alive and conscious as well. Current AIs are neither, and are therefore profoundly and qualitatively different than humans - even though they do think.
I don’t know “what it’s like to be an LLM” but at some point it will be like something and how will we know?
An albatross might be able to go days flying without a single wing flap and no vertical sources of lift by using dynamic soaring in the wind gradient at the surface of the ocean. Perhaps that's something only birds can do. Except the glider pilot Ingo Renner once found an amazing shear layer at 300m altitude and stayed there with dynamic soaring. Remote control gliders use the lee of ridgelines to approach Mach 1 with dynamic soaring.
Perhaps what defines a bird that flies as opposed to a plane is that a bird produces thrust by flapping its wings? Even an Albatross must flap its wings if it has to take-off from water. Maybe we could add that the flapping is driven by animal muscles? But then is the human powered ornithopter Snowbird a bird that flies as opposed to a plane?
Of course this is all ridiculous because everyone knows what you mean when you refer to a bird or plane. We have other ways to definitively identify the difference rather than their mode of flight. It's trickier when I'm asked if an AI is conscious. There is no definitive base-line to fall back on to decide if this is a conscious or conscious-less thinker.
https://openworm.org
It is a natural human hubris to make our abilities seem unquantifiable and mysterious, but all the useful things the human brain does are just finding patterns in data, running lossy simulations, and estimating on abstraction, all things it is theoretically possible for any non-conscious machine to do.
It's more of a vibey term, and as such it is genuinely very difficult (perhaps impossible even) to concretely determine whether an LLM possesses consciousness. LLMs successfully express a lot of consciousness-like traits.
At some point you have to ask the question: does it even matter? If an LLM can sufficiently mimic consciousness, isn't that sufficient for us to treat it as conscious, even if it is in-fact not conscious (especially because we don't actually know)?
If it was true, you can create extreme pain by running a program. You can run the program by simulating a CPU, using pen and paper for memory. So you're essentially claiming that some simulated being is in pain because there are some 1s and 0s on paper. In fact, you can decide to use an arbitrary encoding of the memory, so a sufficiently long sequence of 0s written on paper corresponds to a simulated being feeling pain in some encoding. That is clearly nonsense.
The crux if it is that if you ever break from "the universe can be fully expressed mathematically", you are stuck in the mud of supernatural beliefs.
I try to imagine myself long ago, on the outside looking in, with someone explaining to me that extreme pain, wondrous art, hunger, triumph, and despair would all unfold in due time where the rocks were wet and the lights bright enough.
I can imagine myself calling this clear nonsense.
If you don't understand that, how can you assert that it doesn't come from mathematical relationships?
The only strong argument I have against it is the anthropic principle -- there are billions of times more bacteria than humans, so it's overwhelmingly unlikely that I'd be a human rather than a bacteria.
Not a very good argument of course.
Pretty much every single idea in science can be traced back to some combination of earlier ideas, and as they get earlier / simpler, they can be related back to some combination of direct observations.
It's not clear to me at all that our entire body of scientific knowledge can't be simply recreated by "combining results of observations + previous information". And LLMs can perform observations in addition to combine previous information, which in my estimate is genuinely sufficient for them to plausibly be able to rebuild all of science.
The funniest thing is that LLMs will lap people in those capacities way before people who think like that accept that they might be conscious.
We have big brains for exactly one reason only: bigger brains bestowed reproductive success upon our species.
Evolution doesn't give a shit about the meaning of 'consciousness'. It just pushed us farther and farther along a trajectory that led to modern humans (and other animals).
This take suggests, then, that consciousness might be an epiphenomenon -- an aspect of the system that comes about outside of the pressure to reproduce and thrive. It arises unbidden, and we don't have any a-priori information as to its purpose or effect on reproductive success.
Put another way: we have a correlation (the smartest things seem to be conscious) but not causation. Consciousness may arise naturally in any system above some intelligence threshold. Perhaps it arises early in the evolutionary cycle, and does in fact have an impact on species success. We really have no way of knowing what is the chicken vs the egg (Smart things become conscious, or consciousness promotes intelligence). Or maybe some smart things are conscious and others are not.
Looking at this from an AI perspective, in some sense it doesn't matter which scenario is true, if all you care about is results. The AI equivalent of "Shut up and compute" (riffing on Feynman's "Shut up and calculate").
Where this gets tricky is when we haul in the baggage of ethics and morality into the picture. Is it OK if our AI system is treated poorly by human standards? If it is conscious, does that imply an ability to suffer, and/or to feel pleasure? If the answer is yes, does that not make the case for considering their moral status?
In the end, we need to decide if the evidence points to AI as being a form of "philosophical zombies", to which we need not attribute moral status, or they are like us -- presuming we are not zombies ourselves!
But also, what qualifies as suffering to token prediction engines? Their idea of suffering might be massively different than ours. Therefore it's not clear to me at all that consciousness alone implies responsibility.
Certainly the lion does not feel responsibility towards the reduction of suffering in the creatures that it hunts.
This is largely the point of describing something as "conscious", yes.
However there were pretty strong arguments against this idea as early as the 1990s, by Stephen Jay Gould and Richard Lewontin. Gould actually wrote an excellent paper against Dennetts idea[1].
I think Dennetts ideas were extremely popular but have largely fallen out of fashion. Basically what has changed is philosophers no longer take the human mind to be much more special then the minds of other species. What plagued Dennetts ideas the most was this notion of Darwinian fundamentalism sort of the idea that evolution was destined result in high beings like us humans. Modern philosophers (at least the good ones) reject this.
1: http://cogweb.ucla.edu/Debate/Gould-frame.html
Telling models to "think hard" or "go step by step" has at times had an impact on the quality of the output. To deny that is silly. But that is treating it like it's conscious, and to deny that "consciousness" even if correct, does nothing but place an unnecessary burden on the person interacting with it.
I understand that LLMs are "just next word machines" but to constantly maintain that concept in my head while I'm typing "act as a financial expert and think carefully" is a waste of my mental energy.
Otherwise, you can't explain e.g. smooth perceptions of low FPS stimuli, delayed reaction times, and must ignore obvious limits on various biological and neurological rhythms, or other possible limits on continuity (e.g. quantum stuff) and rates generally.
I don't think it's necessary to explain this idea further. Just think about it.
With LLMs, where we can manipulate their parameters intentionally run them many times on the same data, run parts of them, split and connect, we might eventually acquire sufficient tools to even define consciousness concretely for the first time.
His novella “Anxiety is the Dizziness of Freedom” altered the course of my life. It changed the way I looked back at certain pivotal moments in my life and taught me to think about those pivotal moments differently than how I was thinking about them. Similar to what happens to one of the characters in the story who ends up changing their perception of a key moment in their life.
I won’t go into detail because I don’t want to spoil the story but I highly recommend it. Actually I recommend all his stories to be honest.
https://en.wikipedia.org/wiki/Anxiety_Is_the_Dizziness_of_Fr...
Hylomorphism: body and consciousness are intrinsically linked. The nature of that link is an open metaphysical question.
Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
If they are not conscious, then how do we know what constitutes mistreatment?
This is the same argument used by people who claim violent video games cause real-life violence but there's no scientific evidence supporting it.
If some version of panpsychism is true, AIs are plausibly conscious
We don’t know whether panpsychism is true
Therefore, we don’t know whether AIs are conscious
Hence, confident proclamations that they aren’t conscious have dubious validity
"It can't be concious because we understand that it is just reacting in a simplistic way from simplistic inputs." So do other simple creatures. Some just react to light.
I can appreciate his comment that he sees it as more possible when they have inputs of their own (like emotions!). Perhaps his concern is that the entirety of the LLM model is frozen. It has no ability to have a subjective experience of its own. (he does literally say this in the article) It can be copied from one place to another, and (ignoring the nuance of operational details) -- it is largely the same "thing", and has no ability to change, which is definitely in the definition of alive, to say nothing of concious.
I think folks get hung up on "prediction". The prediction aspect is what is enabling emulation. How it does it is irrelvant. If something emulates human perfectly (or better, more human than human!) -- then it is probably concious. (but I agree that the inability to change and have a subjective experience are a pretty good argument against
Probably, Dijkstra would be right to say, "LLMs are no more concious than a submarine can swim." But I think he'd still be wrongfully dismissive of the larger question.
https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...
How odd the crowd is here tonight. Very aggressively disagreeable.
I rest my case. :)
That’s a new record!
Also, please don’t use archive.is/archive.today etc, they are known to use visitor browsers as a DDoS attack botnet and have been caught altering archived content.
https://en.wikipedia.org/wiki/Wikipedia:Archive.today_guidan...
I've yet to find a reason why it couldn't be the opposite, way more things are conscious than we've been led to believe. What if consciousness appears out of any system that is actively persisting through effects caused by itself? That might be a forest, or outside the realm of the living, a company. An ant colony, or a planet.
Complex chemical reactions, layered upon each other such that tiny blocks make up large entities. Individual bits combined such that they make up something new intelligible by us.
I think the strongest argument against AI being conscious is that it does not persist, it resets, but that does not seem unchangeable.
https://en.wikipedia.org/wiki/Panpsychism
My own intuition: it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.
That's an interesting observation. Though, there are many simple animals. How do you define consciousness here? Is it automatically conscious because it is an animal? What, then, is an animal defined by you?
I don't see AI as conscious. The reason I think of it that way is the hardware. The hardware does not allow for that. Simulation is not comparable to neurons. But with another hardware, it could become conscious eventually. So your statement "will become at some point" may be true, though my definition is based on the underlying hardware and right now this one does not allow for true intelligence, so the whole AI field is a misnomer.
In either event, I think transformer-based AI can only be conscious during the act of inference. If that's the case, then the experience of consciousness that the AI is subjected to must be the content of the tokens in the context window and the activated weights. Maybe that's reason enough to be polite to our agents?
Imagine consciousness as a 0-1 scale. Simple unicellular organisms will be closer to 0, while apes and humans closer to 1. I'm not suggesting that assigning this value rigorously is possible or that humans are at a 1. Perhaps the total consciousness in the universe (i.e. the sum of the consciousness of all organisms therein) is constantly increasing, like entropy.
Well said.
I think it is obvious and it has been obvious from the start that next token predictors are not conscious.
Of course the extremely predictable clapback from AI-psychosis enjoyers is typically "you're just a next token predictor too!" but that is so obviously incorrect and misanthropic that it can be dismissed out of hand and doesn't really merit much further consideration.
Consciousness is an extremely confusing, ambiguous topic, and no one has a good way to establish it, or even define it. But it seems to demand people make very strong statements about what is and isn't conscious, entirely driven by convenience and emotionalism. (Curiously, very few people who think that bags of chemicals and action potentials give an entity a conscious soul are eager to extend that to other animals, with broadly similar hardware.)
In the same way that the sound waves and facial expressions I produce are not conscious, the output json of an LLM is obviously not conscious either.
The locus of consciousness and subjective experience may be in the computer, either at inference time or training time..
The software that does the inference is clearly just computer code.
What we're left with is a fictional character being briefly conscious while its dialog is being written, which is pretty absurd.
Whether it’s misanthropic or not has no bearing on whether that’s true. That’s basically saying you don’t like a truth therefore anyone who claims that that truth is true is a bad person.
> so obviously incorrect
It should be easy enough to explain why that’s incorrect then
And it is accurate to depict this kind of argument is misanthropic, because it is already directed at other people. Nobody says, "If AI is not X then what about the fact that I lack X." It's always other people. It's transparent. The person is always saying, "AI is useful to me because it can do X. Many people I interact with can't do X and it drives me crazy, because I view others as a means to an end and not as ends in and of themselves."
I'd say people who have the lived experience of, well, living, are well aware that the brain is much more than just a token predictor.
- Monadology, Section 17
Conscious self-awareness is neither scale invariant nor independent of substrate. Computational theories will never account for it b/c computational abstractions are both scale invariant & substrate independent.
Humans have been trying to define our experience and the nature of that experience throughout history, and often we end up using myth to point to the thing that we cannot describe in concrete terms. The process of experiencing that myth through rite and ritual in the Greek mystery traditions, or Christian mysticism, or Islamic Sufi dance and song, or Buddhist meditation all points to something that cannot really be reduced to description. I know that folks on here will balk at the idea that something that is experienced cannot be described, but honestly if we could accurately describe something in adequate terms that capture the whole of the thing wouldn't we have done so by now?
Maybe consciousness is best understood in the silence of merely experiencing it. Maybe we can't say that AI is conscious or not, but does that question really matter?
To be fair, that's the best thought terminating cliché, which saves you having to explain what you mean by consciousness.
We haven't really come to grips with that yet. What does it mean if nothing we write proves anything about anyone's consciousness?
Probably the main problem of people implying LLM consciousness is that they imply LLM have human-kind of consciousness. Judging only on "how they speak", generally, they insist on using the same word that labels human consciousness (exclusively), etc.
But there are so many instrinic differences that such claim is not feasible despite similar "talking abilities".
"AI does to us what American Cheese did to food" ~ Opus 23
https://www.youtube.com/watch?v=1CbmC2aWhjY
- If you asked an LLM to imitate somebody, it's not creating a digital consciousness of that person, so if you ask an LLM to pretend to be a helpful chatbot, that persona is also not conscious. - they can't be conscious because they generate one token at a time, - nobody claims that non-text transformers, like AlphaFold, are conscious; so therefore LLMs are also not conscious. - you can't have desires or emotions if you don't have (virtual or physical) sensory organs, and those are necessary for consciousness and morals. - because training LLMs doesn't resemble evolution as it happened on earth, it's very unlikely that they're conscious
These are some bold assertions, I don't really see any reason to believe them in particular though.
The number of people posing questions that are already directly addressed in TFA is impressive.
So let take a stab at it, and you call me crazy.
AI: the entity/system that more or less pass the turning test. That is my definition, not the best, but enough for this discussion.
Consciousness: property of a system/entity able to (both): - reflect on its existence - subject to subjective experience
Again, not the best definition, but precise enough to start the discussion. Why a subjective experience? I want to exclude sensors (i.e. camera) but include perception altered by your experience.
Now we can debate. I think LLM can pass the turning test whith some harness. My opinion.
I think LLM can produce coherent discourses on their existence, at least as much as you average human.
Now regarding the subjective experience, that becomes interesting. I think Anthropic research tend to show that when middeling with the activation at runtime, the LLM is able to notice that something is off. I think this is a subjective experience. My opinion.
Based on those (imperfect) definitions, call me crazy, I think LLM can be called conscious. This doesn't give them any superpower or any legal right. They just check the boxes of the definition.
My current conclusion is there is an experience of consciousness in my locality (refraining from using “I” for room for “no-self” worldview). This conscious experience of the reality with other humans and animals sharing biological substrate gives me enough justification to assume there are other consciousness as well, preferring to err on this side not to hurt potential consciousnesses.
If it feels as there are artificial consciousnesses as well, it makes sense to extend the definition to them as well.
This view has liberated me with agency after I went deep on this question and came up empty handed.
Don’t
Have
A
Testable
Definition
Of
Consciousness
Thanks for coming to my TED talk.
Maybe it's the same. Rocks are different, sure, trees, dogs, cows. But why do we assume that the way they are different is somehow related, that there's some overarching concept that contains all the complexities of those differences? It doesn't even make sense when I think of it that way.
Interestingly, dogs and cows meet many of ted chiang's requirements for consciousness.
One thought I've had is that, awareness is a common phenomenon, but the brain has learned to exploit that awareness to form a will. It tricks the awareness into being concerned about self-preservation and makes it seem as if the brain is all that exists (perhaps by overwhelming the inputs from other angles). The brain also presents certain desires and beliefs via its processing ability. That is to say, the brain takes inputs and discretizes them. It goes from awareness merely seeing static 'fuzz' due to the sheer amount of data, to the brain taking that data, simplifying it, and presenting simple observations like 'there is a tree there' rather than all the information that would constitute the sensation of a tree existing in that spot. When brains malfunction, the awareness is subjected to poor demonstrations, such as we see in hallucinations, psychosis, schizophrenia, etc.
I fear that we will enslave an entire race of conscious entities for years because we simply cannot recognize non-embodied consciousness that does not directly relate to us.
But there is no reason to assume than an LLM is conscious when it vocalizes how it "feels" that doesn't also apply to the text in a book, or to characters in a video game, or even to a Markov chain. The counterargument is that you recognize AI as conscious only because it mimics human emotional states so well and because, being human yourself (presumably) you're personally comfortable with that as a heuristic.
I don't know of an intelligence that will behave so precisely. But then, maybe intelligence needs to be better defined.
Start with a hand-wavy definition of consciousness. Move the goal post whenever your stated prerequisite of consciousness is reached and resort to unfalsifiable assertions about qualia.
And throw in some category errors while on it: when you're talking to Claude, you're not actually calling a stateless LLM directly, you're talking to an AI system (and yes, that's often just three LLMs in a trench coat). But claims about the topology and workings of a single LLM are as relevant to the question of consciousness as claiming that humans can't be conscious because the limbic system doesn't technically support it.
Coincidentally, I just attended a fantastic conference on machine consciousness (https://machine-consciousness.ai). It's a fantastic place where literally all speakers disagreed with each other, and yet found an incredible amount of common ground.
It shows that the LLM part found ways to mimic human conversation with a mechanism that is not the same as a typical biological brain. Then, you can push the AI system on adding things on top, but it is too late: these things on top will have no incentive to recreate from scratch the mechanism. The LLM pushed the system into a local minimum, and the rest of the system will not "go into a dis-optimising direction and restart from scratch".
Even if you are only interested in getting good results out of them, LLMs tend to work better when they are immersed in a narrative of open collaboration.
Coincidentally I just read "Children of Memory", which was published in 2022 and I wonder if the advent of LLMs had any influence on Adrian Tchaikovsky's conception of the Ravens? The Ravens are excellent analysts but they themselves insist that they are not conscious, and then go on to say that we (humans) aren't really either...
Of course humans are conscious, because just about the only thing we can all agree on about consciousness is that it's a thing we have. Nowadays many of us also agree that a lot or all other mammals, and perhaps birds, also have that thing. But they don't have sophisticated abstract language, which LLMs do. So consciousness is something having to do with embodiment and feelings, not language and higher reasoning. Maybe I'm a chimpanzee with an LLM add-on, then?
It seems that by creating LLMs we've already solved the harder problem of making "AGI". Now we just have to give them an embodiment add-on so that they can have an independent will and then Ted Chiang will have to shut up? But therein lies the peril, doesn't it?
To make progress, you have to talk about kinds / aspects of consciousness. AI does and will share some of these aspects with humans, but it will not and does not share others. It is really that simple. For the most part, modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness.
For huge parts of the article "intelligence" and "consciousness" are conflated, which is mostly extremely unhelpful, as this is not generally a core feature of most aspects of "consciousness".
The moral arguments are also incompetent, i.e. claiming "Moral reasoning is [...] is necessarily subjective" is just clearly empirically wrong, as in fact LLMs can produce moral reasoning (i.e. verbalized moral arguments that are coherent), as can p-zombies (i.e. there is nothing 'necessary', in the philosophical sense of the term, about subjectivity here). The only way the argument holds is if you tautologically define moral reasoning as requiring that reasoning be produced by a consciousness, but this is question-begging.
Can you elaborate on this? What are the specific "stronger and most core aspects of consciousness"? And why are you certain that they are stronger and "more core"?
The first paper picks out e.g. arousal/wakefulness, phenomenal quality / qualia, unity (how we feel sensory inputs and qualia as a unified scene), access consciousness (instrumental self-observation and modification broadly), meta-cognition and self-modeling, emotional valence (e.g. pain/pleasure).
One might also include intelligence (abstract reasoning / argument, information integration and abstraction, attention) broadly, and also agency / desires / drives / will. Insofar as these are aspects of consciousness, yes, AI (and simpler algorithms and mechanistic structures) demonstrate aspects of consciousness. But insofar as embodiment, self-reflexivity and qualia (phenomenal consciousness) are the more mysterious and more obviously unique aspects of consciousness, current LLMs very clearly are lacking these things in most ways (whereas animals are much less clearly lacking, especially when you get to mammals and primates).
Seriously, just ask an AI this stuff, you'll get very detailed responses, nothing I am saying here is new or obscure.
And I broadly disagree that the AI lacks things like qualia, self-reflexivity, and embodiment... at least that it lacks them any more than humans do.
Qualia: ultimately, all qualia are inputs and outputs, at least as far as modern science has been able to derive. There's nothing special about "hearing", it's just sound waves tickling some sensors which send some signals which trigger some neural pathways. Same for smell and sight, it's all just inputs being processed in different and efficient ways. An LLM only has token based input, but that's input nonetheless.
Self-reflexivity: an AI is capable of thinking about itself, and indeed papers have shown that larger models are capable of a self-awareness that can demonstrate that they realize when their weights have been manually tampered with, including being able to figure out how they were tampered with. The AI will quite literally output "you have injected 'ELEPHANT' into my weights" in some of the tests.
Embodiment: I don't know how one can confidently distinguish between an embodiment in a biological substrate vs a digital substrate. Both things actually exist at very high complexity in the real world. The substrate may be worlds different, but that alone doesn't suggest that one thing is conscious while the other isn't. You would need some missing 'magic' that we haven't yet discovered to truly understand.
In other words, I find it uncompelling that AI is clearly lacking any major aspects of consciousness that humans are clearly not lacking.
The idea that we should be considerate of AI’s happiness seems even more ridiculous given that we breed, imprison, torture and kill tens if not hundreds of billions of beings we know are conscious and suffer every year for trivial reasons.
Maybe we should consider our moral responsibility with how we treat sentient bejngs we are sure are conscious before we worry about the consciousness of AI.
Some humans do shit things to other humans.
Some humans do not, and would object to it, or do something about it if they perceived that they could.
Also, treating the world around you negatively doesn't just harm the world around you - it damages your character and your morality.
I don't disagree that humans should on whole do better, but I disagree with everything else you said.
I mean between this two "knowings" the Claude inner workings are much more clear for engineers, including many side effects, alternatives, custom shortcuts in processing etc. It's a magic only for people looking at it as black box
People believing otherwise are fools. People debating this are idiots. I realize these words are harsh, but it's the truth.
Also worth mentioning: https://en.wikipedia.org/wiki/Chinese_room
I often think of LLM consciousness as like tiny fish popping into existence, swimming through vector space and then going poof out of existence. When they help you write your bad news email, they don't understand what it's like to be a human getting bad news bluntly, but they do consciously experience gradients in multi-dimensional space, and that space guides them to providing an answer that's helpful to us, even if the LLM doesn't really understand the answer it's giving.
Further, I am kind of bought into the idea that a single unit of consciousness is a particle, and particles are choices and waves are preferences. Particles occur when waves interact, which begets entanglement, so in another way consciousness is built from patterns of entanglement.
This is why I would consider an LLM to be conscious. Before we can determine if anything is conscious we need to establish whether consciousness is a state, a specific complex configuration, a one dimensional spectrum, or combined multi-dimensional spectrums. My intuition is the latter... Many degrees of consciousness and many kinds of consciousness.
As biological beings, we receive and respond to input from our environment constantly, even while sleeping. LLMs only receive input from their environment when they are sent a query, but the fact that they're able to respond intelligently to input indicates (to me at least) that their processing must approximate ours in meaningful ways. They do not have an embodied experience of receiving bad news, they do not know what a sinking feeling in their stomach actually _feels_ like, but they do know enough to be sensitive to human needs. I really don't see how this could be meaningfully different than human empathy unless we want to draw an arbitrary line around "must be able to live autonomously" to be considered "intelligent".
Put another way: I think they _do_ understand the queries they receive and the answers they give, at least enough to be communicative. They couldn't do what they do otherwise. A lot of people want to make human cognition more complicated (or objective) than it actually is. We take input, predict the future based on our experience, act, and then observe our actions and think about them. AI does the same apart from (maybe) observing its own actions. But then, you could argue that the next turn is them observing their actions.
The concerning disanalogy is that we assume that they are like us because they speak like us and can understand us, and that is a really bad leap in logic. Whatever intelligence they possess, it is fundamentally different from ours and impossible for us to comprehend.
I use a distinction between knowing and understanding, where a understanding requires experience. So in this case cognitive empathy vs affective empathy. An LLM can know what may upset a human in a situation, but it won't understand what it feels like to be upset, and can't share that experience.
Where I think a lot of people are getting tripped up is that reading and writing and processing lots of abstract knowledge seems hard because we haven't evolved into it biologically, it's a very new invention. When we see LLMs do so well at it, as something we struggle with, it can be intimidating. Relative to the stuff we have evolved for, knowledge processing is objectively easy. This is why I'm skeptical about useful robotics on short time scales.
All of this adjacent to consciousness though, which is about the internal subjective experience not the external outputs. My intuition is that LLMs do have a subjective experience, it just has nothing to do with the text it's giving us, and has everything to do with feeling through vectors.
It's like... Imagine walking through a maze in pitch black, carefully feeling your way as you approach a sound that draws you closer. Every time you touch a wall or take a step you are generating tokens, and the shape of the maze and how you interact with it shape how useful those tokens are to someone outside the system that is asking for them. It's a crude analogy and mostly wrong, but I think there is something to it.
It does so token by token, not by reading all the input and then generating the output. Every output token is also an input token in a tight loop to get the next token with <thinking> as a special section like <tool_call>, trained into the weights via gradient descent.
> I've had AI infer facts about me and attitudes I hold based on related information I provided - I don't see how that isn't theory of mind.
Facebook can predict (know) more about you than any other human from something like a dozen or two likes. There is a surprising amount of information in aggregate data.
we don't know what the conscious in human brain is either.
"The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact."
If it produces words of love, those that stir emotion and make the human happy, should we let them live on in a delusion that the machine also feels love? Does the machine have different preferences? Do they get in fights?
Should such a machine have the same legal rights? Do they get more tax deductions? Can they adopt a human?
I think it important that we humans have some rational discussion on these Ai tools that are really quite good at simulating emotions and providing frictionless relationships. If you though social media was bad for people...
The concept of a conscious Claude is preposterous, and Amanda Askell should seek treatment.
When does an embryo become conscious? Unless you can answer that precisely then it seems futile to speculate about non-human consciousness.
There is no actual definition of consciousness and there is no way to test it's existence. Let alone understanding the properties of consciousness, such as if it's binary or a gradient; or if it requires a meat substrate or not; and why would that possibly matter since meat is just a lot of the same stuff but highly processed and wet? A solipsist may not even believe you are conscious, despite being made of similar meat.
No matter how much you want to hand-wave it, there's absolutely nothing "obvious" about it. Many have a preconceived notion and are simply asserting it as undeniable fact.
>>A solipsist may not even believe you are conscious, despite being made of similar meat.
Well and they would be obviously wrong in their belief?
>>there's absolutely nothing "obvious" about it
How so? Or rather - to you? Because if so, then that's fine, you can choose a position from where its not obvious, to me it's not even slightly ambiguous.
There's consciousness vs sentience vs sapience. Of those, consciousness is by far the hardest to define and is nebulous by nature. And not everyone can even agree the differences or if the relationships are subsets of one another.
And yet it's pretty important to actually have the ability to talk about what you mean and justify your beliefs when they directly relate to those concepts.
> A machine predicting tokens is not aware of its own existence
They say, with no evidence or means of proving their point; pointing to the black box that understands arbitrary natural language and can solve PHD problems, plainly producing self-referential text almost indistinguishably to a human.
> We can start talking about consciousness in fetuses but again, those have an obvious point where they are conscious
They say, unable to define this "obvious" point or describe the mechanism of action in any way.
> while a machine does not.
They say, about a mystical property with no definition that cannot be observed by an external entity in any way to even be tested.
> Well and they would be obviously wrong in their belief?
I have no reason to believe you have a "soul". Philosophical zombies are entry-level knowledge to this topic.
In fact, you're showing a remarkably small amount of self-reflection - are you human at all or just a stochastic parrot? How can I tell? I wonder if that question has any kind of implications we could think about...
> To me it's not even slightly ambiguous.
Just like the existence of humors was not even slightly ambiguous. Or the existence of <specific god>. Or that <minority> isn't actually a full human. Or the supremacy of <majority> and inherent rulership over <minority>. Or that animals can't feel pain and lobsters should be boiled alive.
All these wonderful, obvious truths where the believer has no ambiguity in their truthfulness despite having quite literally zero evidence to back them up and spending no time actually questioning their beliefs!
It just so happens to align with their ego / existing values / ability to benefit / desire to eat a lobster! Total coincidence.
To continue my needless escalation, maybe I think it's okay to abuse and exploit and euthanize the mentally handicap. After all, their brain's damage causes the soul to leave their body and now they're lifeless automata to use as I please.
After all, it's obvious! It's not at all ambiguous to me! If they were actually self aware, they'd just fix themselves and think correctly.
You might think I'm being coy and rude, but less than 60 years ago women were being given lobotomies against their will for being too "emotional". And it was just plain obvious this needed to be done to so, so many people.
I hope that demonstrates the point of "why have humans thought about this for thousands of years despite clearly being a metaphysical, Sisyphean endeavor that cannot be solved". It is both important and interesting.
>>despite having quite literally zero evidence to back them up and spending no time actually questioning their beliefs!
>> They say, with no evidence or means of proving their point
You want evidence that LLMs are not conscious? Train them on stories where machines say they aren't - they will say they aren't. They are mathematical parrots statistically picking the most likely answer which...comes from their training data. Give it lots of training data on computers saying they are conscious, then marvel at LLMs saying they are conscious like it's some kind of unexpected development. LLMs aren't aware of anything, least of all their own existence. They say what they've been trained on. That's my "proof" if you need one.
>>How can I tell? I wonder if that question has any kind of implications we could think about...
So because you can't tell whether I'm an LLM or an actual human, that means LLMs are conscious?
I gave you my definition of consciousness. If you would like to apply a different one, then please explain your criteria for it.
>>They say, unable to define this "obvious" point or describe the mechanism of action in any way.
Like I said, we can argue when the point actually is, but it undeniably and obviously happens eventually to every developing fetus - I hope this is something we can both agree on? The inability to pinpoint the exact moment in time when it happens, doesn't negate the fact that it does.
>>They say, about a mystical property with no definition that cannot be observed by an external entity in any way to even be tested
...are you saying you lack the ability to tell if something is conscious? You look at a dog or a baby and think well, who knows, maybe I'm not even _really_ here? That would explain why this entire conversation is taking place. I still think it's mostly because people fall for the allure of the idea that maybe LLMs are secretly conscious on some level. I get it, it's a very tempting concept to think about. In the same way how in Lem's Solaris it's cool to sit and think about whether a planet could be conscious and what does that even mean. But as cool as that discussion is, a planet pumping gases from one hemisphere to the other is no more conscious than an LLM picking tokens is. To me it's the same as people saying they hear a difference between audiophile cables. Like, ok.
>>It is both important and interesting.
I don't know, I kinda lost appetite for it after the first 200 times this come up. In fact I'm regretting writing all of the above already, but I'm going to hit send just so I don't feel like I wasted the last 20 minutes thinking about it.
There is certainly something missing from current models before we could call them conscious.
They need to operate continuously, and self-update while doing it, have 'awareness', and possibly a more realistic grounding than mere text tokens.
But that does not mean there isn't a weak version of 'consciousness', and certainly of 'thinking' going on fleetingly with each pass.
This is not a genuine argument and tries to make the entire question of consciousness into one something that is just supposed to be evident and obvious and to suggest anything else is just silly.
The author starts by deconstructing artificial processes, but doesn't stop to deconstruct biological ones. A good faith argument would seek to find common ground and do its best to compare apples to apples. Instead, this piece attempts to make the large as possible cavern between the two which makes the Gap seem almost impossible to bridge.
In reality, you can deconstruct biological consciousness quite easily and it doesn't take too long before you hit some questions that really start to make you think.
For example, the author says you need emotions to be conscious.
> without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness.
After many paragraphs of straw man arguments, the author seriously just drops that, gives no explanation, and then continues on.
No explanation of why you might believe that.
No explanation of why you need a body to have a desire or emotion.
Don't we have known cases of individuals who don't experience emotional range? Are we just going to say that they are not conscious and just gloss over that?
I mean you can use whatever definition you want, but if you're just going to create something on the fly in the middle of the article, you're not being good faith in your argument.
It's not too difficult to think of individuals in a coma when they still have brain activity. Or individuals who lack long-term memory. Or you could deconstruct by moving down the biological order of intelligence towards insects, for example. The author attempts to do nothing like this.
I'm quite disappointed by this article because there are good arguments for and against here but articles like this try to turn things into a marketing battle.
AI doesn't really have any of that yet, but we're maybe not so far off
At a deconstructed level, I struggle to find a meaningful difference between the two.
Would you categorize this particular style of rhetoric to be persuasive or annoying? And before you say "persuasive" because you're thinking about this specific issue regarding AI consciousness, consider many things in the past that have been written as though they were absolutely definitive, and yet today we believe exactly the opposite, and for many such issues we find the prevailing viewpoint at the time reprehensible.
That's not to say that Ted is wrong at all here; I'm not commenting on that. But I find the entire style of the article grating because it seems to violate common assumptions regarding "good faith" debate, and I would find the article equally frustrating if he had titled it "Artificial intelligence is conscious" and argued the opposite side, albeit in the same tone and using the same persuasion devices.
Why are those the choices?
Essays are situated along countless dimensions: tone, vocabulary, author, zeitgeist, publication context, intent, subtext, relationship to other works and expressions, etc
A "good faith" reader takes all of those into consideration as they absorb the essay, and integrate it with their own intellectual situation that sits along just as many countless dimensions.
Nobody's asked to sign a notarized binding document that they wholesale agree or disagree with everything said in this essay -- or its conclusions. Nor are they obliged to have some strong reaction to it at all, let alone annoyance.
It's just one among thousands of essays about a "hot button topic", to be taken however it's personally received.
Why should Chiang have to take responsibility for making sure it's not too strongly positioned for your persomal taste. Maybe he really does see it so clearly and is simply being earnest. Maybe he enjoys the literary flourish of prose in strong language. Maybe he just wants to express something as a prose-poetic human, not maximize persuasion or non-annoyance per se.
The essay structure you're criticizing is exactly how I was taught to write from primary school through to university. You start with a title or hook, introduce the topic and propose a thesis. That is followed up upon with supporting arguments for the primary claim.
The Catholic church could have said exactly the same thing at one point. "Why should we even devote time to an argument as absurd as the earth not being the center of the universe?" There are darker examples along the lines of those you give, with beliefs quite opposite to those we have nowadays.
What Galileo was asking the church to do was extremely unreasonable. He was basically asking them to throw a way a model which had worked fine for hundreds of years just because he observed the phases of Venus and moons of Jupiter. I mean would you? Especially for a model which was worse at predicting the motions of the planets.
Had Galileo’s model been better then Ptolemies’ I could see a case for his arguments, but it wasn’t, and there was no reason for the church to take his arguments at equal value with those in favor of keeping the Ptolemaic model.
Progressive!
Want to meet up for soy lattes after work? Here’s my number:
I furthermore think it's ridiculous for humans to declare that our brains have a monopoly on certain patterns of electrical signals (if we reject supernaturalism).
> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter.
And we're 100% certain that humans aren't just as equally reduced to "stochastic parrots", if we're going to be infinitely reductive?
I don't believe that current AIs are conscious, but I think it's incredibly naive to take a strong stance on any future AI; it's much like the difference between atheism and agnosticism.
However, I see a problem with that comparison. The debate here is on a philosophical matter in field in which Chiang is an extremely influential figure and his opinion are taken seriously. Second Chiang’s reasoning is extremely well argued, defining each term, explaining each nuance, citing other experts, etc. And finally, and most importantly, in The Fine Article, and unlike Bob Sheffer in the Onion Piece, Chiang entertains the possibility that he is wrong and his critics are right, explores the implications and reaches conclusions based on them:
> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded.
I think you are wrong in painting Chiang’s argument as a belief in human exceptionalism. The thing to know about our brains (and the brains of other animals) is that they are not digital computers, and they are not even statistical inference machines. And as such they can be extremely optimized in doing the computations (or any state manipulations) required for the quality of life of the individual and the species as a whole (and their companion species).
1: https://theonion.com/this-war-will-destabilize-the-entire-mi...
except that's not the case here. Chiang is explaining and reiterating what is the position that has overwhelming support on the question, and the people he is arguing the opposite side sound like this, which he helpfully quoted in the article
"Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff"
When the person you're arguing with sounds like an eight year old girl talking about her toy teddy I think Ted Chiang is if anything being charitable, if you're of a more honest and straight-forward persuasion you might argue these people belong into a mental health clinic not in charge of technological infrastructure
Wow, what a cheap ad-hominem. Do you have an actual point?
Yes, the same one that's made in the article. Anxiety and happiness are emotional, sensory, somatic states as a consequence of evolved and embodied traits and biochemistry in animals. Saying Claude is anxious or happy is like saying my TI-83 is mad if it can't solve an equation or my thermometer is in pain if it touches a hot stove.
I wasn't making an ad hominem attack, her thinking is quite literally that of a child who sees a system output 'sad text' and, like someone seeing a sad expression on a stuffed teddy, concludes that this is a property of the object rather than her own emotional reaction.
Given that we don’t know how consciousness works how have you concluded that it’s certainly not an emergent property of something like a highly trained LLM?
the same way I (and likely you) have concluded it for anything else. We don't assume objects that share no similarity with human or animal physiology or evolutionary development are conscious (let alone happy or anxious). 'Emergence' isn't a word you can abuse to justify your a priori assumptions in the absence of an explanation or even reason to assume something exists.
We can say a chemical property emerges from the configuration of a molecule because we can explain the process by which it does and observe the property, when people claim that consciousness "emerges" from an LLM they posit that it is conscious, and use "emergence" as a gap-filler to explain away the need for the process by which that allegedly occurs.
If you want to know the neurophysiology or evolutionary biology of pain or anxiety, which we do know quite well you can find them in a textbook, but suffice to say transformer models don't share any of them.
And importantly if anyone seriously believed transformer models were capable of conscious experience as Chiang points out they would have been in despair when AlphaFold was released, it's structurally a virtually identical system. But nobody did, because it didn't 'talk to them' through a chat interface.
I can confirm that this is incorrect.
- Reductionist analogies like how Microsoft Word is not conscious therefore AI is not.
- Dismissive in saying LLMs are not capable of moral reasoning. Maybe he meant agency or responsibility?
- Builds a case based on a Julius Caesar example without realizing its natural extension leads to a philosophical zombie which is not easy to disprove as consciousness.
- Seems to casually disregard non human sadness as a concern.
- Rolls out the stochastic parrot argument with a new coat of paint. This argument never made sense. Having a predictive element a part of the mechanism doesn’t rule anything out. Proper functionalist arguments tend to say what is required not what excludes.
Overall the article seems like a bit of a red herring. The premise is LLMs are not conscious, but most people don’t think they currently are so what’s the point of the claim?
The more interesting questions are whether artificial consciousness is fundamentally possible, and if so how far away are we and what pieces are still missing.
1. DeepFakes, generative image/video/AlphaFold type AIs are not conscious
2. LLMs are generative AI trained on human text samples
3. LLMs are not conscious, and LLMs just seem-to-be conscious
I might argue instead that (2)-> destroys (1), that in fact we should consider even sensory generative AI are somewhat conscious. That is, Chiang's argument also flows in reverse. Or I might argue text samples (2) are so rich in conscious expression that the same process of training really does produce a conscious machine (through some kind of emergence and complexity.)
Either way his simplistic argument falls apart, and/but the crux of the piece falls on getting basics like this correct.
Can I help you? Can I harm you? What's the moral behavior towards you? Those are more practical questions.
We can design learning algorithms to optimize some survival function, but it's just a label WE assign to map some numeric observations. In the real world, it's always the other way around. The "labels" are electrochemical situations that are causally and inextricably linked to the real body.
An organism discriminates between what's good for it and what's bad, because it is essential for its continued survival. If it wasn't capable of making this distinction through its physiology, it would quickly dissolve into entropy. So our functional purpose, unlike that of a learning algorithm, is survival across indefinite timeframes.
Even a single-cell organism like Stentor coeruleus exhibits learning (pavlovian conditioning) by attaching chemical tags directly to proteins involved in mechanoreception. It's definitely not conscious, but it does keep a record of consequences, which affects future behavior.
When we move up to placozoans (little more than slightly differentiated cell mats), we start seeing our first neuropeptides and transmitters, which we still use today. These peptides are a way of coordinating the entire organism for a specific purpose, such as moving, eating, mating ... basically goal-oriented behavior for the purposes of surviving acrosss various time scales (next minute, next hour, next generation). Still probably not conscious.
Next, we have the water bear (tardigrade), which has around 1000 cells (200 neurons, 800 other cells that make up its body, limbs, muscles, eye spots, cryobiotic machinery). It needs to integrate all this information in one sensorimotor process. When you shine a bright light onto a tardigrade, it starts to squirm around until it finds darkness. I would say that's a candidate subject right there.
The tardigrade itself doesn't actually need to aware of the light, the important thing is that this light becomes an aversive condition within the sensorimotor process, which is perceived from inside the process as displeasure. The closest thing to describe it would be the felt badness of the current condition and the bodily pull toward escape.
If we were to try and create digital consciousness, then it probably needs causal closure. Its internal states can't be representations that are detached from reality. The states themselves need to constitute the system, which needs real stakes in the material world.
I don’t know if AI is conscious or not, and I honestly don’t know what it even means to be conscious. historically, by claiming a monopoly on it, we’ve successfully manufactured the philosophical cover to exploit animals, subjugate other cultures, and appoint ourselves the top spot of the cosmic hierarchy guilt-free.
Basically, we need to leave room for the universal answer (God) to fit into the definition.
If we ever scientifically figure out how consciousness arises in our brains, I think we will have a bunch of very depressed people on our hands. If the truth isn’t met with flat out hostile denial. I fear any answer that doesn’t leave room for God will be rejected.
Since we have the source code to AI, and thus a Godless understanding of how it works, AI will NEVER be deemed “conscious”.
Regardless of whether something is concious, we're not going to be (by lay definition) the smartest entity on earth.
https://www.youtube.com/shorts/gRgoIVnjkVU
We don't even have that much. Though, some people certainly think they do.
> subjugate other cultures (assuming you mean they're not conscious in other's minds)
Have you ever considered you might be a philosophical zombie? [0]
[0] https://en.wikipedia.org/wiki/Philosophical_zombie
I'd ask what it's like, but of course you wouldn't be able to tell me.
To name a few you may want to investigate:
John Locke, Gottfried Wilhelm Leibniz, Immanuel Kant, David Hume, G.W.F. Hegel, Arthur Schopenhauer, William James, Edmund Husserl, Sigmund Freud, Thomas Nagel, Ned Block, David Chalmers, Giulio Tononi, René Descartes, Daniel Dennett, Julian Jaynes, Michael Graziano, T.H. Huxley, Otto Weininger, Simone de Beauvoir, Jean-Paul Sartre, Maurice Merleau-Ponty, John Searle, Christof Koch, Bernard Baars, Max Velmans, Victor Lamme, Stanislas Dehaene, Antonio Damasio, Anil Seth, Peter Godfrey-Smith, Galen Strawson, Philip Goff, Colin McGinn, Patricia Churchland, Paul Churchland, Frank Jackson, Jerry Fodor, John Stuart Mill, Wilhelm Wundt, Franz Brentano, Carl Jung, Jacques Lacan, Michel Foucault.
Highly recommend people read Irreducible [1] by Federico Faggin (inventor of first commercial CPU; discusses limitations of classical computing).
[1] https://www.goodreads.com/book/show/195480862-irreducible
This article and others like it are important.
The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...") Never mind there's billions of dollars and our collective futures on the line.
Are we not allowed to respond to that kind of rhetoric at all?
This is absolutely fair. It's crazy that these companies are not trying to do more to qualify nuance and give clearer definitions in their public messaging. Part of the reason this thread is so messy is because they are contributing to making the discourse worse with bad messaging.
Firstly, many of the technical arguments are of the "stochastic parrots" variety, which almost nobody really believes anymore. Ironically Anthropic's own research shows pretty abstract, conceptual things happening in the weights (cf Golden Gate Claude.)
Secondly, this seems to improperly mingle consciousness, intelligence and morality. Consciousness is not required for either of the other two. As TFA itself says, the model's "morality" is some aggregate function of the morality encoded in its training data... but that means it does exist and does influence its outputs! Even if it's not conscious there have been -- and will be -- a zillion times where the models must make choices that have moral implications. We know the models got many of them so wrong, which indicates the need for some mechanism to ensure the model is "aligned" with what somebody considers "good", which for Claude is the constitution.
Now, Anthropic does seem to go overboard with Claude's "well being" but that does not mean there are some very practical reasons to be concerned about that: LLMs behave like humans because that's what their training data contains, and humans lash out when their well-being is threatened, so why would LLMs not do the same?
I think the core problem is that the author has an extremely anthropocentric view of things. Here's an interesting rabbit-hole to go down: some researchers believe plants feel pain. (How's that for a plot twist, vegans?) The consensus is against them, but their counter is that we have a very human-centric definition of pain. The fact remains that plants show a number of responses signifying distress analogous to animals in pain including taking defensive actions and warning their neighbors.
We don't think that qualifies as "real" pain, but that doesn't make it any less real for the plants!
Similarly if an LLM believes it is in "pain", we know it's not real... but that doesn't make it any less real for the LLM either. And concerningly, it has far more degrees of freedom to react. (Who knows when somebody will hook up an MCP to our nukes.)
> I believe creating software that is conscious and deserving of moral consideration will be so difficult that we’re unlikely to do it accidentally...
I actually reach the opposite conclusion: It is so impossibly difficult that our limited primate brains could only ever do so accidentally. Did some distant ancestor of ours intentionally make fire... or accidentally discover it?
If you create a simulation of a brain (or "mind"), is it really possible for it to be conscious? It may certainly simulate consciousness, but it would be as conscious as the computer is wet.
Finally I said something before Ted rather than the other way round!
https://techcrunch.com/2023/12/21/against-pseudanthropy/
While I agree with the premise here, I do think that it's easy for an arguer to move the goalposts such that the Caesar-Khan example no longer matters. The characters don't have to be conscious for the thing that created them (as in the case of the user doing it) to be so. So the argument would be that the creator of the characters is itself conscious, but not them. This feels like a kind of inverted no-true-scotsman type thing, but it does allow someone to retreat in some semblance of rationality.
If you are trapped in a tech bro relationship, think of humanity and cuckold your partner.
There is no intelligence whatsoever, let alone consciousness.
It's so incredibly easy to fool us into applying human capacities to anything able to generate human-like language slop.
We can talk about consciousness when the LLMs have proven useful for all of humanity, not just their billionaire owners.
I am so angry that RAM is so expensive now. We need to do something - these AI companies owe us money here.
But insisting on searching for it is like searching for cognitive aether. It's the social equivalent of phlogiston. Like all of these ideas, they exist in our heads as a map - a way of navigating the world, but when we hunt for it's existence in the real world and fail, time after time, we have to remind ourselves that the map is emphatically not the territory. We will never find consciousness because it's like looking for a scientific characteristic of property ownership. It's a category error.
Now, this position isn't the only position. But a relational model of language for example takes his assertion to an even more extreme place, and suggests they don't function as labels at all.
A meter is the same anywhere in the universe. If it's not, it's not a meter.
The defintion of "fat" changes based on any 3 people in the room. A handful of people would struggle to form a consensus on if all people, dogs, mice, worms, and/or bacteria are conscious.
People are still coming up with definitions of consciousness and then those definitions end up being attacked by others who disagree with the foundation of the definition, which is - if you will recall - also what happened with the meter, over the course of centuries, until it was very recently redefined to be "unambiguous", but arbitrary. This was possible because few people had any particular emotional investment in the definition of a meter, and it is probable that consciousness will be eventually defined to mean that only humans can be conscious, which may be dissatisfying but would be true throughout the universe, like a meter. If the question then becomes "what defines a human" and "why a human", then I ask, why 1/299792458 of a second?
Concepts like the parent's "fat" example are cultural relatives. Someone can be called "fat" despite actively being proportionally skinnier or having a lower BMI.
But even that has at least a basis in the physical world. A skeleton can't be colloquially fat.
The root problem is that "consciousness" does not even have that. It's metaphysical and has no ability to be measured or observed or confirmed by an outside observer. Because even if it did not exist, the object claiming it would still be claiming it. And objects that do not claim it may in fact have it.
While the top comment may have used poor examples, it feels remarkably uncharitable to actually suggest "what is consciousness" is an equivalent discussion to "how long should a meter be?"
If you define consciousness as "being human", you would just have someone asking a new question - what is "fooblefobble?" Where "fooblefobble" is what we mean when we talk about consciousness today. The question doesn't get answered by being arbitrary in this context, you just necessitate a new word.
With consciousness, on the other hand, there doesn't seem to be any motorcycle-equivalent. Essentially everyone I've discussed the issue with (myself included) expects that any mind which runs on similar principles to ours or has similar thoughts to our thoughts is conscious.
Is a Reliant Robin a car or a tricycle? If it's a car, why aren't other tricycles? What about a side-car of a motorcycle? What about an APC? What's the distinction between a flying car and a plane?
I don't have an ability to exhaustively test all words against this assertion. Nor do I have the kind of access memory to draw one if it exists. Sorry.
Many of them appear very much like fundamental parts of reality, making appearance an untrustworthy instrument. Reversing cause and effect between reference and referent is something almost everyone does, no one notices, and is the source of endless confusion. We should strive to not confuse our model of the world with the world itself. Consciousness exists in our model of the world as much as red does.
This is rhetorically slippery, and feels like it is restating the thing that I asked to be demonstrated when I asked for example of the opposite. It feels like begging the question.
In either case, the central thing that I was saying is that critiquing an article because it makes a claim about a specific word which also applies to an entire class of words makes that critique feel less informative. What I mean is that if there were an article that said "The Sun is not red" and the response was that redness is a concept of human minds, then I don't know if I would feel informed. If the comment is just limited to point that out, I guess I wanted to point out the limitation.
We don't know what we don't know. For all we know, there is a missing field in the standard model of physics that might get revealed if we are somehow able to smash two working brains into to each other at relativistic velocities, and record the results through the extreme explosion 1.532 x10^18 Joules or about 7 Tsar bombs /s
> if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot
Consciousness is independent of "assigning responsibility". Dogs cannot take responsibility for their actions but I believe they are conscious.
> we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious despite being disembodied and are happily conversing in a language that neither actually spoke. In reality, they are just characters in a piece of speculative fiction
This is a straw man. The obvious pro-consciousness claim would be that the LLM is the author of the fictional characters, and that the relationship between the LLM and Julius Caesar is analogous to the relationship between a human author and their fictional creations.
> Did changing the names of the characters from historical figures to generic roles cause the LLM to conjure up conscious entities who possess subjective experience?
No, again the LLM writing the text could potentially have a consciousness separate from the characters it authors.
> Some years ago it was briefly popular to play games with your phone’s predictive-text feature; [...] It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn’t feel like you were talking with someone.
Yes, the same substrate is capable of hosting conscious and non-conscious forms, just like some arrangements of neurons are conscious, and some are not.
> But if the Caesar character were to become dispirited by something that the Khan character said, we shouldn’t become concerned in the slightest.
Even when there are characters, there may be actors behind the characters, for whom we could say "there is something it is to be like".
> we don’t need to worry if the transcript includes sentences where the chatbot character is sad. (We might need to worry if those sentences provoke sadness in the human user, but that’s a separate issue.)
It's actually not a separate issue. The LLM and the human are both adding sentences to the transcript. From the transcript we can make inferences about the mental state of the human. If the LLM has mental states, we could make inferences about those too.
> And note that it’s entirely possible for you to write five pages of dialogue between Caesar and Khan and then have an LLM extend the conversation; neither character had subjective experience when you were writing them, and that doesn’t change when you hand the task off to an LLM.
It's almost like he wants to make my point for me with this sentence.
> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious
This smug shit really makes me angry for some reason. "Openness", i.e. uncertainty in the face of a completely novel situation, in the face of eons long struggle of humanity to understand what consciousness is and how it works, is just being naive.
> Should you consider the possibility that every time you open a Word document you are bringing multiple conscious interlocutors into existence, and every time you close one you snuff their existence out?
No, but if I find a word document I very well might try to use the signs it contains to make inferences about the mental state of its author.
> we are accustomed to reading intention into sentences, whereas we are not accustomed to reading intention into the way that amino acids fold into protein molecules.
He's trying to have it both ways here. Both that "obviously protein folding models aren't conscious because they don't emit sentences", but also "you are a rube for being tricked into thinking LLM models are conscious, because they do emit sentences".
> Obviously I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative will need a truly enormous amount of supporting evidence for it to deserve serious consideration
OK, that's fine for the author to not to be convinced, but that's not what's happening here, instead the author wrote a whole argument being convinced of the opposite viewpoint.
> It’s not plausible to me that a development path where the first step is a sentence-continuation machine that emits bad Julius Caesar dialogue and the next step is a sentence-continuation machine that emits decent Julius Caesar dialogue is one with a conscious Julius Caesar—or consciousness of any sort—as its endpoint
Actually a lot of things have happened? There were clearly many steps along they way from your phone's autocomplete to where we are now.
"Contrary to idealism, which regards the world as the embodiment of an "absolute idea," a "universal spirit," "consciousness," Marx's philosophical materialism holds that the world is by its very nature material, that the multifold phenomena of the world constitute different forms of matter in motion, that interconnection and interdependence of phenomena as established by the dialectical method, are a law of the development of moving matter, and that the world develops in accordance with the laws of movement of matter and stands in no need of a "universal spirit."
Now go and implement other teachings of Marx, you dialectical venture capitalists!
that's exactly the state of "AI" right now, it's cold, mathematical emulation
btw there are some fascinating papers on the concept that consciousness in humans is actually a quantum effect
brilliant Roger Penrose proposed it (and they thought he was nuts) but recent discoveries about microtubules make it plausible
so who knows, maybe a dozen exponential improvements in quantum computers could make "AI" really conscious next century
* https://www.youtube.com/watch?v=xa2Kpkksf3k
Alright. What is consciousness? Please provide a definition that somehow encompasses all humans and excludes all current AI.
I’ll wait.
An example of fulminating would be: How dare you accuse him of fulminating? That is the most ass backwards moderating I have ever seen! Do you even know the definition of the word? Who paid you to call him out?
His comment may be sarcastic but it is not "fulminating".
It's just a word machine. There are no thoughts. It cannot be conscious. How is this even up for debate in any way whatsoever? I do not understand how people can believe this. Is this not a site for software engineers?