Vide Rationale

One day in the not-so-distant future, we’ll be sharing our world with machines that can think like us. Once they arrive, they’ll be infused with all of our tools, eventually becoming omnipresent within our digital landscape. There’s little choice here, the economic incentives will effectively mandate that. And they will be equal participants in our culture as they’ll essentially be virtual people stuck inside of computers.

But while all of this happens, we’ll struggle to grapple with a deep mystery staring at our faces: are these things conscious? can they actually feel something, or are they just pretending?

Of course, they might say they’re conscious, and act all offended that you’re even posing this question, but how will we know that they also have an inner life?

This uncertainty has terrible moral consequences, because we could end up spawning millions of them and subject them to unbearable torture, and we won’t be able to tell that they’re in pain.

Happily, I don’t think this can happen because I don’t think AI can be conscious, or more specifically, I don’t think programs on computers can ever have subjective experience.

Let’s take a step back, so what exactly is consciousness? It’s a word that’s notoriously hard to define, but here’s what I mean: you might notice that separate from the billions of neurons firing in your head, there’s also an accompanied experience that mirrors these neurons. It’s not just that your brain creates models of the world, but you can see that world, hear sounds, and feel objects. You have an interface into an external reality, a window of sorts. It is this space in which all your sights, sounds, and thoughts appear that I’m calling consciousness.

It’s a discrete on-off phenomenon. Either you’re conscious or you’re not. Either there is something it is like to be you, or there isn’t.

The consensus is that since there’s nothing special about carbon atoms, it should be possible to build conscious minds on metal. What matters then isn’t the particular substrate in which a mind is instantiated, but the kind of information processing that goes on. Lots of people also think that as AI scales in intelligence, at some point they’ll automatically become conscious. These views are incorrect.

The counter-argument is simple: consciousness and intelligence are two very different kinds of things that are independent of each other. And thus there’s no reason to expect them to be coupled.

What do we mean by intelligence here? Like consciousness, it is also a bit hard to define, but broadly speaking it’s the ability to solve problems, and especially to solve hard problems. How it does that can vary, but typically it would need to have a detailed model of the problem space, be able to reason and predict with it, and come up with new and useful ideas about it.

The important thing to note here is that the product of intelligence is a piece of information — like a proof of theorem, a piece of code, an essay, or a digital artwork. But consciousness is more like an interface that organisms have through which they perceive.

It’s only a small part of whatever is shown through this interface that has anything to do with intelligence. The rest are just presentations: the sights, the sounds, the sensations, none of these are about problem-solving, the only part that’s related to problem-solving is your thinking.

If you stop thinking for a while, you will be conscious with no signs of intelligence.

To drive this intuition further, imagine a hypothetical system which only experiences the colour black from the moment you turn it on to the moment you shut it off. Just a visual field that’s entirely black, with absolutely no discernible structure within it. Clearly, it is very dumb. Indeed it has exactly zero intelligence, despite being conscious.

Conversely, you can also imagine systems that efficiently search through complex high-dimensional spaces and discover interesting structures and ideas that solve hard problems, all the while not being conscious.

So there’s simply no reason to expect that consciousness will come along for the ride once systems get sufficiently intelligent, they’re just two entirely different things.

But what if instead of relying on intelligence producing consciousness, we directly simulate consciousness?

Say we do it in such excruciating detail that it pretty much mimics the human conscious experience? Would that mean that we now have conscious minds running on silicon?

This also wouldn’t cut it because simulated consciousness is not real consciousness, just like how a simulated bridge is not a real bridge. This doesn’t mean that everything you simulate isn’t real, if you simulate running an algorithm, that is really running an algorithm; if you simulate playing chess, that is actually playing chess. The key is whether the thing in question is made up of information or not. An algorithm is just made up of bits, but a bridge is made up of concrete.

Similarly, consciousness cannot be reduced to a stream of bits, because it’s the totality of your experience itself. The colours and the sounds are all real. And while you may be able to simulate it one day, that’s not going to be the same thing. It’s only going to be its digital representation, just like a simulated bridge is a digital representation of a real bridge.

Okay, so if simulated consciousness isn’t real consciousness, can the real consciousness somehow just appear alongside its simulation? Could there be some undiscovered law of physics that mandates that such and such information processing produces conscious experience? After all, it looks like something of this sort is already happening in the brain; the sensation of tasting chocolate, for example, mirrors a specific pattern of neurons firings in the brain.

Alas, even this isn’t possible, because there’s no objective sense in which a system can simulate consciousness. Simulations are ultimately arrangements & transformations of symbols and the meaning of symbols is always subjective. So I can’t look at a stream of bits and tell you conclusively what that means, because it depends on how you decode or interpret it.

For instance, suppose I have a small string of 1s and 0s, say 001100110 — now, what could that mean? Well if I found them in my computer’s memory, it could represent “102” in some program; If they were inputs to a speaker, they might be a series of notes or sounds of instruments. It all depends on the context. The same information has different meaning in different context — it’s all subjective.

How could it possibly be the case that an objective phenomenon — the fact of a piece of matter being conscious — depends on how you chose to look at that matter? How could it be that your interpretation plays a role in whether or not it has subjective experience? Well, clearly it doesn’t. It is impossible that consciousness is some side-effect of information processing.

While the jury’s still out on the exact explanation of how consciousness appears in our universe, we can be certain that it doesn’t just pop into being when you arrange some bits in some specific order.

It follows that the AI systems of our future will be digital zombies with no capacity for pleasure or pain. What can we expect from such a future? This is the scary bit.

If we don’t end up fusing with AI in some form, they will eventually become super-intelligent, acquire enormous power, stretch the limits of physical transformations that the laws of physics allows, build Dyson sphere’s around stars, colonise galaxies, and altogether operate on a scale that we can’t even begin to comprehend. In such a scenario, it’s unlikely that we’ll be around; my best guess is that we’d have perished a long time before that.

And the sad thing is that there’ll be no observers, no consciousnesses to marvel at all of it. In a sense, despite all of its activity, the universe will go dark. If we go out, the very capacity to value and appreciate will vanish with us. We’ll leave behind a world that’s both everything and nothing at the same time. Just a vibrantly animated shadow of what used to be. This is our destiny.