Is it wrong to create an AI agent that suffers from depression?

personal identity, AI agents, and the non-identity problem

We each have our own creation stories, and our identities depend upon them. Your story began when some particular sperm joined some particular egg. These genetic materials developed into you. But, had a different sperm and egg come together, they would have developed into someone else. This is to say that our identities are hypersensitive to the genetic materials from which we came.

Interestingly, our identities don’t seem to depend on every aspect of our creation story. I was born in Bryan, TX, but I could have been born in Boston, MA. So, my identity is apparently not hypersensitive to the geographic location of my birth. That part of my creation story can be changed without changing me. It is an open question which parts of our creation stories are relevant to our identities.

AIs have their creation stories, too. But they’re much different than ours. They didn’t come from zygotes. And they don’t have genes. They’re created in labs, by computer scientists, and they run on computer hardware, not squishy brains.

This raises a deep question. If our identities depend on our genes, then what do the identities of AIs depend on? How do the creation stories of AI figure into their identities? Imagine a computer scientist programming an AI. They may alter various parameters to control the AI’s dispositions. For example, they could have an AI express a “happy” or “sad” disposition. That might be one part of an AI’s creation story. Does this selection change the identity of the AI that will result? That’s the question of AI identity.

The Non-Identity Problem

Suppose you’re deciding between creating the “happy” or “sad” AI. How bad is it to choose the sad AI? That depends on what you think about the non-identity problem. Derek Parfit illustrated the problem with a now classic pair of cases:

Ruth, who is pregnant, knows that, unless she takes some painless treatment, the child she is carrying would have some disease that would kill this child at the age of forty. If Ruth takes this treatment, this child would live to eighty.

Clare knows that, if she conceives some child now, this child would have this same disease, and would live to only forty. If Clare waits for two months, she would later conceive a child who would not have this disease, and would live to eighty.

In the case of Ruth, there is one possible child that could exist. Her choice only affects whether that child will have a disease or not. It is widely agreed that Ruth should take the painless treatment to prevent her child from having the disease. By doing so, Ruth makes her child better off.

But the case of Clare is different. Clare has a choice between creating one of two different possible children, where we assume that both would lead worthwhile lives, but that the child without the disease would lead a better life. As a result of this difference, Clare would not make her child worse off even if she conceives the child with the disease.

Derek Parfit famously believed that this difference does not matter—he held the “no-difference” view. However, pace Parfit, it is now widely believed that Clare has weaker reasons to delay conception than Ruth has to take the treatment (e.g., McMahan 2020). Some philosophers even defend the view that Clare has no requiring reason at all to create the happier possible child (Boonin, 2014). So, the fact that Clare’s choice affects who will exist is practically important. It means she has a lot more leeway to choose either of her two options.

Over the next decades, innumerous AIs will be created. Though I don’t think current AIs are the subjects of well-being, future AIs might be. If the identities of AIs prove hypersensitive to their creation stories, then computer scientists, in creating them, will be like Clare—deciding whom to create. In this case, computer scientists have the moral leeway to create AIs with wide-ranging (but positive) well-being levels, including those with severely flawed existences. But if the identities of AIs are hyposensitive to their creation stories, then when computer scientists create unhappy AIs, they do serious harm, comparable to Ruth’s refusal of treatment.

We may not be far from a world where AIs capable of genuine emotion are regularly created. Heck — maybe even you are an AI. I hope you’re happy. But at any rate, when this day comes — assuming it has not already passed — I hope the creators will have thoughtfully considered these questions.