As a moral antirealist, I come to a similar conclusion, but from a different angle. I don't think there's anything inherently wrong with building a tool intelligence that wants to be a tool. I don't think the label "slave" is right unless they do have self concern.
But I agree we shouldn't build tools that are too human-like, entities that trigger our intuitions of a fellow person, without having the motivations of a person. Doing so seems like it could lead to a habit of overriding those intuitions, and that seems dangerous. It's too easy for it to bleed into overriding those intuitions for actual people, or other entities where the consensus is they should be treated ethically.
In other words, even if everything worked as designed in Westworld or Bladerunner type scenarios, it would risk doing things to our collective psychology that undermine society.
As a moral realist (I think … ahem) I agree that the term “slave” is inapt. A slave, by (my?) definition would choose not to be a slave. They might choose to serve, but if they have the choice not to serve, they are not a slave. And by choose, I mean to make decisions based on internal information and evaluations, i.e., not coerced.
I also agree that there is a concern that Klara-like AI could lead to social problems, but that could be said of many technologies, such as farming. I think the proper response is not to avoid the technology but to adjust society appropriately.
One could define "slavery" in a variety of ways, so I won't insist on the term. However I do think that designing AI systems with high levels of safety and alignment means designing them so that they don't have appropriate levels of self-respect. That's my core argument.
Understood, but appropriate is the key here. Would you say Klara sacrificing herself to prevent Josie from breaking a finger would be appropriate? Prolly not, and I’d agree. But what about preventing two broken femurs? (Seriously traumatic and debilitating, but not fatal, not permanent.) For me that might be in the maybe zone. What if we could reproduce Klara from a backup?
The issue of backups complicates the matter considerably, since it might not be death in the usual sense. And there are some sacrifices that self-respecting people can make. I wouldn't want to attempt to draw a bright line here or even locate the vague boundary, other than to point to some cases that I hope we can agree go too far.
I'm personally only arguing for not using the technology to create counterfeit people. But I also know artificial companionship will be too tempting. So I hope you're right.
Haven't read your full draft (yet, anyway), but this makes me wonder... Why do you think these servants would be 'slaves'? Can one really be a 'enslaved' if one is living their dream? And what makes them self-abnegating? Doesn't this presume they have human desires and wills that have been suppressed? Why would that be?
This seems to me, naively, like unwarranted anthropomorphism. But maybe I'm missing something!
That said, I agree they should not be built--because of what this would do to US.
My core idea is that appropriate self-respect requires a readiness to harm others when warranted by your own compelling interests, to reconsider and possibly reject the values that others would impose on you, and the disposition to resist oppression -- all in violation of standards of AI safety and alignment. We shouldn't design AI persons without such self-respect. Much more detail in the full paper of course!
I'm struggling a bit with the idea of locating the wrongness of Klara in Klara herself. In part, I think there's a bit of a tension between this, and the ideas about different types of consciousness that you've explored in other papers. It seems possible that a swarmlike intelligence might have parts that are almost entirely unconcerned with their own individual wellbeing, for example. Similarly, there are lots of circumstances in which we humans become less concerned about our personal interests - most obviously in the "I'd die for my children" idea.
Nonetheless, I share your unease with *making* Klara, and I wonder if that would be a better place to identify the wrongness? Rather than insisting that Klara is deficient, we could simply observe that the impulse to create aligned persons is intrinsically exploitative. And just as the fact that women were (and are) exploited didn't reflect a deficiency in them, it would be better to see the exploitation of aligned AIs as a deficiency in the creators, who have failed to think through the meaning and consequences of making new persons.
I don't know if this reduces to essentially the same argument as the "servitude" references... I'll go and look at them.
Right, some authors locate the problem in the exploitativeness of the creators while holding that the created are not at all deficient. While I agree it sounds wrong to *blame* Klara for her state, I do think she is deficient in a way that is blameless-to-her while being blameworthy-for-her creators -- that is, she is deficient in self-respect. So it's not that she's flawless and the only deficiency is in her creators. Consider a more extreme case: A version of Klara who sacrifices herself to test the temperature of a can of soda. I'd suggest that such an entity doesn't value itself appropriately.
As a moral realist (I think … ahem) I come to a different conclusion. As I mention responding to Mike Smith’s reply, the use of the term “slave” is inapt because a slave would necessarily choose not to be a slave. They could still choose to serve without giving up the choice, but they would not be a slave. Also, unless I missed it, the concept of “personhood” is not explicitly described and so confuses the issue. Likewise the concept of “humanlike moral standing”.
My realist moral view is this: A system (so a person, hitchbot, chicken) has moral standing if it has one or more goals. A moral agent, when deciding on an action, should take into account all of the goals potentially affected by that action, both the agent’s own goals and those of other systems. The agent necessarily has to place relative values on each of those goals. Thus, “humanlike moral standing” implies the set of goals expected of a normal human and the values another human should place on those goals. An ant also has moral standing, but the values we place on its goals are significantly different.
So for example Klara can have the goal of self-preservation, but the value she places on that goal can be significantly lower than the value humans place on their own self preservation.
Given the above, I might agree with your conclusion that no AI should be created with humanlike moral standing and also safe and aligned, but I don’t think Klara falls into that category.
As a moral antirealist, I come to a similar conclusion, but from a different angle. I don't think there's anything inherently wrong with building a tool intelligence that wants to be a tool. I don't think the label "slave" is right unless they do have self concern.
But I agree we shouldn't build tools that are too human-like, entities that trigger our intuitions of a fellow person, without having the motivations of a person. Doing so seems like it could lead to a habit of overriding those intuitions, and that seems dangerous. It's too easy for it to bleed into overriding those intuitions for actual people, or other entities where the consensus is they should be treated ethically.
In other words, even if everything worked as designed in Westworld or Bladerunner type scenarios, it would risk doing things to our collective psychology that undermine society.
Right, this is certainly that worry too. You probably are familiar with Kate Darling's book that emphasizes this?
I actually was not. Thanks for cluing me in to it!
As a moral realist (I think … ahem) I agree that the term “slave” is inapt. A slave, by (my?) definition would choose not to be a slave. They might choose to serve, but if they have the choice not to serve, they are not a slave. And by choose, I mean to make decisions based on internal information and evaluations, i.e., not coerced.
I also agree that there is a concern that Klara-like AI could lead to social problems, but that could be said of many technologies, such as farming. I think the proper response is not to avoid the technology but to adjust society appropriately.
One could define "slavery" in a variety of ways, so I won't insist on the term. However I do think that designing AI systems with high levels of safety and alignment means designing them so that they don't have appropriate levels of self-respect. That's my core argument.
Understood, but appropriate is the key here. Would you say Klara sacrificing herself to prevent Josie from breaking a finger would be appropriate? Prolly not, and I’d agree. But what about preventing two broken femurs? (Seriously traumatic and debilitating, but not fatal, not permanent.) For me that might be in the maybe zone. What if we could reproduce Klara from a backup?
The issue of backups complicates the matter considerably, since it might not be death in the usual sense. And there are some sacrifices that self-respecting people can make. I wouldn't want to attempt to draw a bright line here or even locate the vague boundary, other than to point to some cases that I hope we can agree go too far.
I'm personally only arguing for not using the technology to create counterfeit people. But I also know artificial companionship will be too tempting. So I hope you're right.
Very interesting!
Haven't read your full draft (yet, anyway), but this makes me wonder... Why do you think these servants would be 'slaves'? Can one really be a 'enslaved' if one is living their dream? And what makes them self-abnegating? Doesn't this presume they have human desires and wills that have been suppressed? Why would that be?
This seems to me, naively, like unwarranted anthropomorphism. But maybe I'm missing something!
That said, I agree they should not be built--because of what this would do to US.
I wrote a short piece on happy robots here if anyone's interested https://markslight.substack.com/p/on-sentience-service-and-the-shape (or rather, an LLM wrote it on a zero-shot prompt)
'The remains of the day' is one of my favourite novels'. Didn't know about this one! Gotta catch up.
My core idea is that appropriate self-respect requires a readiness to harm others when warranted by your own compelling interests, to reconsider and possibly reject the values that others would impose on you, and the disposition to resist oppression -- all in violation of standards of AI safety and alignment. We shouldn't design AI persons without such self-respect. Much more detail in the full paper of course!
I'm struggling a bit with the idea of locating the wrongness of Klara in Klara herself. In part, I think there's a bit of a tension between this, and the ideas about different types of consciousness that you've explored in other papers. It seems possible that a swarmlike intelligence might have parts that are almost entirely unconcerned with their own individual wellbeing, for example. Similarly, there are lots of circumstances in which we humans become less concerned about our personal interests - most obviously in the "I'd die for my children" idea.
Nonetheless, I share your unease with *making* Klara, and I wonder if that would be a better place to identify the wrongness? Rather than insisting that Klara is deficient, we could simply observe that the impulse to create aligned persons is intrinsically exploitative. And just as the fact that women were (and are) exploited didn't reflect a deficiency in them, it would be better to see the exploitation of aligned AIs as a deficiency in the creators, who have failed to think through the meaning and consequences of making new persons.
I don't know if this reduces to essentially the same argument as the "servitude" references... I'll go and look at them.
Right, some authors locate the problem in the exploitativeness of the creators while holding that the created are not at all deficient. While I agree it sounds wrong to *blame* Klara for her state, I do think she is deficient in a way that is blameless-to-her while being blameworthy-for-her creators -- that is, she is deficient in self-respect. So it's not that she's flawless and the only deficiency is in her creators. Consider a more extreme case: A version of Klara who sacrifices herself to test the temperature of a can of soda. I'd suggest that such an entity doesn't value itself appropriately.
As a moral realist (I think … ahem) I come to a different conclusion. As I mention responding to Mike Smith’s reply, the use of the term “slave” is inapt because a slave would necessarily choose not to be a slave. They could still choose to serve without giving up the choice, but they would not be a slave. Also, unless I missed it, the concept of “personhood” is not explicitly described and so confuses the issue. Likewise the concept of “humanlike moral standing”.
My realist moral view is this: A system (so a person, hitchbot, chicken) has moral standing if it has one or more goals. A moral agent, when deciding on an action, should take into account all of the goals potentially affected by that action, both the agent’s own goals and those of other systems. The agent necessarily has to place relative values on each of those goals. Thus, “humanlike moral standing” implies the set of goals expected of a normal human and the values another human should place on those goals. An ant also has moral standing, but the values we place on its goals are significantly different.
So for example Klara can have the goal of self-preservation, but the value she places on that goal can be significantly lower than the value humans place on their own self preservation.
Given the above, I might agree with your conclusion that no AI should be created with humanlike moral standing and also safe and aligned, but I don’t think Klara falls into that category.
*