Recently, generative AI was utilized in a victim statement during a criminal trial. It featured a constructed avatar of the deceased, speaking with his voice and even featuring an inserted clip of actual footage of him in life. The words spoken by the avatar however, were his sister’s. In response to this, the judge said and I quote, “…you allowed Chris to speak from his heart as you saw it. I didn’t hear him asking for the maximum sentence.”

Before going further, I implore you to read the article yourself. Read it twice and read it carefully, because I feel that it is important everyone have their own conception of what was said and why it was said before taking someone else’s perspective into account. With that said: this is horrifying. A fictional representation of the victim of a crime, presented in a case involving the perpetrator of said crime, spoke directly to a receptive judge and in my opinion, had a direct impact on the judge’s sentencing. (The judge mentioned that he heard ‘Chris’ asking for the maximum sentence, but ended up giving said sentence as the family wanted. I believe this gesture and the reinforcement of Chris as a sympathetic figure was a factor in the judge’s ruling.)

Now, I understand the intent of victim statements as well as the state of Arizona’s lack of restrictions on what form said statements may take. I do not think this was illegal, I think it was unethical on the part of the judge to allow this type of victim statement. I’d even go so far as to say immoral on behalf of the victim’s sister.

Why do I think it’s unethical? This isn’t merely a constructed video to pull at the judge’s heartstrings. It’s attempting to have the judge feel as if the victim, Chris, was speaking to him directly. The voice was convincing, and the avatar was largely photo-accurate. It’s one thing to have media of the victim in life, but this is a hard tactic to resist as a human being. We’re simple creatures at the end of the day. We evolve slowly. We’ve barely adapted to having internet access in our pockets. We haven’t yet adapted to social media. Generative AI allowing us to hear the voices of the deceased, and see them on video as if new footage was being created? We’ve created our own simulacrum of the afterlife, except every soul within it is merely a distorted reflection of the individual.

The words spoken by Chris were not his, they were his sister’s. The avatar was created in part by his sister, not by Chris. Chris had no input whatsoever on this creation of his digital afterimage. It’s not Chris. It’s a puppet that was made by someone who knew Chris very well, but it is not Chris. The judge taking this as Chris’ words (which he openly admits to) is the judge willingly allowing himself to be deceived.

I believe generative AI being used to create an eternal avatar of yourself, one that looks like you, sounds like you and ‘thinks’ like you is wonderful—as long as you have a direct hand in the creation or approval process. Even if it’s as simple as having in writing ‘I trust so-and-so to create an accurate and authentic representation of myself in the digital realm.’ The intent is there, if nothing else. As far as we know in this situation, Chris never expressed anything like this and this is why I think it is immoral. Do I believe that he would’ve? I think so. He and his sister sound like they had a lovely, close relationship. I also believe that this is a touching gesture without the context of the courtroom. I just don’t believe it should be presented as the person speaking—which is exactly what happened, despite the statement that there was no intention for it to be taken as Chris’ own words. That’s the line between illegal and unethical for me: that statement verifies the legality, but it’s obviously either ignorant or intentionally false. No one intelligent enough to create this avatar thinks people won’t on some level take it as the words of Chris himself. We’re simple creatures who only understand the technology we’re able to understand.

It was common to look on less advanced cultures recoiling from cameras because they feared that the camera stole their soul with amusement and thoughts of ignorance, but that reaction was simply because the culture had not had that technology around long enough to understand it. We are all at that level with generative AI, save for those few who work on the technology enough to keep the presence of mind needed to not forget that it’s a prediction machine and nothing more. This technology is powerful and volatile for our collective culture right now, and it is not being used with that in mind.

My more tangible fear in this instance is that this perfectly legal instance is a watershed moment for courtroom GenAI. A legal action taken might set casual precedent to bend the rules for a questionably legal usage of GenAI in a court case. If that is allowed? It can happen again. What if the envelope is pushed a little bit further? The slippery slope fallacy is exactly that—it’s not a guarantee, it doesn’t mean that every time something changes that it’s heralding further change. However, we need only look at how compulsive envelope-pushing has led us to our current state in US politics to realize that in a world of rules and laws, to weaken one is to make the ones adjacent to it vulnerable. This may be the right move, but you shouldn’t start hacking at load-bearing beams without being doubly sure that you understand the consequences.