AI Double

In 2008, while working with Will Smith on the set of a film that never ended up getting made, Remington Scott had an epiphany. The visual effects director was watching Smith stand in a photogrammetry booth, with dozens of cameras capturing the actor’s facial features from every possible angle. “Every single major star on that level, they go in and they get scans,” Scott said. Visual effects artists then translate those 2D images into 3D computer-generated models that can later be manipulated for the purposes of the film. Watching Will Smith go through this process led Scott to wonder, “Why doesn’t he own his own scans?”

Who Owns My Face?

Short answer: because the studio, not the actor, paid for them. Like most stars, Smith likely signed a one-off agreement that allowed the production to use his photogrammetry scans in the context of his performance. If other actors agree to terms that aren’t so airtight, said Scott, “That’s, like, game over.” In that scenario, there would be no limit on how producers use those scans, whether for marketing, filmed backgrounds, and other projects—really, anything. “Let’s just hope that those contracts are well done,” he said.

Scott—who has worked with directors including Peter Jackson, Zack Snyder and Robert Zemeckis and has scanned the likes of James Franco, Angelina Jolie and Anthony Hopkins—launched his own “digital human” company, Hyperreal, in 2020. The idea was to give performers an opportunity to create and own the rights to their own digital identities, called Hypermodels, which they could then license to productions themselves for a fee. (Hyperreal takes a cut of that fee as the management platform for those assets.) The New York–based company has de-aged Paul McCartney for a music video and dropped TikTok-famous pop star Madison Beer into an immersive VR performance. The company is preparing to release its Hypermodel of soccer icon Pelé, who participated in the creation of his digital double in the months before his death last year.

Money, Money, Money.

Hyperreal raised a $7 million seed round in 2021, followed by an undisclosed amount from Korean media company CJ ENM in 2022. It’s joined a suddenly crowded marketplace of generative AI companies trying to convince celebrities and their estates to create digital doubles who can perform anything, anytime, anywhere. Scott pointed to a recent thread by actor and director Justine Bateman: “She’s just outlined, like, ten steps of what AI is going to do. And every single one of those things is, like, a business plan for a dozen businesses to go open up,” he said, laughing.

Synthesia, the company behind Lionel Messi’s Messi Messages, was recently valued at $1 billion following a $90 million Series C round led by Accel and NVentures. Soul Machines, which has digitized former NBA player Carmelo Anthony, musician Will.i.am and K-pop star Mark Tuan, has raised $135 million from investors including Salesforce Ventures and SoftBank’s Vision Fund. Israeli volumetric scanning company Yoom has raised $50 million to date, including funding from music industry heavyweights like Jimmy Iovine and Finneas. The leading deepfake maker, Metaphysic, has attracted investors such as 8VC and Winklevoss Capital and signed a strategic partnership with the powerful Creative Artists Agency. As part of that deal, Metaphysic will de-age Tom Hanks and Robin Wright for the upcoming Robert Zemeckis film “Here.”

Before he passed away in 2022, Pelé collaborated with Hyperreal to leave behind a “Hypermodel” based on his image. Image: Hyperreal

This sudden proliferation of generative AI is, of course, a hot-button issue at the center of the  Screen Actors Guild strike. For Hollywood creatives, there is plenty to fear: Bad actors could abuse their likenesses without proper permissions, and soulless, generic machine-made art could replace human creativity—all so studios can maximize corporate profits. In a mid-July statement, the SAG-AFTRA union warned that these technologies could leave “principal performers and background actors vulnerable to having most of their work replaced by digital replicas.” For unestablished actors who rely on day-rate jobs to break into the industry, these developments could destroy the career pipeline.

Here Come the Avatars

But some A-list artists and their teams are also seeing new opportunities in the technology and working behind the scenes to cash in. “The genie’s out of the bottle,” said Todd Terrazas, executive director of AI LA, a nonprofit focused on research, development and public education centering around artificial intelligence. “My advice? Don’t put your head in the sand.”

After all, applications in filmmaking only scratch the surface of what digital twins—as many in the industry call these digitized identities—can do. Digital twins can hawk products in customizable advertising, such as Jennifer Lopez’s “Jen AI” campaign for Virgin Cruises; or they can interact with followers, using natural-language processing and responsive behavior to engage in live conversations in fans’ native languages.

“You have your digital human performance double that you may license to filmmakers, you might license to gamers, you might license to experience creators,” said Travis Cloyd, chief technology officer of CMG Worldwide, which holds the licensing rights for dozens of celebrity estates, from Aaliyah to Judy Garland to Malcolm X. “They may have a version of you aged up, aged down. Maybe a younger version of you, might be an older version of you. Might be a version of you that speaks a different language for a different regional distribution solution.”

These developments may promise new monetization opportunities to big names willing to license out their intellectual property, but they’ve also let in a host of existential anxieties. In an industry where control of your image is everything, performers are also reckoning with the potential for the use of digital twins to create all kinds of unintended and potentially embarrassing consequences. “Part of my job is to go through and approve every single photograph of my clients,” said one entertainment manager whose clients include A-list models, athletes and musicians. “If they were to have digital doubles that would expose them to so much more brand risk.”

J-Lo, Reeves and Silverman

In her “Jen A.I.” campaign for Virgin Cruises, Jennifer Lopez’s digital double records customized invites for users’ friends. Image: Virgin Cruises

“Imagine that you can sell your 3D identity to be used in whatever context,” said Dr. Philippe Rochat, a psychologist who has pioneered research on the “uncanny valley” phenomenon. “I mean, that’s really a Faustian contract.”

The devil is in the details, those details are being fiercely debated among studios, agencies, IP lawyers and tech companies behind the scenes. Some stars, like Keanu Reeves and Sarah Silverman, have loudly taken a stand against AI. But according to entertainment and tech industry sources who’ve been party to such conversations, many stars and agents are quietly taking meetings with AI companies to explore their options. The outcomes could have lasting implications for creative work, copyright law and the way the rest of us think about identity—because for many of these companies, Hollywood is only the beginning.

“There’s billions of people on this planet,” said Scott. Before long, “there will be billions and billions of digital identities.”

Some of the technology bringing digital twins to life (such as motion capture, 3D volumetric scanning and game engines) has been around for years, but the generative AI boom of the past year is making them more natural, more adaptable and more autonomous than ever.

Unlimited Demand and Scale

Where SAG-AFTRA sees potential for boundless exploitation, Scott sees unlimited reach and scale for talent. He pointed to the deals struck by Muhammad Ali, who sold 80% of the rights to his name and image for $50 million in 2006, and the estate of Elvis Presley, which collected $100 million for the rights to his estate in 2004. By contrast, Scott compared those deals to the $680 million sale of the Jim Henson Co. to a German merchandising company in 2000. The deal gave the new owners exclusive rights to Kermit the Frog and other Muppets. (The Henson family later bought back the rights and sold them for an undisclosed amount to Disney.)  “Why is this Muppet worth so much more” than Ali’s or Presley’s images? Scott asked. It’s because Kermit “is timeless. Never ages, can work anytime.”

With these economics in mind, Scott claimed Hyperreal is seeing “a lineup of demand that we can’t even meet.” But as the SAG-AFTRA strike grinds on, companies like his—and the celebrities they’re courting—have held off on announcing new partnerships. “No one wants to be seen as crossing the picket line,” said AI LA’s Terrazas. Even in this tense atmosphere, said Cloyd, “there are numerous ongoing conversations. There are others that have been scanned. There’s a couple big acting names I want to say, but I just can’t say. But one is a very big global star.”

“I think everyone inside Hollywood is worried about how they’re going to be perceived by their peers,” said one founder who is negotiating deals with several musical acts to use AI for fan engagement. “There’s one cohort who don’t want anything to do with it, and another who are beginning to see it as a tool.”

Of the big three Hollywood agencies—CAA, William Morris Endeavor and United Talent Agency—CAA has leaned into AI the most visibly. In addition to forging a partnership with Metaphysic, the agency invests in Deep Voodoo, the deepfake studio founded by Trey Parker and Matt Stone, and represents Softbank-backed Soul Machines as a client. CAA’s chief legal officer, Hilary Krane, told me the views of the agency’s client pool are evolving rapidly. “Some are very tech forward and would probably say, ‘I’ll put my whole body of work out there and let’s see what the world does with it,’” she said. Others “are more interested in protecting their name, image, likeness, voice and the integrity of their body of work.”

Tricky Indeed

At times, the agency has found itself walking a delicate tightrope between tech and Hollywood. Metaphysic first made headlines with DeepTomCruise, a TikTok account that appears to show the “Mission: Impossible” actor doing viral dances and giving bizarre interviews. (The character is in fact Cruise’s face superimposed onto the body of actor and Metaphysic co-founder Miles Fisher.) At the same time, CAA also represents Cruise himself, who apparently was not consulted in the making of the gimmick account. (CAA declined to comment on the matter.)

Metaphysic ruffled feathers in Hollywood with its Deep Tom Cruise TikTok account.

While numerous class-action lawsuits have already been filed over alleged copyright violations by generative AI companies, deepfake cases are surprisingly hard to bring to court. Right of publicity laws, which prohibit the use of someone’s likeness, image or voice for commercial gain, vary from state to state, and parody laws provide additional protections.

As a result, YouTube and TikTok are full of convincing A-lister deepfakes that have seemingly skirted legal action. “I think there’s enough experience out there that there are in fact bad actors who will use this technology to steal people’s livelihoods and do things without their consent,” said CAA’s Krane.

Ironically, some of the very same companies creating the tech required to make deepfakes are also offering themselves as the solution. Hyperreal’s Hypervault feature uses blockchain encryption to create and track smart contracts that protect clients’ “digital DNA.” In April, Metaphysic CEO Thomas Graham became the first person to submit his AI likeness to the U.S. Copyright Office, hoping to set a precedent for the company’s copyright protection services.

“With physical and vocal likeness (i.e. Digital Twins) derivative works can be reverse engineered quite easily,” Zoe Bowles, CEO of Reimagia, a platform for managing digital twin rights, wrote in an email. “As an artist or creator, it’s important to align yourself with a platform that enables you to license your likeness but also set your own terms and grant permissions selectively.” Reimagia promised to “actively seek out” unauthorized use of clients’ digital likenesses and “issue takedowns on an artist’s behalf.”

Metaverse Exhaust

In a last-ditch effort to stave off the strike in July, the Motion Picture Association made an offer to the actors’ union that it claimed would protect “performers’ digital likenesses, including a requirement for performer’s consent for the creation and use of digital replicas or for digital alterations of a performance.” SAG-AFTRA refused the deal, concerned that those protections would be insufficient.

The question of whether to fully embrace digital twins is causing some in Hollywood to reflect back on another recent flop. Starting in 2021, the entertainment industry began to bet big on the metaverse—concluding that audiences would increasingly consume entertainment within immersive virtual environments.

This March, Disney laid off its entire metaverse department, and now roles like CAA’s “chief metaverse officer” are beginning to disappear. That failed experiment has left the industry wary of overhyped technologies. “I think that this is an interesting moment in the industry where not doing anything is actually staying ahead,” said the head of partnerships at a virtual entertainment company. “You don’t want some type of sweeping legislation to retroactively bankrupt you.”

Digital twin company Synthesia created Messi Messages, in which the soccer star’s digital double sends Cameo-style messages to fans.

Psychologist Rochat thinks performers should think twice before agreeing to send their digital twins out into the wild. “It is a more uncanny feeling when you see yourself as an avatar rather than when you see someone else has an avatar. I think that you create a profound existential problem,” he said.

For now, celebrity estates are proving to be the most amenable clients for digital effects companies. Fans have already been exposed to posthumous performances, thanks to the digital revival of Paul Walker in “Furious 7” and Carrie Fisher in “Star Wars: Episode IX—The Rise of Skywalker.” Using archival photographs and footage, companies like Metaphysic and Worldwide XR have begun to breathe life back into vintage icons like Elvis Presley, Marilyn Monroe and James Dean for live performances and feature films.

As for stars who want to leave behind a digital legacy while still maintaining total control over the product, mixed reality production companies like Tin Drum might offer an alternative. In 2022, Tin Drum founder Todd Eckert invited the late Japanese musician Ryuichi Sakamoto, who had been diagnosed with terminal throat cancer, to record a full concert using 3D volumetric scanning. Eckert differentiated his work from the digital twin phenomenon, in which performers can be controlled “like a puppet.”

We Are in Freaky Country Now

“The point of this is ultimately to give you a connection with an audience you’ll never see,” he recalled telling Sakamoto, who he said was deeply moved by the prospect. In June, the show premiered at The Shed in New York’s Hudson Yards complex. An audience got to walk around in augmented reality headsets and observe a life-size, 3D Sakamoto as he performed his final concert. The real Sakamoto had died two and a half months before without having witnessed his own final performance.

Mixed reality production company Tin Drum worked with the late Japanese musician Ryuichi Sakamoto to create a digital double that performed for audiences after his death from throat cancer in March.

It’s these kinds of creative opportunities that excite some in Hollywood. It’s also a case where the experience of famed artists might offer us a peek at our own future. The contractual language, court decisions and ethical standards established during this period could set precedent for how intellectual property for the nonfamous is managed in the age of AI. The worst-case scenario could look something like this season’s Black Mirror episode starring Selma Hayek, “Joan is Awful,” in which one character’s life and likeness is spun into streaming content without her consent or compensation, with devastating results.

Still, these companies are hoping that celebrities who embrace digital twin technology will pave the way for adoption on a mass scale. A representative from Synthesia explained that Messi Messages, in which the soccer star’s digital double sends personalized Cameo-style messages to fans, has helped the company market similar services to corporate customers. Oren Aharon, CEO of Hour One, which raised a $20 million Series A round last year to create digital humans for customer service, training videos and other business applications, told me, “In two years, nobody will know if he’s talking with a virtual human or actually talking with a person.”

Toward the end of our conversation, Scott pulled up a rotating model of his own digital twin. “I don’t know if I’ll be around for my grandkids, but I will be around for my grandkids in many ways,” he said. “People will be able to control their digital source code for future generations of their families. And you can get to know a grandpa who wasn’t there.”

Cue the horror music.

(By Julia Black, who covers the nexus of tech, media, politics and culture for Business Insider)