Famed actor and motion-capture artist Andy Serkis made a surprise appearance at Epic Game’s GDC press conference Wednesday to show off the latest in real-time human-driven digital acting.
Serkis, best known for his work as Gollum in the Lord of the Rings films and Caesar in the Planet of the Apes films, introduced new work being done by Epic and 3Lateral to better capture the look of an actor and recreate that person or any person using their recorded performance as a believable, near-human digital recreation.
To demonstrate the technology, 3Lateral founder and CEO Vladimir Mastilovic showed off a recreation of Serkis performing a piece of Shakespeare’s Macbeth. The fidelity, which was surprisingly human, was running in real-time using the new tech built on the back of Epic’s Unreal graphics engine. To prove that the video wasn’t simply a canned, pre-rendered performance, Mastilovic had the video run a second time, this time raising and lowering the digital Serkis’ eyebrows and moving his eyes around, manipulating where he was looking.
The volumetric data used in the creation was generated by capturing a series of high-quality, HFR images of Serkis from multiple angles under controlled lighting, according to the company. 3Lateral’s process involved various capture scenarios, some focused on geometry, some on appearance and others on motion. All of these inputs were applied to generate a digital representation of Serkis, and to extract universal facial semantics that represent muscular contractions that make the performance so lifelike.
The tech allows the performance to be tweaked, but also allows lighting and other elements to be changed. To drive home the point of the power and potential of the new performance capture tech, 3Lateral ran the performance again, but instead of seeing Serkis, it was a large blue alien who appeared just as emotive as Serkis had, recreating his emotion, albeit through unfamiliar blue skin.
“The implications of this technology are many,” he said. “We have the digitized actions of Andy Serkis and it can be used to reshoot a scene, react it through a different person or even preserve it in case when he is older he wants to act out the scene as a younger man.”
The company also showed off work they were doing with Epic, Cubic Motion, Tencent and Vicon with a demo they called Siren. That tech was used to create a digital human called Siren which was then puppeteered by Chinese actress Bingjie Jiang.
Both sets of tech are expected to one day be used in everything from video games and VR performances to television, film, live performances and broadcasts.