Computer scientists create algorithm to digitally swap faces, voices in videos

CCTV News

Screenshot from video by Supasorn Suwajanakorn. Screenshot from video by Supasorn Suwajanakorn.

Face-swapping is a fun and popular activity on mobile apps, but the graphics and imaging lab at the University of Washington is taking it to a whole new level.

CCTV America’s Mark Niu reports from Seattle.
Follow Mark Niu on Twitter @MarkNiuWrite

Computer scientists create algorithm to digitally swap faces, voices in videos

Computer scientists create algorithm to digitally swap faces, voices in videos

Face-swapping is a fun and popular activity on mobile apps, but the graphics and imaging lab at the University of Washington is taking it to a whole new level. CCTV America's Mark Niu reports from Seattle.

University of Washington PhD student Supasorn Suwajanakorn is using a face-modeling algorithm that turns photos into a digital “persona” that can say and do things that were never actually done by the person behind the persona. That means, someday, you could have a real conversation with Grandma on your couch even though she lives miles away.

What Makes Tom Hanks Look Like Tom Hanks

What Makes Tom Hanks Look Like Tom Hanks

Video courtesy Supasorn Suwajanakorn of the University of Washington: We reconstruct a controllable model of a person from a large photo collection that captures his or her persona, i.e., physical appearance and behavior. The ability to operate on unstructured photo collections enables modeling a huge number of people, including celebrities and other well photographed people without requiring them to be scanned. Moreover, we show the ability to drive or puppeteer the captured person B using any other video of a different person A. In this scenario, B acts out the role of person A, but retains his/her own personality and character. Our system is based on a novel combination of 3D face reconstruction, tracking, alignment, and multi-texture modeling, applied to the puppeteering problem. We demonstrate convincing results on a large variety of celebrities derived from Internet imagery and video.

Suwajanakorn builds a 3D model of people’s face and captures details such as eye color, skin color, and even the texture of the skin.

Their voice, words, and facial expressions can even be transferred onto images and videos of other people.

For example, George W. Bush’s words can even be placed onto Barack Obama or even Tom Hanks.

George Bush's face superimposed on other people

George Bush's face superimposed on other people

Video courtesy Supasorn Suwajanakorn of the University of Washington: Here textured 3D models are reconstructed for famous people using only 2D photos from the internet and are driven by a video of George W. Bush. This is a result from our paper "What Makes Tom Hanks Look Like Tom Hanks" submitted to International Conference on Computer Vision 2015.

The researchers are currently working on replicating a person’s precise movements.

“Later on we can try to reconstruct other aspects such as their dynamics, their behavior, or their personality, and recreate a model that’s interactive that you can talk to and then we could maybe take video footage of historical figures like Albert Einstein and then create a model of Albert Einstein that you could talk to,” Suwajanakorn said.

That would surpass the limitations of current technology, which requires a subject to be brought into a studio and covered with motion capture sensors.

University of Washington researchers said Hollywood and the video game industry have already shown great interest in their technology. But it does raise some ethical issues. Will we eventually be able to make anyone look like they said something they never did?

This video of a computerized face swap will freak you out

This video of a computerized face swap will freak you out

A team at the University of Washington has created a computer algorithm that digitally swaps faces. Watch as George W. Bush becomes Barack Obama.

“It’s the same process that happened with Photoshop already. If you see some photo you might not trust it right away that it’s real. I think people might start doing that if they know someone can do this,” Suwajanakorn said/ “But in terms of measure or how do we prevent that? I’m not sure.”

One solution would be to create software that could detect the difference.

But for now, the focus here is purely on capturing the dynamics of what makes each person unique.


 

Media consultant Ari Zoldan on the next hot new gadget

CCTV America interviewed Ari Zoldan, chief executive of consulting company Quantum Media Group about the next new trend in technology, including “swallowables”.

Follow him on Twitter: @AriZoldan