EMO Project: generation of videos from a person's photo

  • The EMO Project generates expressive videos from still photos, incorporating sound and movement.
  • Developed by a team at Alibaba's Institute of Intelligent Computing, it uses advanced artificial intelligence.
  • The results are highly realistic, making historical portraits speak and sing.
  • There are ethical concerns about the malicious use of this technology, highlighting the need for regulation.

emo

We are beginning to see with amazement some of the things that Artificial Intelligence applied to our everyday devices is capable of. The results are truly surprising, and this is just the tip of the iceberg. A good example is Project EMO , which consists of generating expressive videos from photos and portraits.

In other words: it is a technology that gives life to static photos, giving them sound and movement. In this post we are going to explain what this idea consists of with some interesting examples.

What is the EMO Project?

EMO is an acronym for Emote Portrait Alive , a project developed by Linrui Tian, ​​Qi Wang, Bang Zhang and Liefeng Bo, three engineers from the Institute of Intelligent Computing , which is part of the Chinese technology and business conglomerate Alibaba.

In the words of its creators, it's a system for generating expressive video portraits driven by audio. This is a fairly accurate definition of what Project EMO is capable of: taking a person's image and giving it expression, voice, and movement. It seems like magic.

This isn't just simple animation tricks that any app could do; it's meticulous, high-precision work reflected in a wide range of facial expressions, as well as head and lip movements. Audio also plays a role, further shaping these movements.

On the other hand, the videos generated can have unlimited duration. They actually depend on the length of the video they are based on.

How Does It Work?

The workings of this incredible tool are explained in detail on the project's website . The method is structured in two different phases:

    1. Initial coding phase in which all aspects of the starting (or reference) image are studied, to better understand what movement and animation can be applied to it.
    2. Processing phase. In it, a pre-trained audio encoder processes the audio embedding, while the facial image generation layer or mask is applied.

Some details of this process should be highlighted, focused on eliminating noise and preserving the identity of the character. On the other hand, other temporal modules are used to adjust the duration of the video and the speed of movement.

The results we're about to present (whether making each portrait speak or even sing) can only be described as spectacular. The AI's intervention achieves such high levels of realism that they completely fool us . Which is undeniably unsettling, to be honest.

EMO Project. Some examples:

We're going to show you some examples of what this technology can achieve. As you'll see, we can use the image of a real person or one generated by AI. We can make it move and gesture, speak in any language we want (saying whatever we want it to say), and even make it sing. Truly amazing.

These are some videos of photos that the EMO Project uses to speak out. Actress Audrey Hepburn comes back to life to talk to us about people's right to cry and express their feelings:

It's also possible to make characters that aren't flesh and blood speak. Here we have Leonardo da Vinci's Mona Lisa , whom EMO Project has brought to life to recite Rosalind's monologue from William Shakespeare's "As You Like It" :

Taking it a step further, we can also take footage of real actors and have them say anything. In this case, we see Rafael Phoenix in his famous role as The Joker , but reciting lines from a different film, The Dark Knight.

Let's move on to the world of music. In the following example, an AI-generated character named Sora performs Dua Lipa 's song "Don't Start Now ." The result is astonishingly human:

Finally, we present a very young Leonardo DiCaprio singing the theme song from the movie Godzilla, composed by the rapper Eminem:

Ethical and legal considerations

The use (or rather, the misuse) of Artificial Intelligence is currently at the center of the debate. It is a disruptive technology whose limits and possibilities we are still unable to glimpse and which, if misused, could have negative consequences in many different aspects.

To cover their backs, on the EMO Project page they make it very clear that all their tests and creations are intended solely for academic research and demonstration of effects. There is no need to look for ulterior motives. However, a technology that can achieve such a degree of precision and realism constitutes a real danger for anyone who wishes to use it to commit fraud, identity theft and other crimes.


Add as preferred source in Google