Project 3
Face Morphing
Face Morphing
In this project, we explore how we can use mathematics to manipulate images of faces and other things!
We will start off by morphing my friend and I into each other
Me
My friend Connor
The first thing we need to do is create a set of correspondences between me and my friend's faces. This is because the computer has to have some notion of how the two images relate to each other so that it can do morphing properly.
The most simple way of doing this is simply by creating a tool where you click on one face, then the other. One example is I would click my chin, then my friend's chin.
This is useful for defining specific places on the image that are equivalent, but what we really need is a way to map ANY pixel in one image to one or more pixels in the other image. In order to do this, we need to extend this notion of equivalence to regions that have a combined area that covers the entire image.
We do this by using something called a triangulation, which can be defined as a set of edges between the verticies we defined, such that no two lines overlap. This will seperate the image into regions such that every point is in exactly one region.
We could do this by hand, but a better way is to use the Delanay Triangulation, which is more optimal in the sense that it maxmizes the smallest angles possible in the triangulation. This is good, because that means there will be fewer long, skinny triangles, which we will see later is suboptimal for morphing. This algorithm is O(nlog(n)) over all verticies.
It is important to note that because the algorithm gives the edges between verticies, it matters whether we use my point set or my friends. In order to give reasonable behavior to both images, we calculate which edges should be connected based on the average of the two point sets. The result of triangulating using those edges is shown on my friend and I below.
Now that we have a method for defining equivalent regions in each image, we need a way of actually computing the transformation. Because we are working with straight lines, a good starting guess would be to work with a linear system of equations.
If it were the case that every triangle had its bottom left point centered at the origin, we would be able to just do a simple linear transformation to map any point on my face or my friend's face to any other triangle. Unfortunately, we cannot make this assumption - triangles can be translated, in addition to being scaled. This means we have to do an aphine transformation of the form y = Ax + b to map one triangle in the source image to the target shape, where x is the original coordinate and y is the resulting coordinate.
Fortunately, it is possible to take these 3 equations (one for each point on the triangle) and stack them into one larger (but simpler to solve) equation of the form y = Ax, where A is a 6 x 6 matrix. Assuming the triangulation was valid, this matrix has a nonzero determinant, and we can simply solve for the mapping.
Finally, by calculating the transformation for every triangle that maps every point in the triangle to the average triangulation of Connor and I, we produce the following images:
My face morphed to the average
Connor's face morphed to the average
Upon averaging the two image's pixel values, we get the "halfway face". Note that there are no correspondences in the background, so they appear messy. The goal was instead to merge the face and hair cleanly, which I beleive has been acheived.
The halfway face
To generate the halfway face, we simply averaged the two images. By instead using the formula (1 - a) * im1 + a * im2, where a is in range [0, 1], we can create a sequence of images, to use in a gif or video. This formula is used for a weighted average of both the triangulation and the pixel values.
Using similar strategies to finding the average of me and my friend's face, it is possible to find the mean face of an entire population. In this case, we will chose to find the average male face based off of a dataset of 37 pre-labelled images.
Note that I relabelled the image of myself to conform to the specfication of this dataset.
Here are some examples, with the triangilization calculated the same way as before:
Example 1
Example 2
Example 3
Average Danish male
My face, morphed onto the average Danish face geometry
The average Danish face morphed onto my face geometry
Earlier, I mentioned the formuala (1 - a) * im1 + a * im2, where a is in range [0, 1]. If we instead chose an a that is outside of this range, we can produce charicatures of each image, where the characteristics that differ in that image from the other are enhanced, rather than attenuated. Here are some examples:
Caricature of an average danish male: a < 0. Lighter, wider cheeks
Average of me and the average Danish male: a = .5
Caricature of me: a > 1. Darker, more depressed cheeks.
Here is a music video made by morphing my friend's faces into each other! The theme is my high school friend group. The song I made myself in FL Studio.