In this assignment we use matlab to implement the required algorithm: Gradient-domain fusion.
In this initial excersize, we tried to match the gradients, plus one pixel intensity, on a toy problem given in the assigmnent. Since we mininimized the energy between an image itself, this way the optimum solution is trivial; it is the image itself. We can check if the module of sparse matrix and the way to formulate the constraints are good.
|1||Minimize ((v(x+1,y)-v(x,y))–(s(x+1,y)-s(x,y)))^2||The x-gradients of v and s should be matched closely|
|2||Minimize ((v(x,y+1)-v(x,y))–(s(x,y+1)-s(x,y)))^2||The y-gradients of v and s should be matched closely|
|3||Minimize (v(1,1)–(s(1,1))^2||The top left corner of the two images should be the same value|
Here are the image we used as shown below, we did get back the origional image, and our sum of squared error between the images was very low. In the table, a map of pixel differences is shown. The error is negligibly small(mean:1.5517e-15,std:1.6817e-16).
|input||output||time||RSS error||map of pixel differences|
The key concept is to solve the following optimization:
In this part, we need to blend part of an image(foreground image) into another image(background image). However the sharp seam associated with directly pasting images on top of one another corresponds to an “unnatural” gradient value. To prevent this, we need to figure out an algorithm which can “fit” the foreground part into the background image.
This is solved by discrete Poisson equations. The pixel colors we want to transplant into an image will be unknown initially, since they will be adjusted (e.g. for color constancy, edges) if we want the composite image to look right. We form a system of linear equations describing these unknowns, using our known data (unmasked colors that are unchanged in the target image) and solve in Matlab. One way to go is to maintain the original gradient of the background image along the border location ,and preserve all other gradients. The blending constraints are given in the Perez paper.
After inplementing the MATLAB code, we tested on several picture pairs and the results are given below. You can see that most of the output pictures is visually pretty “natual”.Successful results
Blending works best when the background of the source image matches the color and general texture of the area it is being pasted into in the background, as above five examples show.Failed results
Here we got an example that is not so successful. When we put Ronald McDonald near KFC restaurant, it seems that he was “welcomed” by a boom to his head. Why is that? Commercial conflict？Not really. It’s because the background’s affects a lot to the foreground part and the color of Ronald McDonald’s upper body is influenced by the color of KFC’s brick wall and glass window.Conclusions
From the results we can see, Poisson blending can change the color of our source image. The changing of color is the algorithm tends to blend the source image into target image to smooth the seam. If two images's color is different too much, the blending results could be bad because the change of color make our object unrealistic.
Ideally, the background of the object in the source region and the surrounding area of the target region will be of similar color. However, if the difference between them is too large, the result image exists weird color and blurred boundaries in the blending area.
Here we impliment a similar method to the previous section, but instead use a mixed-gradient approach, which uses the gradient in either the source or target image as a guide depending on the magnitude of the gradients. The formula is:
Here, d_ij is the gradient from the source image if the magnitude of the gradient of the source image is larger than that of the target image's. If not, then d_ij is the gradient from the target image. Firstly, we do mixed gradients blending. Then, we do poisson blending in order to compare the output images of two blending methods. Results
We note that the result of poisson blending is not satisfying with its noticeable backgroung color from the source image, while the mixed gradients blending provides a wonderful result of words blending into the wall but without any noticeable artificial effects. Although Poission Blending is much better than just copy and paste the pixels from one image directly into the other which create very noticeable seams, it lost all the pixel intensity of the background in the blending area.
Some problems with mixed gradients: Foreground object get transparent in the background. It happens in Mixed Gradient Blending. The reason might be since Mixed Gradient Blending consider both gradient from source and target image, it will prevent the "Object in the target image is replaced/hidden" issue but will change the foreground color and make it transparent.
Bell & Whistles(Extra Points)Color2Gray
When converting an image to grayscale, we can lose a lot of information about the contrast in the image. In this part, we’re going to turn a color picture to gray without loss of contrast and gradient. First I converted the image from RGB to HSV, then looked at the gradient in each channel. Use Mixed Gradient Blending method at each pixel to select the one with larger magnitude, in the mean time keep the grayscale intensity the same. Here you can see some examples.
1) Build Laplacian pyramids for each image.
2) Build a Gaussian pyramid of region mask.
3) Blend each level of pyramid using region mask from the same level.
4) Collapse the pyramid to get the final blended image.
Here are some examples from both the slide and flickr website (using keywords "same place different time") implemented by ourselves, using pyramid level N=5:
|source1||source2||raw blend||laplacian pyramids||poisson blending||mixed gradients|
From the results we can't tell which method performs the best. There is no such an ideal method that can handle all kind of image blending and do the best job in every detail at the same time.
Laplatian blending preserves the best color feature among the three methods. However, people can clearly notice a huge overlapping region with stong low frequency mixing, which might be a bad thing if we really want to have a seamless blending. However, it just smooth the edge, didn't remove the seam at all.
Poisson blending and mixed gradients blending do seamlessly blend two images, but the color of the masked image is totally different from the origin because of obvious mismatch between this two images. Moreover, because there contains both strong gradients in both source and background images, such as trees, the mixed gradient result shows double image effect that the tree trunk in the winter scene also appears in the masked autumn scene.
In a word, one should know the pros and cons for each blending method, and choose the best one for each specific case.Color transfer
We implement the color transfer algorithm described in this paper. Here are some steps and details.
1. Change the source and target images to L*a*b space. This space is similar to how people perceive color.
2. Find the means and standard deviations of each channel for both images.
3. For each channel in the target image, subtract the mean of the target image for that channel.
4. Scale each channel in the target image by (std deviation of target channel)/(std deviation of source channel)
5. For each channel in the target image, add the mean of the source image for that channel.
6. Combine the target channels together again.
We both have so much fun to learn about how to utilize the gradients in an image, and experiment and compare different blending techniques. However, the biggest takeaway for us is the use of sparse matrices in MATLAB. Following the sparse matrices construction method taught in class, our blending algorithms can be done in just several seconds.
-  P. Perez, M. Gangnet, Poisson Image Editing, ACM SIGGAPH 2003.