Research Scientist Taesung Park is using AI and machine learning to transform the old pixel-by-pixel model for editing digital images into something new — image-to-image editing.

We talked to Park about the technology behind image-to-image editing, how he helped create the Landscape Mixer feature in Photoshop, and what he loves most about Adobe Research.

Can you tell us what you’re excited about in the world of image editing and AI right now? How does your research fit in?

Lots of researchers are using AI to understand or perceive things and, more and more, people are using AI to generate content. This past year, we heard a lot about text-to-image models — you type in text prompts and they generate a picture.

This is an amazing use of AI because it understands text and generates the right content for you. But it’s not very useful for image editing because you usually want to start from your own image. That’s a big research challenge — to go from the text-to-image model and make it perform image editing. That’s what I like to do.

My method takes an image as input, performs some editing, and then outputs another image. We call this image-to image, and this approach could be valuable for many Adobe applications.

You started working on image-to-image technology during your two summers as an Adobe intern. Can you tell us about that research, and how it led to the Landscape Mixer feature in Photoshop?

There’s an AI technology called GANs (generative adversarial networks), which generates realistic images. But it doesn’t necessarily generate what the user wants. So the research I worked on as an intern, Swapping Autoencoder, takes the user’s own image as input to start, and then generates variations.

The challenge is identifying what needs to change and what needs to be preserved from the input image. For example, let’s say the user uploads a photo of the Alps to edit. Their intention is to preserve the structure of the scene. In this case, that’s the shape of the mountains. But it’s not obvious how to identify which pixels or attributes need to be preserved.

So, at a high level, my research looked at how to identify the structure and the style of the image so we could maintain the structure but edit the style. To do it, we take very small patches from input images, and shuffle them— we basically turn the input image into a jigsaw puzzle. And then we look at the individual pieces to get the color, tone, and sharpness of the image — the style. That’s the main trick. Then, we can preserve the structure while changing the style.

During my internship, we published a paper on this research. Then, when I joined Adobe Research full-time after completing my PhD, we wanted to transfer the technology into an Adobe product. Our next step was to improve the quality even more. To do this, we used Adobe 3D technology to scan images for an even better understanding of structure. That work went into the new Neural Filter for Photoshop called Landscape Mixer, released in 2021.

How do you think Landscape Mixer changes the experience for Photoshop users?

It’s a very different mode of editing from what people are used to in Photoshop. Usually, you’re moving the pixels around. But, by using generative AI, Landscape Mixer lets users make radical edits that weren’t possible before.

For example, users can bring in their own photos, and if they don’t like the time of the day or season in the image, then they can easily change them. I think it’s a really exciting, pleasant way to edit.

You’ve said that stargazing first sparked your interest in images and AI. How did that happen?

Back in high school, I was into astronomy and stargazing. And if you love astronomy, you’ll probably want to take photos of the things you’re observing. But a single nighttime photo doesn’t usually look very good because there’s a lot of noise. You need computational photography, which lets you take lots of photos of the same object and composite them to for a better output. That’s how I got hooked on image editing.

Then, as an undergrad, I decided to major in mathematics. I wanted to use math as a springboard for something else, and that brought me back to machine learning and computer vision. It’s all about combining math with images.

What drew you to Adobe Research? What do you like best about being here?

The big advantage of joining Adobe is having the freedom to pick your research topic and having tight connections with products, so you can use what you develop in your research to build tools people will use every day. For me, that really gives my research grounding and meaning.  

Interested in an internship or career with Adobe Research? You can learn about opportunities here.