By Meredith Alexander Kunz, Adobe Research
Imagine you’re making a documentary on a budget. You have a rare opportunity to shoot critical footage. Then when you load it onto your computer to edit it, you realize with a sinking feeling that there were specks of dust on the camera’s sensor throughout the shoot. Everything is covered in tiny black spots. Now what?
This is exactly what happened to a group of the beta testers of the new feature created by Adobe Research, Content-Aware Fill (CAF) for video—now released in After Effects. “We were able to help them fix the problem using CAF,” reports Geoffrey Oxholm, senior research manager. After applying the new tool, the spots disappeared. “They were so relieved that it worked, they got emotional. This saved them.”
Since its April release, CAF has already been widely acclaimed by the video and broadcast community. The feature took home five awards from the NAB (National Association of Broadcasters) at the organization’s annual show: the 2019 NAB Show Product of the Year and the IABM BaM Award, as well as the RedShark News Best of Show Award and two more.
What makes CAF for video so special? It empowers editors to automatically remove unwanted objects in their videos in an intuitive way, and it is integrated into Adobe’s widely adopted software After Effects. By carefully analyzing the motion in the video, the intelligent algorithm is able to pull in the right content from the rest of the video, leading to a seamless removal.
“There’s a lot of things that this feature does that would have been impossible in the past,” says John Nelson, senior research software developer, who helped create the Academy-Award-winning After Effects in its early days and now works for Adobe Research.
“Dust, reflections, logos, booms—video editors constantly have to remove them. Doing it correctly with motion in the frame is really hard,” Nelson explains. “We’re making the impossible possible, so people can focus on the more creative aspects of their job instead of the tedious ones.”
Being able to remove things from video isn’t entirely new, Oxholm reports, but this innovative feature—which grew out of Oxholm and team’s work on Project Cloak, showcased at Adobe MAX 2017—makes it a lot easier and quicker for the average After Effects user. And as the team developed the tool, they found that many customer testers agreed that it was a lifesaver.
“Throughout the process, we’ve had customers in the loop. Getting their data and feedback has been essential in making this feature what it is,” Oxholm explains.
“It was super motivating to me to be able to solve customers’ problems. This helps people who are making worthwhile, creative things, but who don’t have the budget of a blockbuster movie,” he adds. He points out that many big-studio movies spend large amounts of money and time selecting items in the frame manually and editing them. CAF for video can automate much of that work.
Oxholm and Nelson partnered closely in the effort to create Content-Aware Fill for video, along with Oliver Wang, Seth Walker, and the After Effects team, to ensure that the technology functioned well and would also be user-friendly for customers. “The teamwork on this project has been outstanding—especially in how we track, reproduce, and follow-up on any issue customers report,” says Oxholm.
They created a “velvet rope pre-release,” where they offered the beta tool to just a handful of customers at a time for testing. Through that process, CAF for video was steadily improved with the users’ needs in mind.
Instead of making it a complex, memory-heavy “effect” in the software, the team took a different approach. “We generate footage that only has holes, and that perfectly infills the holes. It becomes a layer in the video,” Nelson says. He notes that the tool doesn’t add a lot of data that would hamper users and load very slowly. It also offers users the chance to preview results before rendering a full set of footage.
Wang, senior research scientist, points out that another advantage of this tool is that it is “simple enough for beginners, while still providing detailed control for professionals.”
He explains that the feature’s method works well with a “loose mask around the object,” which is easier to create than precise object rotoscoping. “If you’d like to fine-tune the result, we have a workflow where pressing one button allows the user to go into Photoshop and tweak one frame, and then go back to After Effects,” Wang says.
Many professional After Effects users are now fully engaged in this feature. Walker, senior experience designer, was glad to see that customers are already offering tutorials and videos about how to use it. “A lot of people are demonstrating how CAF works, what it works well for, and what it doesn’t. Customers are really excited about it, but they also need to be properly educated,” he says.
More innovation is in the pipeline for this feature and other video breakthroughs from Adobe Research. The team plans to focus on tracking, depth, and 3D video understanding, seeking new ways to enhance the video editors’ toolkit.
Two summer interns in 2019 will work on improvements to CAF for video, and the team will soon meet with high-end visual effects companies in Hollywood to learn how future innovations could serve their needs.
The Adobe Research team appreciated the NAB recognition and see it as encouragement to propel their research and development further.
“It’s really great to get this recognition from customers,” says Oxholm. “It seems to be their way of saying, ‘How can we encourage Adobe to keep giving us products that will save us time and money?’”
Contributors:
Geoffrey Oxholm, John Nelson, Oliver Wang, Seth Walker, Eli Shechtman, Michal Lukáč, Adobe Research
Experiment with this new feature by following these instructions!
Creative professionals are talking about Content-Aware Fill for Video. Check out this review video and find more here and here:
