BLOGS: My COW Blog Adobe Blog Editing Technology After Effects Final Cut Entertainment

What’s the Deal with that Content-Aware Fill for Video Thing Adobe Showed Us?

COW Blogs : Kylee Peña's Blog : What’s the Deal with that Content-Aware Fill for Video Thing Adobe Showed Us?
Last month at Adobe MAX in Las Vegas, we saw Project Cloak for the first time. The research project and experiment is still in development, and it may or may not ever be in a Creative Cloud product, but it sure was captivating: draw a box, do some Adobe Sensei magic, and poof: that ugly lamp post is just gone. Logo on a t-shirt? GONE! Weirdo in the backgrond of your shot? Yep: EXTREMELY GONE.

If you’ve ever spent hours of your day rotoscoping something to remove it from a shot, you might share the sentiment: I need this. Are we playing God, or are we taking back precious hours of our time? Why not both!

To find out more about the research, technology, and thinking behind it, I talked to Research Engineer Geoffrey Oxholm.

Creative COW: What drove the development of Project Cloak? Was it knowing that people rotoscoped out stuff in a tedious way all the time? Or was it more like “we’ve done this in Photoshop, I wonder if we can do it on a moving image?” Was the genesis more about solving a problem, or more about experimentation?

Geoffrey: The Adobe Research team has strong partnerships (and friendships) with our colleagues on the product teams. We’ve known for ages that “video inpainting” (removing stuff from video) would be hugely impactful to our customers. One of our researchers published the seminal work on this topic as part of his PhD thesis back in 2004. Since then, it has been a problem that the academic community has been working on, but mostly coming up with solutions that have significant barriers to being ready for consumers (limited use-cases, running times, memory constraints). Recently, there have been some breakthroughs in related areas that inspired us to take a closer look.

How long are things like Project Cloak worked on before they get shown at MAX? I’m sure it must vary, but are people working on these things for months, years, or longer?

This problem has been actively researched for over a decade by the wider academic community. While I can’t speak to other Adobe MAX sneaks, the code that’s running behind Cloak was engineered over about 4 months.

Bye, haters.

What more can you tell me about how Project Cloak works under the hood that wasn’t addressed in the MAX presentation?

I can’t dive into a lot of detail, but we call the area you wish to remove the “hole”. The algorithm works by estimating the motion within the hole. We use dense tracking to keep track of what part of the video has seen the hole region either before, or after, the current frame. Using the motion estimation, we essentially shove the pixels through the hole, and record what the frames look like. If there are portions that are not seen in the video, then we fall back on Content-Aware-Fill to guess what the appearance looks like. There are some details around ensuring the appearance looks good across the whole video, but that’s the basic idea. In summary, with Adobe Sensei, Project Cloak uses an advanced algorithm to dramatically accelerate a time-consuming process, delivering results that previously would have required many hours of editing.

Last year there was a bit of backlash and increased interest in Sneaks thanks to VoCo, which was essentially Photoshop for Audio. In this political climate where there’s been a big focus on what’s fake and what’s real. How do you address people that may be concerned about that?

Our Vice President of Creativity, Mark Randall discussed this topic in detail in a blog post following our demo of VoCo last year.

These questions, this dilemma, is part of our legacy as tool builders. We’re deeply inspired by helping real people create amazing new things, works of art, the future, with our tools. Adobe shows projects like Cloak at MAX Sneaks before they’re ready to be out in the world because we welcome the conversations and feedback from the community. As responsible global citizens, we’re committed to thoughtful development of our tools.

As researchers, our life’s work is on magnifying creativity and opening up new avenues for expression. These are real problems that amazing creative people have, and helping them is what motivates us.

And of course: can we please have Cloak in After Effects? Please?

Glad to hear that you’re excited and eager to see Cloak in one of our shipping products.

Posted by: Kylee Peña on Nov 22, 2017 at 2:01:12 am

Focusing on post-production, from editing and motion graphics to personal experiences and the psychology of being an editor.


March 2019 (1)
February 2019 (1)
January 2019 (3)
November 2018 (1)
August 2018 (1)
July 2018 (1)
May 2018 (2)
February 2018 (5)
January 2018 (9)
December 2017 (1)
November 2017 (1)
October 2017 (4)
July 2017 (3)
March 2017 (1)
February 2017 (1)
January 2017 (1)
September 2016 (1)
August 2016 (1)
April 2016 (1)
January 2016 (2)
May 2015 (1)
April 2015 (4)
March 2015 (1)
February 2015 (1)
January 2015 (2)
November 2014 (2)
October 2014 (1)
September 2014 (1)
July 2014 (1)
April 2014 (4)
March 2014 (1)
February 2014 (1)
December 2013 (1)
November 2013 (2)
October 2013 (2)
September 2013 (5)
August 2013 (3)
July 2013 (1)
June 2013 (1)
May 2013 (2)
April 2013 (10)
March 2013 (1)
January 2013 (1)
December 2012 (3)
November 2012 (3)
October 2012 (1)
September 2012 (3)


nab show (16)
post production (13)
editing (8)
post production (6)
video production (6)
editing (6)
Nab show (5)
post-production (5)
southeast creative summit (5)
video production conference (5)
Sundance film festival (4)
women (4)
film (4)
filmmaking (4)
mike api (4)
Sundance (4)
television (3)
parenthood (3)
video editing (3)
olympics (3)
avid (3)
x-files (2)
internships (2)
social media (2)
gender equality (2)
adobe max (2)
independent film (2)
show more
© 2020 All Rights Reserved