This Neural Network Restores Old Videos


Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. In this series, we often discuss a class of
techniques by the name image inpainting. Image inpainting methods are capable of filling
in missing details from a mostly intact image. You see the legendary PatchMatch algorithm
at work here, which is more than 10 years old, and it is a good old computer graphics
method with no machine learning in sight, and after so much time, 10 years is an eternity
in research years, it still punches way above its weight. However, with the ascendancy of neural network-based
learning methods, I am often wondering whether it would be possible to take a more difficult
problem, for instance, inpainting not just still images, but movies as well. For instance, let’s take and old old black
and white movie that suffers from missing data, flickering, blurriness, and interestingly,
even the contrast of the footage has changed as it faded over time. Well, hold on to your papers, because this
learning-based approach fixes all of these, and even more! Step number one is restoration, which takes
care of all of these artifacts and contrast issues. You can not only see how much better the restored
version is, but it is also reported what the technique did exactly. However, it does more. What more could we possibly ask for? Well, colorization! What it does is that it looks at only 6 colorized
reference images that we have to provide, and uses this as art direction and propagate
it to the remainder of the frames. And it does an absolutely amazing work at
that. It even tells us which reference image it
is looking at when colorizing some of these frames, so if something does not come out
favorably, we know which image to recolor. The architecture of the neural network that
is used for all this also has to follow the requirements appropriately. For instance, beyond the standard spatial
convolution layers, it also makes ample use of these blue temporal convolution layers,
which helps “smearing out” the colorization information from one reference image to multiple
frames. However, in research, a technique is rarely
the very first at doing something, and sure enough, this is not the first technique that
does this kind of restoration and colorization. So how does it compare to previously published
methods? Well, quite favorably. With previous methods, in some cases, the
colorization just appears and disappears over time, while it is much more stable here. Also, fewer artifacts make it to the final
footage, and since cleaning these up is one of the main objectives of these methods, that’s
also great news. If we look at some quantitative results, or
in other words, numbers that describe the difference, you see here that we get a 3-4
decibels cleaner image, which is outstanding. Note that the decibel scale is not linear,
but a logarithmic scale, therefore if you read 28 instead of 24, it does not mean that
it’s just approximately 15% better. It is a much, much more pronounced difference
than that. I think these results are approaching a state
where they are becoming close to good enough so that we can revive some of these old masterpiece
movies and give them a much-deserved facelift. What a time to be alive! This episode has been supported by Weights
& Biases. Weights & Biases provides tools to track your
experiments in your deep learning projects. It can save you a ton of time and money in
these projects and is being used by OpenAI, Toyota Research, Stanford and Berkeley. They also wrote a guide on the fundamentals
of neural networks where they explain in simple terms how to train a neural network properly,
what are the most common errors you can make, and how to fix them. It is really great, you got to have a look. So make sure to visit them through wandb.com/papers
or just click the link in the video description and you can get a free demo today. Our thanks to Weights & Biases for helping
us make better videos for you. Thanks for watching and for your generous
support, and I’ll see you next time!

100 Comments

  1. CheSemah said:

    NICE!

    February 8, 2020
    Reply
  2. TheCroMagnon said:

    i was 5 min late
    🙁

    February 8, 2020
    Reply
  3. 𝙑F𝙓وحش الخدع والافلام said:

    🙂,

    February 8, 2020
    Reply
  4. Karan Mistry said:

    What a time to be alive!

    February 8, 2020
    Reply
  5. kumar mahesh said:

    Congratulations for 500k👏👏🎊

    February 8, 2020
    Reply
  6. Emily Moss said:

    Wow! This would be amazing for restoring old historical or newsreel film

    February 8, 2020
    Reply
  7. DonCDXX said:

    I want to use this on old episodes of MST3K.

    February 8, 2020
    Reply
  8. Mátéツ said:

    Magyar vagy?

    February 8, 2020
    Reply
  9. John Smith said:

    great vid!

    February 8, 2020
    Reply
  10. Tim Solinski said:

    Next Up: Neural Network HD-ports old video game classics in real time
    … zero work for the publisher big bucks in the reboot -_-

    February 8, 2020
    Reply
  11. ∞ Woof said:

    Ok.

    February 8, 2020
    Reply
  12. samurai1993 said:

    Really really impressive. As I have been following the fan made remaster to the original Star Wars films, I wonder how this technique would perform in that task.

    February 8, 2020
    Reply
  13. Che Burns said:

    That first video clip at 0:46 is from the Hanford nuclear site, where the plutonium for the Nagasaki bomb was produced! I've been to both Hanford and Nagasaki, and let me tell you, it's a sobering experience.

    February 8, 2020
    Reply
  14. a51mj12 said:

    Daamn, those nervous networks… That feels like it would be soon the most important technology!

    February 8, 2020
    Reply
  15. muchappreciated said:

    Where we going to get color reference images for old movies?

    February 8, 2020
    Reply
  16. Subramanya M said:

    While I was thinking about it.. You guys are doing it.

    February 8, 2020
    Reply
  17. Novalium Company said:

    But can it run Minecraft on full graphics?

    February 8, 2020
    Reply
  18. Sachin Lodhi said:

    at 3:20, Legend says, "What a time to be alive".
    Thanks Legend.

    February 8, 2020
    Reply
  19. Shubanova said:

    This is absolutely fantastic work guys. Amazing.

    February 8, 2020
    Reply
  20. gregallagher said:

    Very, very cool. The thing that I'm eagerly waiting for, which I'm sure is somewhere down the line, is an AI that can simulate detail in low-res footage—whether using data from surrounding frames or just pulling reference imagery for similar faces, backgrounds, etc. that would, when blurred and artifacted, result in the same image. Probably fairly fair away still, but a boy can dream.

    February 8, 2020
    Reply
  21. John Smith said:

    You keep saying "what a time to be alive" every single video. And wow, it really is.

    February 8, 2020
    Reply
  22. Jody Mitoma's Videos said:

    Holy smokes, man. This is actually remarkable! Thank you so much for all these incredible papers you're sharing with us, Two Minute Papers, and a huge congrats on 500,000 subscribers! So well deserved. It's no wonder I have you guys in my channel's homepage top 10 favorite channels! 🤗❤

    February 8, 2020
    Reply
  23. remliv said:

    Nice.

    Also Jody 499 000 subs.

    February 8, 2020
    Reply
  24. Nick Bryant said:

    Thank you for doing a vid on this

    February 8, 2020
    Reply
  25. Edbrad said:

    Imagine being able to upscale old video games

    February 8, 2020
    Reply
  26. Leandro Gabriel Casas said:

    This is a money maker

    February 8, 2020
    Reply
  27. INDIVELOPE said:

    wowzers!

    February 8, 2020
    Reply
  28. eric vosselmans said:

    Wow. as always amazing stuff!!!!

    February 8, 2020
    Reply
  29. tamenga88 said:

    I would love to see fully restored in 4K Hydrogen Bomb test shots.

    February 8, 2020
    Reply
  30. TallBoy said:

    This is the BEST channel on youtube. Concise videos, fascinating content, no garbage

    February 8, 2020
    Reply
  31. Ash Ketchum said:

    This video: exists

    1969 porn: "It's free real estate"

    February 8, 2020
    Reply
  32. Ayush Bakshi said:

    I heard you name hundreds of time (in your intro) but I never really understand it 😛
    Thanks for converting research papers into videos for us normies.

    February 8, 2020
    Reply
  33. Mauritium said:

    Wow, this is weird. The idea of a restoration AI for old movies just popped into my mind, because I've watched some of your videos lately (Without subscribing), so I searched for your channel to see if you ever uploaded something related to it, and you just posted this video half an hour ago. What are the odds? Apparently I don't even need to subscribe, but I will anyways.

    February 8, 2020
    Reply
  34. mfaizsyahmi. said:

    What are the coincidence that this video comes out less than a week after somebody upscaled and interpolated Lumiere's [Arrival of a Train at La Ciotat] to 4k 60fps, with already astounding results… And that was assembled manually from discrete programs. If this can be done with NN (and even colourized!) then it'd revolutionize the speed of which such content can be converted.

    Please check out that video, it's great! –> https://www.youtube.com/watch?v=3RYNThid23g

    February 8, 2020
    Reply
  35. tamenga88 said:

    We can see Charlie Chaplin movies in colorized 4K

    February 8, 2020
    Reply
  36. Spicy Taco said:

    hmm. I wonder when audio cleaning will come to really restore old films.

    February 8, 2020
    Reply
  37. JAFFE KīkëSTEIN BLAHA said:

    I saw about this script on 8ch last summer

    February 8, 2020
    Reply
  38. AmazingAgent said:

    This is amazing

    February 8, 2020
    Reply
  39. ୲ 𐙼 𐚮 ୲ said:

    I don't know how it was done, but seeing the colorization of the train fade at a distance, it seems to me the algorithm could be improved if the neural net was also provided with a generated video motion estimation layer. There's dedicated HW for it in CPUs, so it could be really fast… Maybe that's what they did though…

    February 8, 2020
    Reply
  40. Shayan said:

    dislikes are old directors who want their movies to be experienced in their 'original' form

    February 8, 2020
    Reply
  41. Emmanuel Nabagabo said:

    3 dB ~= double

    February 8, 2020
    Reply
  42. Atlas WalkedAway said:

    Restoring old movies and family videos, good use of AI.
    Teaching networked AGIs to outwit our best strategist, stupid use of AI.
    One would think this would be more obvious to researchers abroad.

    February 8, 2020
    Reply
  43. veggiet2009 said:

    Now we need one for degraded VHS footage

    February 8, 2020
    Reply
  44. DouglAsshat said:

    A friend and I have been using this for a few months now and it's amazing! We recently did a full colorization and restoration of the Bonzo Dog Doo Day Band's "Metaphorically Speaking", using this as well as DeOldify, EBSynth, and a good bit of manual cleanup of keyframes with PhotoShop.
    Truly outstanding stuff!
    https://youtu.be/y_GTAM8um8M

    February 8, 2020
    Reply
  45. Truth Teller said:

    Sell it to YouTube!

    February 8, 2020
    Reply
  46. Marco Zolo said:

    Star Trek Deep Space Nine and Babylon 5 remastered in 4K is finally within reach 😎

    February 8, 2020
    Reply
  47. alexander Filyaev said:

    stalin would love this

    February 8, 2020
    Reply
  48. Hitasssss said:

    OMG. What a time to be alive!

    February 8, 2020
    Reply
  49. Lugmillord said:

    Man, so eventually old black and white movies will be transformed into HD.

    February 8, 2020
    Reply
  50. Dry Saliva said:

    Nice

    February 8, 2020
    Reply
  51. Mathematical Ninja said:

    Combine with this super resolution and you can see old movies colored in 4k 60fps.

    Combine this with other techniques you can give your face to a character in that movie
    Combine with other techniques you can give your voice to that character in that movie
    Combine with other techniques, you can change the dialogues to how you speak in real life while preserving meaning

    God damn, the future is full of verisimilitude!

    February 8, 2020
    Reply
  52. Lyautey said:

    W H A T A T I M E T O B E A L I V E

    February 8, 2020
    Reply
  53. Skeleton Dude said:

    Let's find the oldest possible footage and colorize it

    February 8, 2020
    Reply
  54. Johannes Zwilling said:

    How does it compare to Jason Antic's DeOldify?

    February 8, 2020
    Reply
  55. Troy Rubert said:

    Am a complete noobie if I wanted to do this at home what is the easiest most efficient way to get this working?

    February 8, 2020
    Reply
  56. Isaak Loewen said:

    The Lone Ranger rides again!

    February 8, 2020
    Reply
  57. Loy71 said:

    What a coincidence, found this today: https://youtu.be/3RYNThid23g

    February 8, 2020
    Reply
  58. Y2Kvids said:

    1930s : What a Time to be Alive . Today's video would be enhanced and colorized in the future .

    February 8, 2020
    Reply
  59. Cosmic Rider said:

    This youtuber does this a lot

    https://www.youtube.com/user/Rick88888888

    February 8, 2020
    Reply
  60. Vicente Vasquez said:

    PLEASE apply this technique on Chaplin’s movies

    February 8, 2020
    Reply
  61. S2 Turbine said:

    If a movie is a masterpiece, keep your crayons off of it!

    February 8, 2020
    Reply
  62. Richard Benson said:

    WATOBA

    February 8, 2020
    Reply
  63. Muonium said:

    Clarke's third law in full effect

    February 8, 2020
    Reply
  64. Conlang no vids said:

    we could use this for video of first steps on the moon did they do that?

    February 8, 2020
    Reply
  65. Quinn Connell said:

    I bet the folks who worked on They Shall Not Grow Old for five years were absolutely THRILLED to see this paper come out

    February 8, 2020
    Reply
  66. xl said:

    Did you download, compile, and test the source code though ?

    February 8, 2020
    Reply
  67. ipulettershere said:

    naked camera app will be a thing one day

    February 8, 2020
    Reply
  68. clapton79 said:

    I am amazed by the complexity of the architecture of the neural network. Would you like to in deeper details? Also, any WWI or early 20th century footage restoration examples are welcome.

    Keep up this good work!
    Sok szerencsét!

    February 8, 2020
    Reply
  69. Carson Smith said:

    This one takes the cake. Brilliant work .

    February 8, 2020
    Reply
  70. TallBoy said:

    WHAT A TIME TO BE ALIVE!

    February 8, 2020
    Reply
  71. Grant Warwick said:

    Can it be used on modern greyscaled footage to produce style transferred lighting?

    February 8, 2020
    Reply
  72. SHAWNFROMPORTLAND said:

    I love the hold onto your papers icon

    February 8, 2020
    Reply
  73. Sekir80 said:

    Imagine enchancing the images with a scaler for more resolution, like here:
    https://www.youtube.com/watch?v=HvH0b9K_Iro

    February 8, 2020
    Reply
  74. shortbreadedtrash said:

    this is amazing

    February 8, 2020
    Reply
  75. Charles Farm said:

    Makes me wonder about Doctor Who, not just for recovering video quality, but also for restoring certain episodes which have lost their color data but which have surviving color telesnaps — John Pertwee episodes. Not to mention the original version of The Lathe of Heaven which was lost save for a single OTA VHS recording

    February 8, 2020
    Reply
  76. Calvin Price said:

    cheating at life

    February 8, 2020
    Reply
  77. Ryan Vickers said:

    This kind of colourization and restoration isn't new by any means, but as far as I know it's always done by hand, frame by frame, so this brings the ability to do that work probably 100x more easily

    February 8, 2020
    Reply
  78. davidinark said:

    This is fantastic! I would love to see restoration based on outlandish reference images, just for fun.

    February 8, 2020
    Reply
  79. Rachel Slur said:

    O

    February 8, 2020
    Reply
  80. Yami said:

    Dear fellow scholars, this Two Minute Papers with cut off your knife a' here.

    February 8, 2020
    Reply
  81. Think said:

    I believe we are at the beginning of the Simulation Age.

    This is the age in which humans integrate their culture with computers in order to prevent an A.I. takeover by going the Augmentation route. Simulations will be used very very heavily to solve all kinds of problems, and humans will spend a lot of their time (perhaps even most) in simulations, since most jobs for humans will involve a lot of simulation work. While robots run their stores and farms and even drive their cars, humans will be the masters of attention, and decision making and VR and AR will be integrated in with that.

    Simulations will become very advanced and be used to recreate basically every single video ever, including eventually even all youtube videos, in infinite-resolution holographic form, and experienced as vividly as regular senses by being directly fed in through our neural links.

    February 8, 2020
    Reply
  82. MrJacobegg said:

    I would really, really like to see this technique applied to VHS video restoration! I have some old videos and camcorder footage from the 90s that I've digitized, but they were in pretty bad shape. This gives me hope that someday soon, I may be able to restore them to what they looked like when they were first recorded.

    February 8, 2020
    Reply
  83. KnightsWithoutATable said:

    But how much processing power does this AI neural network technique take? Are we talking about tons of cloud server time or something that a high end desktop with a good GPU can pull off in a few days for an hour of film?

    February 8, 2020
    Reply
  84. Heinrich Wonders said:

    No! Not my papers!
    I just finished sorting them.
    Again.

    February 8, 2020
    Reply
  85. Alex G said:

    Here they restore an old video about a train by Lumiere brothers, upscaling and adding frames (not color):

    https://www.youtube.com/watch?v=3RYNThid23g

    February 8, 2020
    Reply
  86. Rob ʺEuphoricAgnosticʺ McDoritos said:

    You should make official merch for the channel with the phrase "What a time to be alive!"

    February 8, 2020
    Reply
  87. Sam10947 said:

    Charlie Chaplin movies…..apply plz

    February 8, 2020
    Reply
  88. ProjectPhysX said:

    Color makes old black and white footage so much more vivid. This is really groundbraking work.

    February 8, 2020
    Reply
  89. Joey Skylynx said:

    Would be funny if you would take all of the footage of "paranormal" stuff and ran it through this. I'm thinking bigfoot stuff would be the start.

    February 8, 2020
    Reply
  90. Josef Slotosch said:

    Finally we can have clear Videos of Bigfoot and flying saucers!

    February 8, 2020
    Reply
  91. JaGoodzianka said:

    Imagine removing artifacts, colorizing, and boosting to 4k and 60/120fps
    that would be amazing

    February 8, 2020
    Reply
  92. phero said:

    What exactly does it mean “it’s 4 dB cleaner”? How does dB relate to images?

    February 8, 2020
    Reply
  93. yourtime said:

    2 questions:
    First how are you able to shoot these videos so fast out in this quality, I mean you check them out, but also make your own results
    Second, what do you think of the vienna deep learning meetup or Vdlm and are you there sometimes?

    February 8, 2020
    Reply
  94. Dissonance Paradiddle said:

    You humans are getting so clever 🙂

    February 8, 2020
    Reply
  95. frank x said:

    Wonder if anyone has done similar with early sound recordings.

    February 8, 2020
    Reply
  96. Existenceisillusion said:

    Hmmm … I don't think it will be able to regenerate lost footage … which is what I'm really interested in.

    February 8, 2020
    Reply
  97. pav2k said:

    Stop saying "what a time to be alive" in every video, say it only when you really excited! 🙂

    February 8, 2020
    Reply
  98. Mehregan Karbasi said:

    How do these researchers know about all this different conv layers? I barely know conv. How do they come up with such algorithms?

    February 8, 2020
    Reply
  99. 47guitartom said:

    Chromatic aberration distorts different colors in photographic images to greater or lesser extents. I wonder if it would ever be possible to use ai to analyze chromatic aberration of old images and film to restore the true colors – rather than just colorize the images.

    February 8, 2020
    Reply
  100. x x said:

    ok lets get to work colorizing kurosawa's movies

    February 8, 2020
    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *