You've probably seen this video, right?
It's all over the internet now.
Why?
Because it's a video from 1911 that has been
converted from this old camera footage to
4K, fully colorized video, running at 60 frames
per second.
That in itself is just insane, but guess what?
All of that was done automatically, by using
AI - or artificial intelligence.
That immediately got me interested in how
to do this...
In this video we gonna convert this...into
this.
By the end of it you should be able to convert
ANY black and white old footage into 4K, 60fps,
fully colorized video by using AI on your
own computer!
Stick around, you gonna learn so much and
you gonna be surprised how easy it is.
Hey what's going on everybody, Tom here with
another video for you my fellow creatives.
This tutorial is very special because we're
going to look into AI + video.
It's something that, quite honestly, blows
my mind.
If you are interested in mind-blowing content
for creatives or learning tools like After
Effects, Premiere and so on, hit the red subscribe
button below and also that little bell next
to it so you get notified when there is a
new cool video on our channel.
Right, so first talk about what we gonna be
doing, what tools you gonna need and what
is the most important computer hardware when
it comes down to AI.
As you saw in the intro section, this video
from New York was originally recorded in 1911
by a Swedish company on their trip to America.
And this one was created by Denis Shiryaev
by applying different AI tools to create the
final look.
One was used for changing the original framerate
to 60fps, another one for colorizing the footage
and another one for upscaling the it to 4K.
Now, I've spent some time playing with different
techniques for this tutorial, because when
it comes to AI it feels very overwhelming,
especially if you've never done anything like
this before.
So I've found a good workflow how to accomplish
everything we need without any coding or going
into some advanced technical topics.
I think this tutorial is great if you want
to start with AI and see what it can do - and
who knows maybe it will inspire you to dig
deeper, learn Python and get properly involved
in AI for video.
Probably the most important thing to check
before you begin is your GPU.
Most of these tools I'm gonna be showing you
require Nvidia with CUDA.
You can check if your card supports CUDA on
this link over here, I'll also put it in the
description below.
It's an official nvidia page with the current
list of CUDA supported GPUs.
Unfortunately, CUDA is only supported on Nvidia
cards so if you have AMD GPU, most of these
AI things won't work on your machine.
There are ways how to overcome this, like
running it on a virtual machine - I'll talk
about it briefly as we go through these steps.
First, we gonna get the footage and prepare
it by removing duplicate frames, stabilizing
it, removing scratches and use AI to convert
this choppy, low fps video, into butter-smooth
60 frames per second by using a tool called
DAINAPP.
Then we add color to it by using another tool
called Deoldify which can automatically add
color to our original black and white footage.
Then we gonna take that and upscale it to
4K by using an app called Gigapixel and finally
we gonna tweak the final result in After Effects.
It's way less complicated than it sounds so
don't get scared, it's not gonna be difficult
at all.
For this video I'm gonna use the same footage
of 1911 New York just to show you that you
can achieve exactly the same effect as Denis
did in his original video but of course, this
will work with any other footage as well.
My personal recommendation for finding these
old videos is either from BFI website or archive.org.
Both of them are absolutely stunning.
I especially love how you can search BFI by
using this interactive map.
You can find anything there, it's like Netflix
of old times.
It's great.
The first thing we need to do with this footage
is to cut out a segment that we gonna be working
on.
Look, the video is 8 minutes long so forget
about doing all of that in one go.
So let's jump to Premiere, or whatever else
you prefer, import the video there and cut
out a section that you want to use.
Now, this is an old footage so it's very likely
it is not 30fps but something lower, like
10 or 15 frames per second but converted into
fake 30fps somewhere in the process of that
video being let's say on Youtube.
What that essentially creates are these duplicate
frames.
This is a problem for us because we really
need to think about AI and time it will need
to go through every single frame.
You don't need to feed it duplicate frames
as it will take longer, plus, AI might produce
weird results.
So let's get rid of them.
Select your sequence, make sure the end of
the sequence is set like this and then go
to File -> Export -> Media or Ctrl + M. We
don't need to do much here just select h264,
disable Audio as we don't need that and set
our export path and name over here, let's
say we gonna name it original_footage_cut.mp4
The easiest way to remove duplicate frames
is to use the latest version of DAINAPP that
comes with duplicate frames removal built-in.
Now you might thinking, "hey, you said DAINAPP
is for converting our footage to 60fps" and
you are right, we gonna use it later in the
process to do that, but for now, we just need
it to remove the duplicate frames first.
So google DAINAPP, go to this page and then
click on the Patreon link.
Why?
Because we need the latest version that seems
to be only available on Patreon at this point.
You don't need to pay anything, but it would
really help the team behind this project to
keep working on updates.
So if you plan on using DAINAPP more often,
I suggest you chip in and help these guys
that way.
Right, so scroll down and find the latest
version, click on it, then click on Download.
When it's done, unzip it, get inside and right
click on this exe file and choose Run as Administrator.
Ok so let's browse for our footage and make
a note of the original fps that is displayed
over here, we gonna need it in a second.
Select output folder and then click on Step
1 - Create frames.
This is going to export our mp4 into this
PNG image sequence.
Head over to DAINAPP again, choose a different
folder for exporting and this time, select
Remove duplicate frames here.
DAINAPP is going to identify and remove all
duplicate frames so that means we will end
up with less frames and that means our footage
will look faster.
Because it probably removed half of the frames,
it will be twice as fast.
But you don't have to worry about this as
we let AI to recalculate these frames later.
What you do need to know is the actual fps
of this new footage without those duplicate
frames.
So in this folder we ended up with all of
these PNG files, if you select them all it
will tell you the total number of them.
Let's just get rid of this last one as it's
just a black screen so in total we have 445
frames.
This is the total number of frames coming
from that footage before we removed any duplicates.
Write this number down and then look inside
that export folder where we exported the footage
without duplicate frames.
If you select them all you will see there
is 228 files, minus the last black one - it's
227 frames.
Divide these two numbers small by large.
That is your multiplier you need to use to
multiply the original fps value, remember,
the one we got when we loaded the original
footage into DAINAPP.
And we get something around 15.
This means our new footage, with removed frames,
is in fact 15 fps.
This is going to be useful in just a second.
Just remember, this is not a universal number,
so it could very well be that your original
footage was 30fps, you removed duplicates
and calculated the final fps to be let's say
10, or 20 or whatever.
Of course this usually only applies to old
footage that has been converted to higher
fps by using frame duplication.
Because this footage was originally recorded
on a film roll, over time it developed small
imperfections in form of these random scratches.
There are many tools that can remove them,
some free, some paid, but a lot of those built-in
tools tend to soften your image by blurring
parts of the footage that should, in-fact,
stay sharp.
You don't want to end up with something that
looks too over processed.
I've tested a few of these solutions for doing
this stuff but I think Neat does a pretty
good job without needing a degree from spacecraft
engineering.
They have a free version available for non-commercial
use, but it puts a large annoying rectangle
over your footage which is not great but will
do the job if you just want to test it out.
Remember, you can skip this step completely,
it's not going to affect your final result
as much, it just looks a bit better I think
without scratches and other imperfections.
If you want to upgrade to full version, it
costs around $130.
So let's get Neat for Premiere from their
website, close Premiere if it's still running
and install it.
Now, open the original Premiere project we
did before and import our newly exported 15fps
image sequence - the one where we removed
duplicate frames.
Create a new sequence from it and rename it
to let's say prepared_footage.
I'll just switch to Editing workspace here
to see effects panel and search for Neat,
drag and drop it onto your footage, in my
case I'm still running on version 4 so let's
use that one and head over to Effect Controls.
As you probably know, Neat is a noise removal
tool kit so we need to set it up for that
even though we're only going to use Dust and
Scratches filtering.
So click on Prepare, then Build and you should
see this noise profile screen.
If you're removing noise from your footage
as well it might be a good idea to play with
these settings, but for this tutorial let's
hit Auto Profile and Neat will try to create
that noise profile for you.
Sometimes it can't find a suitable area but
it's not a massive deal for us as we're mainly
focusing on removing scratches.
So if you see any popup like this, click on
Yes, then switch to Noise Filter Settings
over here.
On this screen, you'll see your preview window
where you can toggle between your original
and filtered footage.
Let's jump to this sidebar over here and disable
Spatial, jump back to Temporal and enable
Dust and Scratches.
Depending on your footage you might need to
adjust these settings over here, perhaps enabling
Advanced mode down here that will allow you
to dig deeper into them.
You can also choose a different frame down
here, just to double check and when you're
happy, hit Apply.
Another thing we need to do is stabilize the
footage.
Premiere comes with a quite powerful warp
stabilizer so let's use that.
Just search for it in the Effects Panel and
drag it over onto your footage.
Give it a few seconds and you should have
see a stabilized version.
If there are any issues, play with the settings
in Effect Controls over here.
One thing you probably want to change is Framing
and switch to Stabilize Only because otherwise
Premiere will automatically scale up this
footage.
I personally prefer to do this later in the
process where I can decide if I want to do
this or not.
Perfect, so now we need to get it out of Premiere
as a new image sequence.
Select your sequence, make sure the end of
the sequence is set like this and then go
to File -> Export -> Media or, again, Ctrl
+ M. We don't need to do much here, just double
check the resolution and if the frame rate
is correct, and disable Audio as we don't
need that.
Now to the export path, click on this name
over here and create a new folder called prepared_footage.
Inside of that folder create another one called
original_frames and name your file exactly
"0.png".
This is very important for DAINAPP to be able
to locate these files in the next step.
Hit export and we should be ready for the
next step, that is, converting our video from
15 to 60 frames per second.
This is the section where you gonna push your
GPU to it's limits.
Think about it, it has to calculate all the
missing frames between 15 and 60.
That's 3 brand new frames for each existing
one.
As I said before, this requires CUDA supported
GPU and the more memory you have, the better.
I'm gonna use quite outdated Nvidia 980Ti
with 6gig of memory just to show you a couple
of tips you can do if you have one of these
older GPUs.
So let's open DAINAPP as Administrator again
and choose Input PNG Sequence.
As our input folder we need to select the
prepared_footage folder, not the one called
original_frames, and as an Output we need
to select the parent folder of prepared_footage
folder, which in our case is the main folder.
If you choose anything else, DAINAPP will
throw an error.
The second thing we need to do is to tell
DAINAPP what is the original fps and what
we want back.
As you remember, we calculated that the original
fps was 15 so let's put that here and choose
4 times in this dropdown.
This basically means DAINAPP will try to create
all those missing frames and output our footage
as native 60fps.
You can go higher if you want but of course
that will take a lot longer to process.
If you hit this middle button down here called
Step 2, it will start processing those images.
Now, it's very likely you will see something
like this.
This popup is telling you that your GPU doesn't
have enough memory for this operation.
There are several methods how to overcome
this.
First, very obvious one is to buy a more powerful
GPU, but there are also options within DAINAPP
that can help.
This one in particular is quite clever.
If you enable it, it will split your image
into several smaller sections, process them
one by one and then stitch it back together
before it renders out that frame.
I'd recommend playing with both Section Size
and Section Padding and see if it helps.
For my Nvidia 980Ti, something like 400 and
100 does the job.
Keep trying different values and see what
works for you.
If everything goes well, you should start
seeing something like this in that terminal
window that was opened by DAINAPP.
This will take some time to process all of
those files but you can always go inside or
this folder that was created next to your
original frames and see the result.
So while we're waiting for DAINAPP to finish
processing those frames let's talk about something
that is becoming a new trend now.
It's converting static photos into moving
images.
And guess what, this video is brought to you
by Photomotion that can do all of that directly
inside of your After Effects.
It helps you convert your static 2D pictures
into professional-looking 3D photo animations.
It comes with multiple tools specifically
designed for photo animations such as portrait
animation, parallax images, depth animations
and pretty much everything else, including
cinemagraphs and plotagraphs.
This is the only photo animation toolkit you
will ever need to create high-quality, professional-looking
animations.
If you've never done any photo animation before
- it's probably good time to start now as
we've got you covered with a lot of video
tutorials and our own dedicated animation
team on live chat that will help you whenever
you need.
So if you are thinking about making your images
stand out, try creating these 3D photos.
They can really capture the attention and
make sure your pictures stand out on platforms
like Facebook, Instagram, documentaries or
your own website.
I'd love to see where you're using Photomtoion
and how it helps you to stand out so leave
a comment below and let me know.
Okay, it took about an hour to process all
these frames but before we jump to colorizing,
it's a good idea to check your results first.
Hop into Premiere, import it there, then right
click on the image sequence and choose Modify
-> Interpret Footage and change this field
to 60fps.
Now, just play or scrub through the footage
to see if you can spot any issues.
Like this one for example.
We just need to re-render those couple of
frames again by going to that folder, deleting
those frames that we want re-rendered and
then jumping back to DAINAPP and clicking
on Step 2 again.
Now of course, if you closed DAINAPP in the
meantime, make sure you're using the same
settings as before and then click on Step
2.
DAINAPP is smart enough to only re-render
those missing frames.
If you're still having issues, the problem
might be in the original file so have a look
there or jump to Photoshop and fix it over
there.
To be honest this only happened to me once
so it's not something you need to do every
time.
Cool, already looking really good but it's
missing color.
To do this we're gonna use a great tool called
DeOldify.
It is really really amazing as it can colorize
static images and videos too.
Of course, it might not be 100% historically
accurate in terms of color but it does a really
good job.
You have a few options how to use it.
The very obvious one is installing it on your
computer.
Mind you, this is quite difficult to do on
a Windows machine.
I've done it on my EC2 instance running on
Linux but the problem there is with the pricing.
See, Deoldify needs a powerful GPU too and
of course those EC2 instances are quite expensive.
Well, depends how often you need them but
there is a better option that is actually
free.
Some good people created Colab notebook for
DeOldify so we can use it for free as it runs
on machines provided by Goolge.
Sounds too good to be true?
Well it's actually quite good and simple to
set up as well.
First, we need to export our 60fps image sequence
into video and upload it on Youtube, Vimeo,
Twitter, Dropbox or your own server.
Just be careful here because as soon as you
upload that video on sites like Youtube it's
gonna get compressed in a quite significant
way.
Just look at this example.
This is the original video, and this is the
one that Deoldify downloaded from Youtube.
So to avoid this issue, I'm gonna use Dropbox
to host this file instead, because Dropbox
doesn't compress your videos and therefore
Deoldify will download the original footage
before adding color to it.
Okay so let's jump to Premiere, select your
sequence, make sure you have the end point
here and go to File -> Export.
This time we gonna need h264 codec, keep the
settings on high as we don't want to loose
too much quality - although to be perfectly
honest, this is an old footage and we're going
to upscale it in a second, so it really doesn't
matter that much.
Time to go to Deoldify Github page.
Scroll down and click on this Colab button
that will take you to Google Colab.
You need to log in with your Google account
and then we just need to make sure our runtime
settings are set to Python 3 and GPU.
If this code doesn't make sense to you don't
worry, you just need to click on this play
symbol and wait for it to finish spinning.
Do the same thing with the others as well
until you get to this one, that requires URL.
As I said before, this video needs to be hosted
somewhere on the web but try to stay away
from sites like Youtube because of those compression
issues.
So let's head over to Dropbox and upload it
there.
When it's done, click on Share then Create
Link down here and then just copy this URL
and paste it here.
I'm gonna leave everything as it is and disable
watermark, then click on play.
So Colab will now download the footage from
Dropbox over to their server and then run
Deoldify on it.
This will take about 5 minutes for our footage
and then you should be able to see your colorized
video over here.
It's important to let this spinner finish
even if this progress bar shows 100%.
You can now download this video to your computer.
In case you don't see your final video even
after the spinner stopped spinning, go to
Folders over here, click on Deoldify, then
video, result and download this video.mp4
file.
It's already looking really good but it's
still in 1080p and we want it in 4K right?
You can of course just scale it in Premiere
or use Detail preserving scale in After Effects
but that's not real 4K.
That's where Gigapixel comes into play.
Gigapixel is a stand-alone app from Topaz
Labs that uses AI to upscale your images.
Yes, images.
They've recently released another app focused
on videos but I find it slower to work with
than Gigapixel.
They also offer a trial version so let's head
over to their website and find Gigapixel download.
Run the installer and give it a few minutes
to download necessary data for that upscaling
AI engine.
So this is how Gigapixel looks.
It expects images so we need to head back
to Premiere and export our latest, colorized
version out as a PNG sequence that we can
import it into Gigapixel.
Back in Gigapixel click on open images, find
your image sequence folder, select all of
them and give it a few moments to load.
Now you can see the list of all input images
here, previews down here and settings are
over here.
You can immediately see what you're doing
by looking at the preview section which is
great for tweaking those settings.
As a first thing here we need to change the
output resolution.
You can see here that the input is 1080p and
output is a massive 8K, so let's change that
by going to Scale over here and clicking on
2x so our 1080p footage will be scaled twice
to create 4K.
If you are thinking about using 4x, which
in our case would be an 8K image in hope to
get more detail into the final result and
then scaling it back to 4K - I've done a few
tests and the difference is really minimal
so no point spending extra time on this.
Now switch these settings to Manual and play
with these two sliders while looking at your
preview section.
They help quite a bit but of course you don't
want to overdo it.
Especially this Remove Blur slider - it helps
to make your picture look sharper but it also
introduces imperfections to your image so
try to balance those two.
And finally, we should get rid of the Suffix
down here as this is an image sequence so
we don't want to be adding anything at the
end of our filenames - use prefix instead,
something like 4K underscore.
Hit start and depending on the performace
of your GPU this will take anything between
a few seconds to few minutes per frame.
What I like to do as a final step is bring
it into After Effects and do some final tweaks
like color correction, maybe play with that
crop a little bit etc.
Ta-da, we're done!
You've successfully used AI to upscale, colorize
and interpolate this old footage into beautiful
4K running at 60fps.
I hope you guys have learnt something cool
today and you see, starting with AI and video
is not that difficult.
Continue expanding your knowledge as AI is
definitely the future when it comes down to
work like this.
I'll definitely keep you in the loop with
new videos so let me know what kinda tutorial
you'd want to see.
And of course hit the big red subscribe button
below and the bell icon next to it so you
don't miss any of them.
Until next time my fellow creatives!
