• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Real Video Upscaling using AI!

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
Recently I saw on a website, details about a home software upscaling of StarTrek Deep Space 9, by a fan, and the results were very good.


That got me intrigued!
I did not think it was possible, not only to upscale, but to create new detail and pixels using AI, that would look believable. Just go the link and see some results.
So I decided to get my own feet wet, and try some, It bloody works!
See this: (starts to download)
- Bill Withers original BBC.
- Same as above, but upsampled.
- Forbidden Planet 1956 Original HD on YT.
- Same as above, but upsampled. (@4:30 you can compare)
The last clip blew me away!
On my humble system, it takes about 50X the original footage in seconds, to upscale and encode. So it is a slow process, but results can be breath taking.
I used, Topaz Video Enhance AI.
Does anyone have any deeper knowledge, suggestions etc. on this subject?
I love to learn.
 
Last edited:

Curvature

Major Contributor
Joined
May 20, 2022
Messages
1,074
Likes
1,353
To my eye, the movements look more agitated, not necessarily better. I'm sure good work can be done with upscaling, with enough manual intervention and expertise. For archival purposes I can see these tools causing real problems for keeping track of originals, and elsewhere improving the recoverability of info from partially destroyed media.

Some degree of processing is already built into modern TVs, so I wonder how different these tools really are, or how to understand that.
 
OP
Ken Tajalli

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
To my eye, the movements look more agitated, not necessarily better.
Which video clip?
On the music video clip, his afro looks odd, I agree.
But the "forbidden planet" clip the improvements are phenomenal !
Besides, these are my first attempt, on a home system.
I have now realized, I can apply filters only, upscaling is not a must.
I'm sure good work can be done with upscaling, with enough manual intervention and expertise. For archival purposes I can see these tools causing real problems for keeping track of originals, and elsewhere improving the recoverability of info from partially destroyed media.
With many containers (mp4, mkv etc.) the original video can be included as a secondary stream.
Some degree of processing is already built into modern TVs, so I wonder how different these tools really are, or how to understand that.
I have the top of the range, latest Samsung Qled TV.
There is no comparison .
 
OP
Ken Tajalli

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
The forbidden planet demo is very impressive.
Play it on a good TV. computer or laptop monitor doesn't do it justice.
For fun, I ran 2001 a space .. through the filter.
Original was 4k HDR, so pretty good! (85GB good). on the enhanced clip, I could read the
"zero gravity toile instructions" for first time! well some of it. :)
 
OP
Ken Tajalli

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
A YouTube Channel called Stick Hits has been doing something similar to some vintage RUSH videos.

Interesting!
I have an old Camel concert in DVD quality. I am gonna do the same.
watch this space.
 
OP
Ken Tajalli

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
On a side note:
I use an AMD rt 5700 xt graphic card to do the above.
Since the process is very GPU intensive, my card started to over heat, see this:

Screenshot 2023-06-06 140124.jpg


Junction temp. was spiking to over 100 degrees, with fan going at 3300 rpm! it crashed a few times.
So I ripped it open and re-did the paste and pad replacement, properly, with great care.
Now see this:
1686241195838.png


Junction temp. is down to 70 degrees, fan at less than 2000rpm.
How could factory assemble this card so badly??
Shame on you AMD!
 
OP
Ken Tajalli

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
A YouTube Channel called Stick Hits has been doing something similar to some vintage RUSH videos.

here is an enhanced clip from one of my favourute old concerts, perhaps because I was there!
My firefox, only plays audio on the links, but Chrome plays it correctly.
here are the clips:
- Before (direct from DVD original)
- After (enhanced and upscaled)
look for artefacts, noise and motion blur on original, compared to After.
I had to recompress the clips, to put them on the web, so a little picture quality is lost on both, but I used the same engine.
Enjoy.

BTW
others had a go too! see for yourselves:
- Attempt one
- Attempt two

cover_523251862017_r.jpg
 
Last edited:

fpitas

Master Contributor
Forum Donor
Joined
Jul 7, 2022
Messages
9,885
Likes
14,191
Location
Northern Virginia, USA
A YouTube Channel called Stick Hits has been doing something similar to some vintage RUSH videos.

I imagine Coleridge would be mystified. But flattered, nonetheless.
 
OP
Ken Tajalli

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
Update:
I have been doing a number of classic movies.
So far I have done more 40.
The Ai upscales, range from breathtaking to good improvement compared to originals.
I have also figured out how to tone-map to HDR, though I am still practising.
If anyone is interested in having a copy to check my work (;)), PM me.
My recent work, on "Once upon a time in America" which I upscaled to 4K HDR looks fantastic on my screens. For the first time, I can truly see what is going on in the movie.

Few screen captures to whet your appetites :
Click for largest images allowed on ASR.

Once upon a time in America HDR.mkv_snapshot_01.45.11.305.jpg

Once upon a time in America HDR.mkv_snapshot_00.07.42.963.jpg



Amadeus 1984 (2001) DC 4K-Ai.mkv_snapshot_01.51.19.923.jpg

Amadeus 1984 (2001) DC 4K-Ai.mkv_snapshot_00.25.35.617.jpg



The Hateful Eight 2015 HDR 4K.Ai.mkv_snapshot_00.05.07.975.jpg

The Hateful Eight 2015 HDR 4K.Ai.mkv_snapshot_00.15.23.755.jpg


The big country (1958) 4K-Ai.mkv_snapshot_02.32.09.268.jpg

The big country (1958) 4K-Ai.mkv_snapshot_00.38.03.531.jpg
 

napfkuchen

Active Member
Joined
Mar 9, 2022
Messages
295
Likes
397
Location
Germany
Looks promising. Imagine applying this in real-time, similar to DLSS or FSR, to any source material. Maybe especially dark scenes on Netflix 4K won't remind me of a DivX cd-rip from 2000 anymore.
 
OP
Ken Tajalli

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
Looks promising. Imagine applying this in real-time, similar to DLSS or FSR, to any source material. Maybe especially dark scenes on Netflix 4K won't remind me of a DivX cd-rip from 2000 anymore.
Well there is such a thing, well almost.
My Samsung TV (TOTR) manages to enhance contrast and colour and upscale on fly.
If you have heard of madVR, they are working with hardware people, and there is a system out already.
But Ai upscaling and enhancement is really processor intensive.
I have a PC with i7 13th get and an NVidia RX 3060ti , 32GB RAM.
At full load, upscaling 1080p to 4K happens at 3 to 5 frames a second! depending on video input and filters applied.
Tone mapping to HDR afterwards & recompression is done at 8 frames a second.
But the results are worth it.
I did Alien 1978 .
Then an official 4K HDR was released after much fanfare.
My version compared to official, has less noise, is sharper and resolution and HDR effect on a par.
 

napfkuchen

Active Member
Joined
Mar 9, 2022
Messages
295
Likes
397
Location
Germany
But Ai upscaling and enhancement is really processor intensive.
I have a PC with i7 13th get and an NVidia RX 3060ti , 32GB RAM.
At full load, upscaling 1080p to 4K happens at 3 to 5 frames a second! depending on video input and filters applied.
Tone mapping to HDR afterwards & recompression is done at 8 frames a second.
Nvidia RTX 4000 series has almost doubled power efficiency compared to the previous generation. We'll get there, eventually, if software is also improved and tv manufacturers stop going cheap on chips. :)
Development is rapid:
 
Last edited:
OP
Ken Tajalli

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
Nvidia RTX 4000 series has almost doubled power efficiency compared to the previous generation. We'll get there, eventually, if software is also improved and tv manufacturers stop going cheap on chips. :)
At the moment, possibly NVidia A100 could do it in real time.
none of the RTX series can manage real time.
But Yes, we will get there.
I just results today! also it is a hobby.
 

riker1384

Member
Joined
Jul 19, 2019
Messages
66
Likes
94
I was under the impression that the DS9 set footage was shot on film so AI upscaling of that wouldn't be necessary, just re-scanning the film. The issue is the special effects and space scenes were done in SD, and the 3D spaceship assets were lost. It seems to me that if they don't have the budget for a full remaster like they did with TNG, they should be able to compromise, and rescan the film of the actors on set, but use AI upscaling for the space scenes.

I'm not sure what they would do with shots where the actors are interacting with special effects. Maybe AI upscaling for that, but then the picture quality would change from shot to shot which would be distracting and break immersion. Maybe upscale the SD finished version, and superimpose the upscaled special effects on top of the re-scanned film footage? Or maybe recreate the special effects from scratch with modern CGI and such? I don't know.
 
OP
Ken Tajalli

Ken Tajalli

Major Contributor
Forum Donor
Joined
Sep 8, 2021
Messages
1,996
Likes
1,773
Location
London UK
I was under the impression that the DS9 set footage was shot on film so AI upscaling of that wouldn't be necessary, just re-scanning the film. The issue is the special effects and space scenes were done in SD, and the 3D spaceship assets were lost. It seems to me that if they don't have the budget for a full remaster like they did with TNG, they should be able to compromise, and rescan the film of the actors on set, but use AI upscaling for the space scenes.

I'm not sure what they would do with shots where the actors are interacting with special effects. Maybe AI upscaling for that, but then the picture quality would change from shot to shot which would be distracting and break immersion. Maybe upscale the SD finished version, and superimpose the upscaled special effects on top of the re-scanned film footage? Or maybe recreate the special effects from scratch with modern CGI and such? I don't know.
That's interesting.
Regarding DS9, which in my opinion had the best plot line, upscaling FX is actually much easier that organic scenes.
Ai can do a wonderful job of FX, as it assumes there are no random fine details.
Organic scenes such as faces, is the hardest. Human eye is sensitive to faces! filter just a tad too much and you get wax-works.
Also we are so accustomed to noise and film-grain, that we desire it. On my software, I can add grain if I want to. Some prefer it.
 
Top Bottom