What does HackerNews think of waifu2x?

https://github.com/nagadomi/waifu2x

https://github.com/nagadomi/waifu2x

And other implementations thereof, e.g:

https://github.com/nihui/waifu2x-ncnn-vulkan

I am not aware of any singular FOSS project in creative work that performs similarly to Topaz Labs’ product for all input videos (note to reader: if I am wrong, correct me— I only have second-hand anecdotes to go off of from my friends who work with this kind of work in their careers, and precious little first-hand experience from upscaling old family memories to experiment with the software). As far as I am aware, this is because different upscaling models are trained on, and thus effective / mainly used for a specific type of content. The same upscaling model that was trained on interpolated 480p videos with compression artifacts will not produce the same results as one trained on, say, anime videos/manga, e.g. the models used by Waifu2X [1]. Hence, with Topaz Labs’ application, you select the model that was trained on footage that best matches the footage you wish to upscale.

All that being said, I do know that some (if not all) of the upscaling models Topaz uses are FOSS. Much of their application is just syntactical sugar upon the models it uses, making it easier for non-SWEs to use. I’m not sure whether or not the models that Topaz distributes have any level of training done by them, in house— logically, I would assume as such, otherwise their product wouldn’t be as performant as it is.

[0] https://www.topazlabs.com/video-enhance-ai

[1] https://github.com/nagadomi/waifu2x

See e.g., https://www.topazlabs.com and https://github.com/nagadomi/waifu2x

Online demo: http://waifu2x.udp.jp/

Repo: https://github.com/nagadomi/waifu2x

Reddit discussion: https://www.reddit.com/r/VRchat/comments/bkw543/tutorial_how...

There are three benefits that one could get from reducing the file size:

1. Reduced storage cost

2. Reduced bandwidth cost

3. Better user experience

In my models, storage cost matters a lot. You can't come out ahead here, however, if you still have to keep JPEG copies of all the images.

Benefits in terms of 2 are real.

Benefits in terms of 3 are hard to realize. Part of it is that adding any more parts to the system will cause problems for somebody somewhere. For instance, you can decompress a new image format with a Javascript polyfill, but is download+decompress really going to be faster for all users?

Another problem is that much of the source material is already overcompressed JPEG so simply recompressing it with another format doesn't lead to a really improved experience. When I've done my own trials, and when I've looked closely at other people's trials, I don't see a revolutionary improvement.

A scenario that I am interested in now is making desktop backgrounds from (often overcompressed) photos I find on the web. In these cases, JPEG artifacts look like hell when images are blown up, particularly when images have the sharp-cornered bokeh that you get when people take pictures with the kit lens. In that case I can accept a slow and expensive process to blow the image up and make a PNG, something like

https://www.mathworks.com/help/images/jpeg-image-deblocking-...

or

https://github.com/nagadomi/waifu2x

The other approach I imagine is some kind of maximum entropy approach that minimizes the blocking artifacts.

Edit: Actually looking at the source further it appears they also have models for photos as part of that project in addition to the model for anime.

[1] https://github.com/nagadomi/waifu2x

[1]: https://github.com/nagadomi/waifu2x

[2]: https://github.com/zachsaw/MPDN_Extensions

But if they wanted they could probably try to train their neural network for pixel art specifically, similar to https://github.com/nagadomi/waifu2x

They won't tell you it's NNs but it is. Sony distributes a lot of movies, they use their database of movies to train the upscaling models (which is obviously NNs https://github.com/nagadomi/waifu2x ) and then put the chip in the TV.

It's something almost equivalent to storing Pride and Prejudice and Zombies in your TV in 4K, and then reproducing it when they match it with what's playing on TV.