What does HackerNews think of lepton?
Lepton is a tool and file format for losslessly compressing JPEGs by an average of 22%.
https://github.com/dropbox/lepton
The goals and results appear similar. Is the primary difference that brunsli will likely actually be part of a standard (JPEG XL)?
We're working on a paper for submission to a research conference soon, so every viewer gives us helpful data for the system to learn in-situ and improve its performance over different kinds of Internet connections and congestion situations. (We are randomizing connections to get either our in-development algorithm or a bunch of competing ABR and congestion-control schemes so we can properly evaluate, so if you get a stall or bad quality...... we'll blame that.)
All the source code is at https://github.com/StanfordSNR/puffer and is or will be released with an open-source license. The study is led by my doctoral student Francis Yan, along with Sadjad Fouladi, Hudson Ayers, Chenzhi Zhu, and my colleague Philip Levis. The goal is to train video-streaming algorithms (bitrate selection and congestion control) online, continually, in situ, to minimize stalls and deliver the best picture quality across a diverse range of real-world Internet connections and users. This is from some of the same people who brought you Salsify (functional video compression for lower-latency real-time video: https://snr.stanford.edu/salsify), Lepton (distributed/parallel JPEG compression: https://github.com/dropbox/lepton), Mosh (functional terminal emulation for mobility: https://mosh.org), Remy (machine learning for congestion control; http://mit.edu/remy), and the Pantheon of Congestion Control (https://pantheon.stanford.edu).
There are some unusual details of the Puffer system to make experimentation easier (and, we hope, performance better): the ABR algorithm is server-side, and video is streamed continuously over a WebSocket and the client gives asynchronous feedback, instead of using DASH HTTP request/replies. The tcp_info struct (including the delivery rate estimate) from the congestion-control layer is plumbed through to the ABR, and to the neural network that tries to predict "how long will it take to send a video chunk of length y," so it's weakly cross-layer. The "transmission-time predictor" is probabilistic instead of simply producing a point estimate. We encode with libx264 and then measure the SSIM of every encoded chunk to calculate a reward for each possible stream (instead of making the assumption that more bitrate is better -- the correlation is not so good when you consider different times in the same video). And we encode 10 variants of each channel, so more than typical. And everything is done at 60 fps. Of course some of these details mean these algorithms are not going to be deployed any time soon on a production CDN or streaming service, but the hope is to demonstrate the value of some of the ideas in an academic setting for future commercial deployment. The use of WebSockets, MSE, and Opus is why this doesn't work on Safari or on iOS.
FAQ here: https://puffer.stanford.edu/faq/
Also I'm happy we were able to make some of the Grafana monitoring public: https://puffer.stanford.edu/monitoring
b) then compress it with zlib -6 to get an idea of how compressible the data is.
c) if the data compresses by at least a percent with zlib, it's likely to do significantly better with more advanced compression like DivANS. About 1/3 of files in Dropbox fall into that >1% category but aren't compatible with Lepton. For those compressible files, DivANS gets 12.08% savings over zlib with the settings we chose in the blog post. Brotli, in contrast, gets 9.64% savings over zlib on that same data.
If a browser natively supported Lepton, you could serve the compressed version over the wire, and still get the bit-exact original .jpg out at the end, if the user wanted to save the file.
Of course, browser support doesn't exist yet. But browser implementors, consider the gains. This isn't promoting an incompatible format, it's getting bandwidth savings while being fully backwards compatible.
[1] https://blogs.dropbox.com/tech/2016/07/lepton-image-compress... [2] https://github.com/dropbox/lepton
* Implemented in C++ (-std=c++0x and -std=c++11 work, -std=c++98 doesn't work).
* Needs a recent g++ to compile (g++-4.8 works, g++-4.4 doesn't work).
* Runs on Linux and Windows.
* Runs on i386 (-m32) and amd64 (-m64) architectures. Doesn't work on other architectures, because it uses SSE4.1 instructions.
* Can be compiled without autotools (http://ptspts.blogspot.ch/2016/07/how-to-compile-lepton-jpeg...).
Given the recent usage of Rust for the implementation of Brotli compression (https://blogs.dropbox.com/tech/2016/06/lossless-compression-...) and that it's used for data storage (http://www.wired.com/2016/03/epic-story-dropboxs-exodus-amaz...) this somewhat surprises me.
The reasons given for Rust as stated in https://blogs.dropbox.com/tech/2016/06/lossless-compression-... would seem valid here too: > For Dropbox, any decompressor must exhibit three properties: > > 1. it must be safe and secure, even against bytes crafted by modified or hostile clients, > 2. it must be deterministic—the same bytes must result in the same output, > 3. it must be fast.