What does HackerNews think of TurboBench?

No, No and No. It's algorithmically impossible that zstd is decompressing faster than lz4 by using the same environment.

You are comparing the zstd with asynchrounous I/O + multithreading against a single thread simple lz4 cli.

Additionally you're using some exotic and not representative hardware.

In theory, the only single thread decompression case where zstd can beats lz4 is when you have slow (reading) i/o.

lz4 & zstd are mostly used as libraries and real benchmarks are not considering I/O or multithreading.

This is what I'm getting with TurboBench & enwik9. lz4 decompressing 2,8x faster than zstd.

      C Size  ratio%     C MB/s     D MB/s   Name            
   356828015    35.7     400.71    1557.59   zstd 1 
   416874817    41.7     312.31    1967.66   lzav         
   509199084    50.9     630.78    4395.08   lz4 1           

Download TurboBench [1] from [2] and make the tests on your machine.

[1] - https://github.com/powturbo/TurboBench

[2] - https://github.com/powturbo/TurboBench/releases

I've made a benchmark with real html data including memory usage in [2] and a recent benchmark in [3] with TurboBench [1].

In the speedup plots you can see the best compressors for content providers:

- brotli 11 is best for static content

- brotli 5 is best until 1MB/s network transfer speed

- libdeflate 6 is best from 1MB/s to 6MB/s (followed by brotli,4)

- igzip 1,2 is best for very fast networks > 10MB/s

brotli brings little value at decompression for users

[1] https://github.com/powturbo/TurboBench

[1] https://sites.google.com/site/powturbo/home/web-compression

[2] https://encode.su/threads/2333-TurboBench-Back-to-the-future...

Build or download TurboBench [1] executables for linux and windows from releases [2] ans make your own tests comparing oodle,zstd and other compressors.

[1] https://github.com/powturbo/TurboBench

[2] https://github.com/powturbo/TurboBench/releases

There are nice plots [1] to see the tranfer+decompression speedup depening on the network bandwidth.

This is for html web compression, but the results are similar for other datasets. For internet transfer more compression is better than more decompression speed.

You can make your own experiments incl. the plots with turbobench [2]

[1] https://sites.google.com/site/powturbo/home/web-compression [2] https://github.com/powturbo/TurboBench

Already done. The claims are not based on speculations, but on numbers coming from verifiable benchmarks with "https://github.com/powturbo/TurboBench" including the Daala_EC and the AOM AV1 entropy coder.

> lz4's decompressor is simultaneously simple and also the fastest thing aroud, at least at the moment

The proprietary LzTurbo [0] and free Lizard [1] both claim to be faster at decompression while having better compression ratios.

[0] https://github.com/powturbo/TurboBench

[1] https://github.com/inikep/lizard

> Brotli's fastest compression is slightly faster than zstd's.

Come on, this is not serious.

Brotli's fastest compression algorithm is still significantly slower than zstd. And more importantly, it compresses _much worse_.

For a 3rd party evaluation, one can try [TurboBench](https://github.com/powturbo/TurboBench) or even [lzbench](https://github.com/inikep/lzbench) which are open-sourced. Squash introduces a wrapper layer with distortions which makes it less reliable, and more complex to use and install, quite a pity given the graphical presentation is very good. I'm interested in speed, and in this area, all benchmarks point in the same direction : for a given speed budget, Zstandard offers better ratio (and decompresses much faster).

Also lzturbo is very interesting, closed source but you one can download exec: https://sites.google.com/site/powturbo/ https://github.com/powturbo/TurboBench

hi Nathan, glad you want to benchmark TurboRLE on Haswell and Skylake. Well, it is very simple: 1- download the lkatest TurboRLE from github 2- cd TurboRLE 3- make 4- ./trle file

The enwik9bwt must be generated. The two other files can be directly downloaded. You must also download rle64 if you want to use test it.

The better option is to use TurboBench: https://github.com/powturbo/TurboBench