Pseudorandom Number Sequence Test + LZ Bench

[felice2011]

( Overview to Entropy of a File )

Simplistically entropy is disorder, or better in computing the density of information that a data stream can contain.
So the more the content of a file will be predictable, the higher the entropy.

The software ENT (Pseudorandom Number Sequence Test Program, http://www.fourmilab.ch/random)
performs various statistical tests, providing output in the following report:

1) Entropy

is the density of information contained in a file expressed in number of bits per character. The maximum entropy is 8, when we find a file with entropy or 8 means that it is perfectly random, or is compressed.
In fact, taken a bitmap, its entropy is 4.724502 bits per byte, if you turned it into jpeg becomes 7.938038 bits per byte. If you compress the bmp with winrar I get 7.996259 bits per byte. This is clear.
I take a text file where I have written a thousand times the same word we get is that its entropy Entropy = 2.545246 bits per byte. If we compress with winrar we get Entropy = 6.747827, and winrar maximum compression Entropy = 6.756800.

2) Test of Chi square

Used for the study of random data streams. If we apply it to image files as a result they are random data.
In practice, it occupies the deviation percentage of the flow of data from a real random sequence.
However, if the result is> 99% or <1% of the data stream is not random. If it is between <5% and> 95% of the flow is random suspiciously, if intermediate then we are on random.

3) Arithmetic Mean

Sum all the bytes and divides them for the length: it is a type of arithmetic mean. The closer the number is 127.5 more random.

4) Test of Pi-Greco Montecarlo

The more the value is close to pi-greco (3.14 ..) plus the data stream is random / compressed.

5) Coefficient of Correlation

Ie how predictable a byte knowing his previous. More the value is close to 1 and more is predictable, more and more close to 0 is random.

SOME EXAMPLES ON THE IMAGES :

We analyze a bitmap file ...

Entropy = 4.724502 bits per byte.
Optimum compression would reduce the size
of this 5443254 byte file by 40 percent.

Chi square distribution for 5443254 samples is 373809870.53, and randomly
would exceed this value less than 0.01 percent of the times.

So definitely not random ...

Arithmetic mean value of data bytes is 160.4207 (127.5 = random).
Monte Carlo value for Pi is 1.912926349 (error 39.11 percent).

So far from pi greco

Serial correlation coefficient is 0.818560 (totally uncorrelated = 0.0).

Note that the file is not full of information because the maximum entropy is 8 is in fact the information is compressible.
The chi-square gives us a value of 0.01 it says that the flow is not accidental but it is a picture is not reliable.
The average is 160 and deviates from 125 and is therefore not random. Even the monte carlo is far from 3.14.
The correlation is 0.81: bitmaps are always close correlation to 1. If they were random data would be 0.

Now let's look at the same bitmap converted to jpg ...

Entropy = 7.938038 bits per byte.

Chi square distribution for 231584 samples is 27771.22, and randomly
would exceed this value less than 0.01 percent of the times.(non random?)

Arithmetic mean value of data bytes is 122.7712 (127.5 = random).
Monte Carlo value for Pi is 3.215483069 (error 2.35 percent).
Serial correlation coefficient is 0.039713 (totally uncorrelated = 0.0).

Also here very close to 0, identifies it as random.

Here the entropy is very high (7.9), the file is very compressed.
Test Monte Carlo 3.21, quite close to the pi greek, so close to random.
Correlation coefficient close to 0. understand that it is compressed.







After this short overview this GUI natively leverages the application ENT, to calculate the entropy of a file, or an full data folder, providing a report based on the reduction of the file or folder data, and its total compression ratio, for know quickly if a file type and / or folder, will have a high or low compression.

Classification of the file or group of files:

The scanning of the file or folder data, it's divided into 5 blocks with a calculation of the entropy range from 1.0 to 7.0 for Deflate e Text, and 1.0 to 7.5 for Void and Msrsolid, through direct reading of the file arc.group during the scanning, and based on the reading of the extensions of the 4 masks "Void, deflate, Msrsolid and Text" and the basic method.

The files with higher entropy than 7.0 or 7.5 or an extension not set in arc.group file are classified and added to the basic method.

I chose the level 7 and 7.5 on the basis of various tests performed out on individual files of various formats, a file with the entropy level from 7.0 to 7.5, with a strong compression carried out with different samples of compressors you get a reduction of 20-25% and a compression ratio of 75-80%.

Each block contains additional information, according to the main method and masks, number of files scanned and belonging, percentage of size reduction, percentage of compression ratio, total size of files added in the belonging block.



Creating a masked method estimate in based on the entropy and scanning of files:

With a choice of 44 compressors on 4 masks, these will be activated or deactivated, in based on the scan and the entropy of the previously evaluated files, in order to speed up and simplify the creation of the final masked method with a correct of compression estimated of 90% on the compressed files.



LZbench v1.7.1 by Inikep : " Benchmark Compressors LZ77/LZSS/LZMA " https://github.com/inikep/lzbench

Thank Inikep from encode.ru, the application incorporates a modified and adapted for the complete benchmark on a single file or entire directory of 63 compressors of the Family LZ77/LZSS/LZMA, with a full report out for each file (compression speed in Mb/s, decompression speed in Mb/s, original size, compressed size, ratio and file name).
In the same way we will have a final scan with the reduction in size and compression ratio, to compare the various compressors and choose based to the speed of compression, decompression and ratio, on the types of scanned files.

In LZBench no size file limit, even using a low amount of memory the average of the ratio is calculated for the number of divided parts obtaining the overall result of the compression ratio.

blosclz 2015-11-10 [1-9]
brieflz 1.1.0
brotli 2017-03-10 [0-11]
brotli22 2017-03-10 [0-11]
brotli24 2017-03-10 [0-11]
crush 1.0 [0-2]
csc 2016-10-13 [1-5]
density 0.12.5 beta [1-3]
fastlz 0.1 [1-2]
gipfeli 2016-07-13
glza 0.8
libdeflate 0.7 [1-12]
lz4 1.7.5
lz4fast 1.7.5 [1-99]
lz4hc 1.7.5 [1-12]
lizard 1.0 [10-49]
lzf 3.6 [0-1]
lzfse 2017-03-08
lzg 1.0.8 [1-9]
lzham 1.0 -d26 [0-4]
lzham22 1.0 [0-4]
lzham24 1.0 [0-4]
lzjb 2010
lzlib 1.8 [0-9]
lzma 16.04 [0-9]
lzmat 1.01
lzo1 2.09
lzo1a 2.09
lzo1b 2.09
lzo1c 2.09
lzo1f 2.09
lzo1x 2.09
lzo1y 2.09
lzo1z 2.09
lzo2a 2.09
lzrw 15-Jul-1991 [1-5]
lzsse2 2016-05-14 [0-17]
lzsse4 2016-05-14 [0-17]
lzsse4fast 2016-05-14
lzsse8 2016-05-14 [0-17]
lzsse8fast 2016-05-14
lzvn 2017-03-08
pithy 2011-12-24 [0-9]
quicklz 1.5.0 [1-3]
shrinker 0.1
slz_deflate 1.0.0 [1-3]
slz_gzip 1.0.0 [1-3]
slz_zlib 1.0.0 [1-3]
snappy 1.1.4
tornado 0.6a [1-16]
ucl_nrv2b 1.03 [1-9]
ucl_nrv2d 1.03 [1-9]
ucl_nrv2e 1.03 [1-9]
wflz 2015-09-16
xpack 2016-06-02 [1-9]
xz 5.2.3 [0-9]
yalz77 2015-09-19 [1-12]
yappy 2014-03-22 [0-99]
zlib 1.2.11 [1-9]
zling 2016-01-10 [0-4]
zstd 1.1.4 [1-22]
zstd22 1.1.4 [1-22]
zstd24 1.1.4 [1-22]

NOTE : (The entropy and classification of the files and the estimated masked methods, is not enabled for the moment with the use of LZ-BENCH).



"arc.groups" updated to version 3.0, based on version 2.5 of Panker1992, they were added over 200 popular formats used in the area of gaming.

We Avoid the Vultures, those who not give credit and thanks for all the work, please do not use the application and not download...

UPDATED : BE_Parent_Dir

The file parent directory is displayed in the masks box.
Other minor fix.

In Down.

[EzzEldin16]

can i ask you something i know it's not about this program but i was wondering is it possible to add files after compressed with specific compressors into inno script like WPI
exe so it can work as self extracting archive ?? so no archive needed seprately ?

[felice2011]

@EzzEldin16 if you want to ask me some questions not relative to the thread or the program, please contact me in private, or open a new thread of help.

[ChronoCross]

Excellent Job!!.
But I do not know if all users know how to get the maximum use of this great utility.

[EzzEldin16]

@felice2011 okay i'll message you

[TWOELV]

i don't get any of this at all but i am pretty sure its a really helpful tool so great work

[felice2011]

Originally Posted by ChronoCross

Excellent Job!!.
But I do not know if all users know how to get the maximum use of this great utility.

Originally Posted by TWOELV

i don't get any of this at all but i am pretty sure its a really helpful tool so great work

I do not blame those with a low level knowledge of compression or other to know the potential of this program, but for other people here on the forum, with a level of knowledge in some fields more of my, it seems that my work not interest and has no credits. But the nice thing that just the latter, more than once have complained it had no credit, nor interest, of their work posted, cataloging the majority of users like vultures, but did you ever wonder who are the real vultures?
I had great ideas for this application, with new updates, but if results and interest are low, do not update my works here on the forum.

[felice2011]

@PrinceGupta2000 ... For the simple reason, that to get a good and right compression is necessary to know the structure of a file, and in this case its entropy.
I give you an example, the file (RE7) "re_chunk_000.pak.patch_001.pak" from 2.5GB get a compression ratio of 87.46% scanned in 30 seconds.
Now I know that giving an estimate of 7-8% scrap, should get the maximum with any compressor or precompressor existing today in a method or mask, a decrease in the ratio of 7-8%, in fact, with such a method "pzlib+srep64+delta+exe2+4x4:lzma" get the 79.6 %, the 8% of less.
Also I can know in advance the estimated compression ratio of any file in seconds or minutes, without having to test thirty compressors, with their methods or masks, with having to compress for half a day using continuous writes on HD and occupying space useless.
Let's see what altro..ahhh ..
Ability to create your own mask, so right and correct, but I think, that you make little use of masks, to understand the real need to use a mask is a pipe dream for many of you.
Integra the possibility to verify that I have said in the first lines in the example, with LZ-Bench, without the use of space and writing in HD, in fact with the same file get a compression ratio of the 84.31% with lzma test a level 6.
What else you had to do even coffee.

EDIT: Prince .. what do you do, first ask the questions and then they delete.
Same thing my reply goes for who else is interested.

[Gupta]

the file (RE7) "re_chunk_000.pak.patch_001.pak" actually contains about 3000 RIFF waves about 1 GB, feed them to Ogg and u saved about 800 MB
a little bit of hard work and gradual increase in compression ratio without going into many details
you can save more retro porns in those 800 MB
masks never works for me another pak in the game actually have a lot more RIFF waves and some biks so if u r a nerd like me you can save more space
and knowing only the entropy will never save this much space for retro porns

>>without having to test thirty compressors
I never test compressor, Lzma is the best

>>Prince .. what do you do, first ask the questions and then they delete
Shhhhhhhhhh. Someone is Watching

>>pzlib+srep64+delta+exe2+4x4:lzma
never use srep64 and exe2 after srep or rep compressed data, this will decrease the overall compression ratio,
if really want delta and exe2 use them before this is increase overall speed with least decrease in compression ratio

[felice2011]

Prince, I trouble understanding what you say, between porn and entropy, the method mentioned was just an example to make you understand the concept, but you can use any other method or compressor, put first, remove or add "srep, rep, lzma with or without 4x4 etc.." you'll get to max than 10% less than the estimated compression ratio on the entropy of a file.
Or you can make magic with a compressor as "LZAF" Shhhh ... ehhh Oh Yes, someone has already read what you write.

[rinaldo]

ci credo che hai difficoltà a capirlo ahah.. dagli una leccalecca e ti scodinzolerà
very complicated pg

[gozarck]

Nice work felice, optimum software for large file batches. You should take a look at datasmoke by Bulat Ziganshin

[romulus_ut3]

An option to save the file names including categories and their full path including sub-directories would be nice.

[felice2011]

Originally Posted by gozarck

Nice work felice, optimum software for large file batches. You should take a look at datasmoke by Bulat Ziganshin

Excellent Idea Bro .. compile the source and testing..

EDIT:





I think something good comes out ....

[romulus_ut3]

Hello, Felice.

Firstly, I'd like to take this opportunity to thank you for this brilliant tool!

I would like to request a simple feature. (I know you're busy with Datasmoke)

Is it possible to list files category wise with their file path, and save the results once a directory has been scanned?

The ratio for the compression isn't necessary to be included.

For example:

Method:
Game\files\xxxxxxxxx.xx
Game\files\xxxxxxxxx1.xx

Void:

Game\Movies\xxxxxxxxx.bik
Game\Movies\xxxxxxxxx1.bik

Deflate:

Game\Sound\xxxxxxxxx.ogg
Game\Sound\xxxxxxxxx1.ogg

MSRSolid:

Game\Help\xxxxxxxxx.jpg
Game\Help\xxxxxxxxx1.jpg

Text:

Game\Docs\xxxxxxxxx.txt
Game\Docs\xxxxxxxxx1.ini

This'll hugely speed up workflow for some of us!