[Dev]XTool

[Razor12911]

Ok I decided to get Need For Speed The Run because I remembered that the game required a lot of decompression memory for srep to best demonstrate what I want to do with this deduplication idea.

I was surprised by the results

Xtool 2019:

Compressed 120 files, 14,503,034,446 => 29,120,915,487 bytes. Ratio 200.79%
Compression time: cpu 17.25 sec/real 807.42 sec = 2%. Speed 17.96 mB/s

100%: 29,120,917,667 -> 8,068,516,819: 27.71%.  Cpu 223 mb/s (124.672 sec), real 225 mb/s (123.527 sec) = 101%.  Remains 00:00
Decompression memory is 5660 mb.  2,922,729 matches = 46,763,664 bytes = 0.58% of file

Compressed 1 file, 8,068,559,139 => 4,315,048,356 bytes. Ratio 53.48%
Compression time: cpu 10.02 sec/real 172.58 sec = 6%. Speed 46.75 mB/s

Xtool 2020:

Compressed 120 files, 14,503,034,446 => 12,121,343,984 bytes. Ratio 83.58%
Compression time: cpu 16.13 sec/real 368.86 sec = 4%. Speed 39.32 mB/s

100%: 12,121,346,164 -> 8,010,462,480: 66.09%.  Cpu 141 mb/s (81.719 sec), real 145 mb/s (79.890 sec) = 102%.  Remains 00:00
Decompression memory is 759 mb.  2,105,722 matches = 33,691,552 bytes = 0.42% of file

Compressed 1 file, 8,010,469,968 => 4,296,282,176 bytes. Ratio 53.63%
Compression time: cpu 9.88 sec/real 159.45 sec = 6%. Speed 50.24 mB/s

Overall test results (compression):
13.5 GB (original) > 27.1 GB (xtool 2019 / 13 mins 27 secs) > 7.51 GB (srep m3f / 5660mb dec memory) > 4.02 GB (lzturbo 32 b128)
13.5 GB (original) > 11.2 GB (xtool 2020 / 6 mins 9 secs) > 7.45 GB (srep m3f / 759mb dec memory) > 4.00 GB (lzturbo 32 b128)

Not only is xtool 2020 faster but it produced a slightly better result while reducing srep's decompression memory. That 13.5 GB > 11.2 GB is not a mistake, xtool did precompress the data but there were too many repeated streams in the input and it removed a lot of them which resulted in negative ratio.

Notes: Xtool does incur additional decoding memory but it will definitely not be 5.6GB, it will far less. Maybe 2GB but if I apply proper memory management, I can reduce it to 1GB saving an additional 3GB of ram for lolz or something else.