[Dev]XTool

[Mini]

usa_world.dat
Overall size: 304237021 >> 824626021

works!

[Razor12911]

I figured out what they used with the spare break I had, they used the Hydra codec if I'm not mistaken. I didn't add Hydra support in xtool but I will in future.

[Razor12911]

Originally Posted by Mini

usa_world.dat
Overall size: 304237021 >> 824626021

works!

Could you rerun test, I want to see if correct and improved information is found. I played around with the scanner a bit.

@everyone
Participate if possible, try testing this with several games that use oodle, leave out Project CARS 2 out for now since that game is encrypted.

[Mini]

usa_world.dat
Overall size: 314783065 >> 844382444

[Edison007]

Originally Posted by Mini

The Crew 2 - Open Beta
xtool not working

files https://www64.zippyshare.com/v/dbhy7vY9/file.html

I had time, and i added support for this files.
Also I added parsing FAT from watch_dogs, but without (de)compression yet.
It takes time to deal with the xcompress-library.

[ZakirAhmad]

@Razor: is xtool going to b updated.

[Razor12911]

Hi guys, long time

I haven't coded in a while so you can imagine:
96806729_3811479972226955_3580484423860092928_n.jpg

I came up with an idea before I went hiatus which was deduplication of streams in xtool. Basically xtool permutes or disregards then removes repeated streams to improve performance. It's the same stuff that srep does only, xtool does it before srep so it improves IO and lessens memory usage.

What I want you guys to do is check out if this thing improves ratio if any and if decompression memory of srep goes down significantly, example. For examples like GTAV and lastly if there are no issues, I did rework the source a bit to make it stable.

My test input is No Man's Sky with the Beyond update (seems like devs have done a good job making sure they don't repeat game resources, wasn't the results I was not expecting...)

xtool 2019:
Compressed 104 files, 9,610,947,737 => 22,079,473,953 bytes. Ratio 229.73%
Compression time: cpu 12.20 sec/real 559.10 sec = 2%. Speed 17.19 mB/s

srep393b:
100%: 22,079,475,924 -> 14,306,151,312: 64.79%.  Cpu 123 mb/s (171.078 sec), real 126 mb/s (167.671 sec) = 102%.  Remains 00:00
Decompression memory is 699 mb.  6,413,440 matches = 102,615,040 bytes = 0.72% of file

lzturbo:
Compressed 1 file, 14,306,163,456 => 7,573,816,911 bytes. Ratio 52.94%
Compression time: cpu 16.19 sec/real 255.92 sec = 6%. Speed 55.90 mB/s

xtool 2020:
Compressed 104 files, 9,610,947,737 => 18,987,442,809 bytes. Ratio 197.56%
Compression time: cpu 10.94 sec/real 483.97 sec = 2%. Speed 19.86 mB/s

srep393b:
100%: 18,987,444,781 -> 14,276,551,933: 75.19%.  Cpu 113 mb/s (160.000 sec), real 114 mb/s (158.234 sec) = 101%.  Remains 00:00
Decompression memory is 436 mb.  4,696,348 matches = 75,141,568 bytes = 0.53% of file

lzturbo:
Compressed 1 file, 14,276,560,829 => 7,565,632,525 bytes. Ratio 52.99%
Compression time: cpu 16.86 sec/real 250.04 sec = 7%. Speed 57.10 mB/s

What do these results mean? and why is xtool 2020 producing less output compared to 2019 but somehow after using srep, the output is better?

Well as usual, I always have ideas to try to improve the xtool project for even more speed. I reviewed the project from all possible angles and found that optimisations of the program are nearing its limits meaning I cannot make the program any faster.

However then came an idea to steal some of the work srep does and make xtool involved, this resulted in the idea of making xtool detect and permute repeated streams, as a result. 2020 xtool most of the time will produce less output but when you compare it to 2019's version after using srep, the output is roughly the same or better.

You might ask, what is the point of taking some of srep's work and making it for xtool. Well this is because games over the years get endless updates, example is No Man's Sky and GTAV so there is a high chance that there are several repeated streams and as a result, if xtool doesn't take the job, it will be a job for xtool to deal with when installing a game because srep will require more decompression memory and xtool will process more streams resulting in extended installation times.

Note: I was expecting srep to produce the same result from both 2019 and 2020, because the main idea here is to improve installation times.

TL;DR

Xtool 2020 will produce less output compared to xtool 2019 because it takes some of the work of srep to reduce memory usage when installing a game while also improving speed significantly due to less streams being processed and less IO. (Theoretically at least)

Overall test results (compression):
8.95 GB (original) > 20.5 GB (xtool 2019 / 9 mins 19 secs) > 13.3 GB (srep m3f / 699mb dec memory) > 7.05 GB (lzturbo 32 b128)
8.95 GB (original) > 17.6 GB (xtool 2020 / 8 mins 4 secs) > 13.2 GB (srep m3f / 436mb dec memory) > 7.04 GB (lzturbo 32 b128)

I used "lzturbo 32" to speed up the final compression results (I'm a bit busy), but you will get the idea when you run tests yourselves.

Decompression mode is unavailable, I still have to write the code for it.

If you don't have GTAV or any other game which has live updates or a game with a lot of repetitions, you can just make a copy of the same input then give it to xtool to see what happens.

These optimisations are making me feel like Intel right now... 14nm+++++++++++++++++++++++

[Cesar82]

@Razor12911
Thanks for the great job.
After completing this if you have time (also if possible), do a functional calback for your cls-diskspan library.
There are also some bugs with this library, using compressions formed by 3 methods.
Thank you!

[doofoo24]

so i guess only game with zlib for now maybe game like mad max will see improvement last time i compressed the game it was with xtool+srep:m2f+lolz memory usage was way less than srep:m3f but the final game size was 300-400mb more but game install was much more faster...
if i remember correctly m3f dec mem was around 17gb but with m2f around 3-4gb...

[devil777]

To be... honest... desiring a Xtool version... which will work like Pzlib v3 4ever


Thanks razor for all the works

[Razor12911]

Originally Posted by Cesar82

@Razor12911
Thanks for the great job.
After completing this if you have time (also if possible), do a functional calback for your cls-diskspan library.
There are also some bugs with this library, using compressions formed by 3 methods.
Thank you!

Noted

[Razor12911]

Ok I decided to get Need For Speed The Run because I remembered that the game required a lot of decompression memory for srep to best demonstrate what I want to do with this deduplication idea.

I was surprised by the results

Xtool 2019:

Compressed 120 files, 14,503,034,446 => 29,120,915,487 bytes. Ratio 200.79%
Compression time: cpu 17.25 sec/real 807.42 sec = 2%. Speed 17.96 mB/s

100%: 29,120,917,667 -> 8,068,516,819: 27.71%.  Cpu 223 mb/s (124.672 sec), real 225 mb/s (123.527 sec) = 101%.  Remains 00:00
Decompression memory is 5660 mb.  2,922,729 matches = 46,763,664 bytes = 0.58% of file

Compressed 1 file, 8,068,559,139 => 4,315,048,356 bytes. Ratio 53.48%
Compression time: cpu 10.02 sec/real 172.58 sec = 6%. Speed 46.75 mB/s

Xtool 2020:

Compressed 120 files, 14,503,034,446 => 12,121,343,984 bytes. Ratio 83.58%
Compression time: cpu 16.13 sec/real 368.86 sec = 4%. Speed 39.32 mB/s

100%: 12,121,346,164 -> 8,010,462,480: 66.09%.  Cpu 141 mb/s (81.719 sec), real 145 mb/s (79.890 sec) = 102%.  Remains 00:00
Decompression memory is 759 mb.  2,105,722 matches = 33,691,552 bytes = 0.42% of file

Compressed 1 file, 8,010,469,968 => 4,296,282,176 bytes. Ratio 53.63%
Compression time: cpu 9.88 sec/real 159.45 sec = 6%. Speed 50.24 mB/s

Overall test results (compression):
13.5 GB (original) > 27.1 GB (xtool 2019 / 13 mins 27 secs) > 7.51 GB (srep m3f / 5660mb dec memory) > 4.02 GB (lzturbo 32 b128)
13.5 GB (original) > 11.2 GB (xtool 2020 / 6 mins 9 secs) > 7.45 GB (srep m3f / 759mb dec memory) > 4.00 GB (lzturbo 32 b128)

Not only is xtool 2020 faster but it produced a slightly better result while reducing srep's decompression memory. That 13.5 GB > 11.2 GB is not a mistake, xtool did precompress the data but there were too many repeated streams in the input and it removed a lot of them which resulted in negative ratio.

Notes: Xtool does incur additional decoding memory but it will definitely not be 5.6GB, it will far less. Maybe 2GB but if I apply proper memory management, I can reduce it to 1GB saving an additional 3GB of ram for lolz or something else.

[78372]

FreeArc 0.67 (March 15 2014) creating archive: data19.arc
Compressed 1 file, 197,759,488 => 941,714,574 bytes. Ratio 476.19%
Decompression memory is 116 mb.  146,130 matches = 2,338,080 bytes = 0.71% of file

FreeArc 0.67 (March 15 2014) creating archive: data20.arc
Compressed 1 file, 197,759,488 => 734,656,804 bytes. Ratio 371.49%
Decompression memory is 43 mb.  136,880 matches = 2,190,080 bytes = 0.66% of file

Good work

[Cesar82]

@Razor12911
Good work! Great results!

I don't understand practically anything about compression and flow methods, etc., but CIU has XTool (old Xtool, not 2019) and ZTool, and as far as I know some of the flows (I think that's it) are not supported by Xtool.
I would like to know if in this version when it is finished it will support all flows that old ZTool and Xtool support, or it will always be necessary to keep XTool + ZTool in CIU if I use XTool 2020.
It would be great to be able to remove some of the compressors from CIU.
Thank you!

[FitGirl]

Thanks for returning to the project, deduplication is a very useful feature.
I have an idea which will reduce the required RAM for dedup. You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen, cause reads will be rare and the RAM won't be used that much. 1-2 GB is a pretty big amount even for machines with 8 GB. And for users with 4 GB installation will be almost impossible, considering srep and lolz/lzma. Even with page file. So reduction/control over used RAM is a must, I think.

I'd recommend you Halo Reach for testing dedup, it has tons of duplicate streams of a different size.