XTool 2020 (Main Project)

[Razor12911]

Originally Posted by FitGirl

I suspect it depends on number of duplicated streams

you answered yourself

but as I said, I have a few ideas to reduce the memory requirements like getting rid of streams from memory that have restored the duplicated streams. I am not sure if srep does this already but if it doesn't then it should be a bonus. GPU memory usage is another thing that will be added but in future. I need start adding support for other codecs, especially the external ones. What I plan is for people to write their own plugins regardless whether they can write code or not. If you can figure out a header structure of a game, just write and ini file and give it to xtool and it will do the rest. This is mostly for lz4 games, encrypted ones or ones that require something special. I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)

Here's an example of Saints Row IV Remastered, instead of making another side project, you just write this and give it to xtool.

[stream1]
Codec=lz4hc:l10 // lz4hc compression used, level 10
Header=00BADBEE0FEEDBEE // this is the magic bytes
Structure={Header(8)} {CSize(4)} {DSize(4)} {Stream} // this is the header structure
Condition1=DSize > CSize // the conditions for xtool to accept the stream
Condition2=CSize > 64
Condition3=CSize < 16 x 1024 x 1024

Maybe it's possible for xtool to report it after initial decompression is done?

this I can do


@everyone

here are some benchmarks, I finally written the decompression code, though it needs some tweaking before I can post update

normal xtool

Compressed 1 file, 320,782,304 => 108,018,914 bytes. Ratio 33.67%
Compression time: cpu 0.47 sec/real 22.64 sec = 2%. Speed 14.17 mB/s

Tested 1 file, 108,018,914 => 320,782,304 bytes. Ratio 33.67%
Testing time: cpu 0.38 sec/real 14.91 sec = 3%. Speed 21.51 mB/s

+ memory database
+ deduplication

Compressed 1 file, 320,782,304 => 107,858,828 bytes. Ratio 33.62%
Compression time: cpu 0.44 sec/real 15.98 sec = 3%. Speed 20.08 mB/s

Tested 1 file, 107,858,828 => 320,782,304 bytes. Ratio 33.62%
Testing time: cpu 0.44 sec/real 9.32 sec = 5%. Speed 34.41 mB/s

thus far, the output is slightly better for more speed

[FitGirl]

Configuration feature is great! It would be so much easier to repack some obscure formats.

[dixen]

Originally Posted by Razor12911

I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)

For LEGO series games this would be fine)

[FitGirl]

Originally Posted by dixen

For LEGO series games this would be fine)

It will never work with LEGO series, cause there are no public compressors for algos used in those games, and the decompressors used by Luigi are in binary (ripped from DLL) format, so they can only be extracted, not packed back.

[Razor12911]

Originally Posted by FitGirl

Any predictions on RAM requirements for both imported/memory DBs?

back on the SEACREST input from Need For Speed Hot Pursuit

I ran a memory usage test as you asked and these were the results.

xtool-virtual-memory.tmp = 328 MB
Decompression memory is 22 mb. 80,927 matches = 1,294,832 bytes = 0.10% of file

= total 350 MB (with deduplication)

Decompression memory is 1485 mb. 147,537 matches = 2,360,592 bytes = 0.18% of file

= total 1485 MB (without deduplication)

NOTE: Xtool uses memory of its own and this was not factored, about 190 MB
so if you do the math, that's 350+190 = about 540 MB total memory while if xtool doesn't remove deduplicates and lets srep all the work that's 1485+190 = 1675 MB

540 MB vs 1675 MB memory usage, which is better?

when 540 MB ram was used, xtool was decoding at 88.91 MB/s
and when 1675 MB ram was used, xtool decoded at 27.45 MB/s

so yes more speed and less memory usage, potato pcs should benefit greatly from this (like mine )

[Masquerade]

Originally Posted by Razor12911

xtool-virtual-memory.tmp = 328 MB
Decompression memory is 22 mb. 80,927 matches = 1,294,832 bytes = 0.10% of file

= total 350 MB (with deduplication)

I know you don't have much data to go on, but how does xtool+dedup compare against xtool+srep?

[FitGirl]

Originally Posted by Masquerade

I know you don't have much data to go on, but how does xtool+dedup compare against xtool+srep?

The idea of new xtool is not replace srep. It's purpose is not to recompress duplicate chunks, but just copy them, which makes less CPU load and faster overall decompression. You still need to run srep after xtool, cause it will deal with unpacked duplicated chunks.

[Masquerade]

Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB



So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.

[FitGirl]

Originally Posted by Masquerade

Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB



So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.

Afair Scribblenauts is encrypted. You are lucky to always choose encrypted games for your tests

[Harsh ojha]

Originally Posted by Razor12911

so yes more speed and less memory usage, potato pcs should benefit greatly from this (like mine )

Same here potato pc 😅

FitGirl Repack take more HHD space for installing game ex. Far Cry 3 duology
Need 17 GB HHD space free
But the game Size was 10 GB ( language english.
Without FC3 mapEditor )

*Sorry for my bad English as English is not my mother tongue 😅*

[Masquerade]

Originally Posted by FitGirl

You are lucky to always choose encrypted games for your tests

Sucks to be me I guess. I don't knowingly do this.

[Razor12911]

Originally Posted by Masquerade

Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB



So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.

hmm, final size after compression?

[Masquerade]

Originally Posted by Razor12911

hmm, final size after compression?

With xtool r12: 1.38GB

I feel slightly ashamed for not thinking the data could be encrypted.

Didn't test to make a full archive with 2020, I will do this if you wish.

Every time I put xtool 2020 up against xtool r12, r12 always produces a larger size. Both run at 128m chunk size.

[Razor12911]

and the final size of 2020?

[shazzla]

@Razor12911 :

Few days ago i started playing with XTool 2008. Great job ! Thanks !
But i found this :

Frostpunk - On the edge (Is its streams ZLIB? It must be IMHO)

It cant precompress those .dat files. (For example ,common.dat and languages.dat were tested).
Common.dat nearly a 600 MB file,languages.dat is a much smaller.
In both case ,the output is the same.
Older generation of XTool (AFAIR V0.9) precompresses commond.dat well to ~1.5 GB, languages.dat ,too.
XTOOLs (older) setting was : 128mb
XTOOLs (2008) setting was : 128mb ,d0 and d9 ,after this 256mb. (doesnt matter)

In my other test i faced with this (dont remember what game,maybe Hellbound):
XTool 2008 inflation was 2 gigs AFAIR ,compressed with LOLZ ,after that compressed the game without XTOOL2008 and the same LOLZ parameters and it was smaller with some MBs ! (maybe a special case ,dont know)

ATM im away from my computer for the next...lot of days ,i cant send example files.

Keep up the good work !