XTool 2020 (Main Project)

[kj911]

infovs: Please testing more version older dlls...

Example: 1.2.11/172 544byte and 104 448byte releases. And older 1.2.8/1.2.3 and more.

This (p)reflate library impact or not decompression speed? hif/raw dlls and preflate dll present it unpacker files folders.

[infovs]

Originally Posted by kj911

infovs: Please testing more version older dlls...
Example: 1.2.11/172 544byte and 104 448byte releases. And older 1.2.8/1.2.3 and more.
This (p)reflate library impact or not decompression speed? hif/raw dlls and preflate dll present it unpacker files folders.

I tested all zlibwapi.dll versions that I could find since ztool..Razor12911 version from 2020-07-18 (535552 bytes) was the fastest at decompression. I also did my own compile of zlibwapi.dll 1.2.11 with MS VC 2010 and also 1.2.13 with MS VC 2010/2015/2022 -> VC 2022 1.2.13 compile produced best result but Razor12911 version was still fastest at unpacking, although not by much, speeds were pretty close. Actually only this latest 2023-07-11/512000 bytes version speed drop was noticeable for me, that's why I posted my observation.
Also, if you use MS VC and want your compile to be XP compatible you are pretty much forced to use VC 2010..yes, you can use XP platform toolset on some later MS VC versions but I had some issues with it. Never compiled with MinGW though so I can't comment on that.

[Razor12911]

vc22_vista_1.2.13: 24688ms 24687ms 24750ms 24750ms 24782ms
vc17_winxp_1.2.13: 24375ms 24172ms 24156ms 24266ms 24359ms
rad_studio_1.2.13: 24359ms 24219ms 24203ms 24265ms 24313ms

vc19_vista_1.2.11: 21734ms 21672ms 21610ms 21843ms 21578ms
vc22_vista_1.2.11: 21391ms 21359ms 21391ms 21343ms 21282ms
vc17_winxp_1.2.11: 21453ms 21469ms 21359ms 21328ms 21360ms

vc22_vista_1.2.8: 21282ms 21281ms 21266ms 21281ms 21281ms

zlib-ng_1.2.13: 7843ms 7797ms 7781ms 7813ms 7828ms

Originally Posted by infovs

Don't really know what you did with 1.2.11/2020-07-18 compile because that was clearly the fastest zlibwapi.dll, at least for me. Yes, I know it is older 1.2.11 version and maybe missing some updated code but I never encountered any problems with it

I didn't do anything to 1.2.11 to make it fast, 1.2.13 itself is the one that is slow as you can see with the benchmarks, winxp compile has little to no impact on performance.

There's an issue opened on github regards to the speed which can be found here.
https://github.com/madler/zlib/issues/820

There was also a security vulnerability discovered in 1.2.11 and I don't know if patching this issue is what made zlib slower or not moving forward.
https://www.cybersecurity-help.cz/vdb/gnu/zlib/1.2.11/

The beauty of xtool is that the user can swap libraries if they prefer to use their own, there's also zlib-ng (utilizes modern CPU features) benchmarks added, as you can see it is faster however I doubt it produces matching crc as standard zlib but good to know nonetheless.

[Razor12911]

Update available

Changes

- added universal scanner for DirectStorage gdeflate streams
- added the use of gpu for caching and virtual memory purposes
- updated depthing feature to improve stream detection when used by plugins

Notes
With the release of the game ratchet & clank: rift apart which used direct storage's gdeflate compression within its game files, I thought... hm, why not just add a scanner for these streams and so I did after investigating the stream structure (was easy to figure out actually). Use via -mgdeflate.

Will games use direct storage? I don't know but if they do then at least xtool has support for the streams used in the games, I just added it so that repackers don't flood me with DMs when they have troubles repacking games, maybe Starfield might use this...

After several tests, the gpu feature can now be used by the general public, use via -g#

[L33THAK0R]

Is there any way to use XTool to erase compressed streams? I'm currently looking at the various Quantic Dream PC ports (i.e. Heavy Rain, Beyond Two Souls, Detroit Become Human) and they have the localized audio compressed using deflate/zlib, I'm not sure if its already possible with XTool but can a database be generated, and then an input folder in addition to a generated database be fed as parameters to XTool to target and blank out these streams?

[daveyrob]

Originally Posted by L33THAK0R

Is there any way to use XTool to erase compressed streams? I'm currently looking at the various Quantic Dream PC ports (i.e. Heavy Rain, Beyond Two Souls, Detroit Become Human) and they have the localized audio compressed using deflate/zlib, I'm not sure if its already possible with XTool but can a database be generated, and then an input folder in addition to a generated database be fed as parameters to XTool to target and blank out these streams?

xtool erase <folder> <archive> <database>

[Razor12911]

Update available

Changes

- fixed an issue where xtool would crash on Windows XP and other systems upon closure
- memory optimisations
- improvements with database "based" plugins

[infovs]

Thank you Razor12911, also..is there a reason not mentioning there is also new CLS version of xtool in your 0.7.8 update? Not stable/recommended yet?

[Masquerade]

^^
Run some tests and see what you find. That's what I'm gonna do when I return to my PC.

Please note that FreeArc is an x86 program, so in order to use the x64 CLS variant, you will first need to use some kind of wrapper such as wrapcls by 78372 or FAZip (not FAZip32).

It's gonna be a lot of fun to play with. A cls is what I've been secretly wishing for, thanks Razor

[shazzla]

@Razor :

Maybe an XTool 0.7.8 bug ,using FLAC. Why is that increased memory consumption ?

Compression : -mflac:l8:c128mb , 1 thd ,approx. 800 MB RAM usage
Decompression : 1 thd ,approx. 180 MB RAM usage
Decompression : 2 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)
Decompression : 4 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)

Compression : -mflac:l8:c128mb , 4 thds ,approx. 1.8 GB RAM usage
Decompression : 1 thd ,approx. 180 MB RAM usage
Decompression : 2 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)
Decompression : 4 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)

All decompression settings used % to set the number of threads. Using T4 instead of t100p cannot decompress the archive :
ERROR: general (de)compression error in flac:l8:c128mb

Maybe other cases affected. Dont know.

i5-6600

[Razor12911]

There's two different code sets for decoding in xtool, one is for multi threading which caters for speed at a cost of increased memory usage because the streams have to be reorganised before they are written to disk and as such, high memory usage because they are stored in memory first. The other code is when you have set 1 thread of if a block has 1 stream in it, for this as soon as a stream is processed, it is written to disk hence why the reduced memory usage in -t1.

For flac, I'd suggest using -t1 for decoding simply because flac is decompressing when installing unlike other codecs where they require multi threading to be fast as they are doing the opposite (compressing) during installation.

A small test on some audio files

Tested 39 files, 745,011,186 => 1,379,878,320 bytes. Ratio 53.99%
Testing time: cpu 1.08 sec/real 4.68 sec = 23%. Speed 295.10 mB/s

Using 12 threads, used 1.23 GB ram

Tested 39 files, 745,011,186 => 1,379,878,320 bytes. Ratio 53.99%
Testing time: cpu 0.97 sec/real 10.18 sec = 10%. Speed 135.58 mB/s

Using 1 thread, used 185 MB ram.

The single threaded speed is reasonable while using 12x number of threads only gave 2.2x speed benefits while using nearly 7x memory. Obviously due to IO bottleneck show that it's better to just use 1 thread.

[infovs]

Hi Razor12911, I know similar question has been posted before but can you please re-consider renaming hardcoded srep.exe call in x64 xtool version to original, srep64.exe ? Since you made xtool available for both x86 and x64 versions I started using both when appropriate (making XP compatible installers for games that can be played on XP...) and hardcoded srep.exe is giving me headaches because both x86 and x64 are executing srep.exe. x64 xtool version should be executing srep64.exe IMHO
Anyway, thanks again for your xtool, best regards

[Razor12911]

Update available

Changes

- added recompress streams feature
- added reassign streams feature
- added dummy codec
- added data transfer buffer for srep when dd# is used
- added memory checks to ensure memory usage does not run wild
- exectuable plugins (via xtool.ini) are no longer required when decoding if they were never used
- internal stream deduplication now reports speed and memory usage benefits
- configuration based plugins can now have multiple names (if multiple games use the same configuration)
- fixed xtool crash if an incorect path for plugins was provided
- fixed lz4/lz4hc codec bug when used directly without any plugin
- fixed an issue where using fast-lzma2 compression would crash when decoding
- fixed an issue in DirectStorage gdeflate codec
- fixed an issue with execute command in stdio mode
- deduplication memory requirements improved
- memory optimisations
- reduced memory requirements for large streams that require patching
- removed ability to inject libraries to exe (buggy)
- removed patch command (uses too much memory)
- removed archive command (no one uses it)
- updated oodle scanner
- updated lz4/lz4hc universal scanner
- updated lz4/lz4hc/lz4f codecs
- updated zlib codec
- updated zstd codec
- replaced gpu caching feature with normal system memory cache (it doesn't work well on AMD gpus), use -p#
- replaced xdelta3 with zstd patching engine
- srep64 executable considered in x64 build of xtool

Notes

First and foremost, huge thanks to KaktoR for helping with this update, he was full of energy when we started but I think I broke him towards the end...
kekw.png

This update was supposed to be released a long time ago, but he would find more bugs hence the long list of changes.

I won't say much other than notable changes, xdelta was replaced by zstd patching engine (yes... zstd can make patches too, shocking I know...), reason for replacement xdelta gives me less options coding wise to work with in terms of memory management.

Injecting libraries feature also removed as some libraries when injected completely stop working or worse, cause xtool to crash.

I've quietly added cls plugin in the previous update and did not tell anyone about it simply because it was never tested, for this update It was tested and it works so long as you never used fast-lzma2 in any of the archives when using it with installations created with inno setup as it would simply cause a crash.

I've left 0.7.8 on the main post in case there are issues with this update even after extensive testing.

[L33THAK0R]

Very embarrassed to ask such a simple question regarding database creation, but I can't seem to wrap my head around it. I've got some assets I'm attempting to rip some assets that are compressed inside an archive using deflate.

To do this I'm pretty confident in saying xtool just needs a database to reference, with the command looking like

xtool erase <folder> <archive> <database>

as a user pointed out to me a while ago.

Where I get stuck is creating the database, the title in question is "The Wanderer" a UE4 title (unencrypted) with a singular ~30GB ".pak" archive containing said assets, I've extracted the assets I'm interested in ripping to their own folder and have so far tried using xtool 0.7.9's GUI to generate the database since I'm not too confident the syntax I'm using is correct when running xtool from the command line.

Here is how I've setup the database creation though I'll admit I'm unsure of whether I've got the "Input" and "Source" correct.



However all I get from this configuration is the following message, which hangs for a few minutes despite some disk activity as per task manager, and then indicates the process is completed however no database file is created.



Very unsure and confused about what parts I'm mucking up or whether XTool just has a limitation on the size of inputs that can be processed.

[Masquerade]

Don't use the GUI. In that screenshot, you are using the wrong database generator and your inputs+outputs need to be reversed.

And you cannot rip compressed data. It must appear in plaintext in order to be erased by xtool.