XTool 2020 (Main Project)

[Razor12911]

Hi, the older versions of xtool and the current version have different code infrastructure and porting the deduplication code would be time consuming, also a standalone deduplication program wouldn't make sense simply because the xtool version is stream based meaning, it relies on whatever streams are detected, however if you just wanted deduplication in stdio mode, you can just use srep in -m0 mode rather than -m#f, not sure if that answers your request

[Razor12911]

Update available

Changes

- added ability to redirect base directory for plugins and libraries
- added restrictions to avoid errors with experimental codecs
- added optimize option to speed up the decoding process for zstd and oodle codecs
- added dictionary parameter for fast-lzma2
- added memory caching when decoding to alleviate speed bottleneck
- fixed bug with download feature for inputs in URL format
- fixed issues with exporting precompression database
- fixed issues with executable plugin support
- fixed issues advanced configuration based plugin support
- fixed potential decoding issue upon using plugin support functions
- fixed issues with deduplication feature
- fixed issues with jojpeg codec
- replaced crc32c with xxh3_128 to reduce collisions when using the database and deduplication feature
- replaced memory manager with FastMM4-AVX to improve scaling in multi threaded scenarios
- improved user interface
- improved oodle codec performance for 2.6.0+ libraries
- improved encoding speed when using internal codecs
- improved processing speed when depth is used
- removed fast lzma2 multi threaded decompression due to excessive memory requirements
- removed debugging information when using the patch function
- removed ability to toggle database feature and ability to export database files (now enabled by default)
- updated deduplication virtual memory allocation
- updated reflate codec to verify streams prone to data corruption

Notes
Database files created using old tools (ucas database, dunia2 database etc) are not supported in this version, wait for updates for these tools.
Reflate may be slightly slower compared to previous versions and that's because verification on suspected streams that may cause crc errors has been added.

Speed when using -d# parameter has been improved as show below on the game Star Ocean The Divine Force's chunk_2.cpk file.

0.6.9

Compressed 1 file, 641,373,728 => 1,104,208,408 bytes. Ratio 172.16%
Compression time: cpu 0.53 sec/real 37.94 sec = 1%. Speed 16.90 mB/s

Tested 1 file, 1,104,208,408 => 641,373,728 bytes. Ratio 172.16%
Testing time: cpu 0.41 sec/real 73.39 sec = 1%. Speed 8.74 mB/s

0.7.0

Compressed 1 file, 641,373,728 => 1,104,208,460 bytes. Ratio 172.16%
Compression time: cpu 0.61 sec/real 30.71 sec = 2%. Speed 20.88 mB/s

Tested 1 file, 1,104,208,460 => 641,373,728 bytes. Ratio 172.16%
Testing time: cpu 0.45 sec/real 22.72 sec = 2%. Speed 28.23 mB/s

[dixen]

Just a tip, nothing more..
What about the progress status for -dd3 and fast-lzma? For example this
<input_data_size> >> <Output_data_size>..

[shazzla]

Something is wrong with Xtool 07 hotfix.

It doesnt catch up any stream on Snowrunner's EDITOR.PAK ,while Xtool 069 does ! (069 fails at the end on this file with reflate ,zlib works fine !)
See below :

......

Edit #2 :
Ohh ,maaan! ALL DLL's must be in the libraries folder ! Not only the external ones ,like unreal ,unity,etc...

[Razor12911]

Originally Posted by dixen

Just a tip, nothing more..
What about the progress status for -dd3 and fast-lzma? For example this
<input_data_size> >> <Output_data_size>..

Streams: 58080 / 58080
Time: 00:01:27 (CPU 00:15:18)
Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB]
Srep decompression memory: 45.0 MB [113 MB*]

Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma)

I'm not sure what the question is, but the information shows up when you use the program w/o FA.

@shazzla

be mindful of base directory or -bd#, this allows xtool to relocate libraries and plugins (including xtl and ini files) from another directory

I added this because there were just too many files near xtool.exe, also if you want to compress games but rather than moving and deleting oo2core..., you can just change the base directory.

[shazzla]

This "base directory or -bd#..." is not so clear...
How to use/set -bd# ?

[Razor12911]

Let's say you want want to move all libraries and plugins to a folder called "libs" which is near xtool.exe, you specify base directory as -bdlibs or -bd.\libs. or you can just leave it empty to make xtool work the same as before.

[dixen]

Sorry for dumb question, but how get this info in xtool?

Streams: 58080 / 58080
Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB]
Srep decompression memory: 45.0 MB [113 MB*]

Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma)

[KaktoR]

Originally Posted by dixen

Sorry for dumb question, but how get this info in xtool?

With these settings

[Razor12911]

The information is always visible and verbose option isn't necessary because that will force xtool to use 1 thread, whether you use GUI or CLI. It's always there.

[Razor12911]

There's a feature I'd like to point out which was briefly added in 0.6.8 but wasn't finished until this current release and that is on the fly compression, basically if you wanted to quickly find out the theoretical compressed size of a game without consuming disk space.

Reasons you may want to use this feature:
+ If you just wanted to find out the compressed size
+ If a game uses kraken and wanted to see if it was worth it to compress it at all
+ If the game is big and you don't have space for compression
+ If you don't want to reduce lifespan on SSD

XTool is created by Razor12911

Streams: 1415335 / 1415335
Time: 00:01:21 (CPU 00:04:03)
Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB

Done!!!

13GB is the size that srep is expecting, so you change duplication preset to srep, then set input size to at least 15GB then enable compression, set desired level and dictionary.

Streams: 1415335 / 1415335
Time: 00:04:36 (CPU 00:34:05)
Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 751 MB [5.15 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.03 GB >> 4.65 GB

Done!!!

So then you know that 15.3 GB becomes 4.65 GB in less than 5 mins but without creating any temp files as the whole compression takes place in memory. You also know srep decompression memory too.

_________________________________

The other thing, as you know how much I tried to make xtool to perform as fast as possible, the thing that always bothered me was that lzma was slow and that's why I added fast-lzma2. I noticed something interesting about this compression library in the sense that it's able to increase the number of threads used for compression without increasing memory usage.

Here's a few results

4x4:lzma (via fazip) (b128mb:t6), ultra, 64mb dictionary, 12 threads used 2781 MB ram

Compressed 288 files, 16,391,993,193 => 4,869,896,897 bytes. Ratio 29.71%
Compression time: cpu 13.50 sec/real 662.85 sec = 2%. Speed 24.73 mB/s

fast lzma2, level 10, 64mb dictionary, 12 threads used 403 MB ram

Compressed 288 files, 16,391,993,193 => 4,929,451,310 bytes. Ratio 30.07%
Compression time: cpu 11.84 sec/real 425.39 sec = 3%. Speed 38.53 mB/s

Not only is memory used by fast lzma2 low, but it's much faster while producing similar results and what is impressive is, if you had like 32 thread cpu, you can increase its speed even further without increasing memory usage by too much but if you use the 4x4 method, the memory usage will increase even more.

[infovs]

Thank you Razor12911 for this fantastic tool...have one question, if it is not too much trouble, can you please consider naming x64 srep.exe (since xtool is hardcoded using this exact name) like Bulat original named it as srep64.exe?
I'm using original naming for many years, I need 32bit srep.exe for some compatibility reasons of my own...and your renaming 64bit srep64.exe as srep.exe is causing problems.
Maybe at least consider making xtool using srep64.exe name first, if found, then if not found use srep.exe or whatever srep name you consider acceptable.

Thanks in advance,
best regards

[kj911]

Originally Posted by Razor12911

Streams: 58080 / 58080
Time: 00:01:27 (CPU 00:15:18)
Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB]
Srep decompression memory: 45.0 MB [113 MB*]

Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma)

Final compressed size without srep?? Better or worser?

FLZMA2: This really? Small memory footprint + utilize extra more cores? (This feature available from native 7-Zip releases?)
Likelys the feature from LOLZ. More 4-12C/4-24T CPU's from in visible this ONE virtually core utiiized during in compression. (Faster speed in LDMF1 mode.)

[Wanterlude]

Guys, does anyone know how to fix this?
Game: A Plague Tale Requiem
Files: *.DPC
Xtool: last ver 0.7.0 hotfix
Problem: after unpacking 1 file (28.9gb) xtool goes beyond its allocated memory. (x4)
The picture shows the information from windows resource monitoring. (I have 16 GB RAM)

[External compressor:xtool]
header = 0
unpackcmd = "xtool.exe" decode -t100p -dm20p -sm20p - - <stdin> <stdout>

[External compressor:xtool]
header = 0
unpackcmd = "xtool.exe" decode -t100p -sm20p --mem=20p - - <stdin> <stdout>

In 1 case it happens as shown in the picture. In 2, the memory increase can be seen even in the windows task manager.

arc.ini:

[External compressor:xtool]
header    = 0
default   = -c128mb -t100p
packcmd   = xtool.exe precomp -mzlib -d1 -db -dd5 - - <stdin> <stdout>

-----------------------
upd. (14 may): preflate has the same problem | xtool 0.6.9 - same
-----------------------
upd. (15 may): fixed by 2nd xtool and -dm20p -sm20p parameters.
I think the problem was that I used 1 xtool for both zlib and dedup
-----------------------

[Razor12911]

Update available

Changes

- fixed issues with fast-lzma2 being unable to set correct compression level
- updated deflate stream scanner