XTool 2020 (Main Project)

[huluwizard]

Forgive me if I have missed something obvious, but the first post in this thread mentions documentation, but I'm not able to locate any documentation for xtool. Specifically, I'm trying to figure out the command line. I can get the results I want from the GUI plus a plugin, but ultimately I'd like to pair it with a batch file that does the same thing.

[KaktoR]

xtool.chm is what you are looking for.

[huluwizard]

Originally Posted by KaktoR

xtool.chm is what you are looking for.

Oh wow, it really was right there all along. Thanks!

[dixen]

And yet I don't understand how to use the memory limit in the -dd# function. The --srepmem=# parameter does not give anything at all. Maybe I'm doing something wrong?

[Cesar82]

Originally Posted by dixen

And yet I don't understand how to use the memory limit in the -dd# function. The --srepmem=# parameter does not give anything at all. Maybe I'm doing something wrong?

I think the parameter for srep memory usage in XTool 0.70 is "-sm#" and not anymore "--srepmem=#".

[shazzla]

It would be nice if someone document this parameters....

[Masquerade]

Originally Posted by shazzla

It would be nice if someone document this parameters....

CMD> xtool precomp

XTool is created by Razor12911

precomp - data precompressor

Usage:
  xtool precomp [parameters] input output


Parameters:
  -m#  - codecs to use for precompression (separate with "+" if more than one)
  -c#  - scanning range of precompressor [16mb]
  -t#  - number of working threads [50p]
  -d#  - scan depth [0]
  -dd  - use stream deduplication
  -l#  - compress data using fast lzma2 (separate params with ":")
               d# - dictionary size
  -lm  - low memory mode
  -s   - skip stream verification
  -v   - enables verbose
  -df# - set xdelta threshold to accept streams [5p]
  -x#  - extract streams to directory path
  -dm# - deduplication memory usage limit (#=size) [75p]
  -sm# - srep memory usage limit (#=size) [75p]

[Razor12911]

Executing xtool with the command as the only parameter will display information of how the command is used alongside the available parameters. The reason certain parameters remain undocumented is simply because they were added for personal usage or as a gimmick and not a lot of testing has been done and as such, they may not work as intended and may corrupt your data in certain instances.

Any of the listed parameter is fine to be used by you, the end user however anything not listed here is limited to advanced users who may have gone through the source code over on github to even know of these parameters existence.

[shazzla]

Thanks,but i miss exactly those undocumented parameters!

Somehow i missed github. lol
Going there now,btw thanks!

[Razor12911]

Update available

Changes

- memory usage optimizations

Notes

Resources utilised by zlib, lzo, zstd and some other codecs have been made to initialize only when used, this was down to reduce memory usage and allocation.

If you're users like shazzla however, who want to utilise features of xtool before they are tested and ready for, there's a new parameter introduced in this release which makes xtool utilise the GPU's VRAM to improve precompression speed and reduce memory usage even more when decoding by offloading deduplication memory requires onto the GPU.

-g# (# may be a percentage or specific value), default value is 0

So how does it work? When precompressing, xtool reads, processes then writes, it does this again and again until it finishes. The problem with this however is there's a bottleneck when it is writing especially if you're repacking on HDD because xtool may be processing faster than it is actually writing to the disk so that's where caching feature introduced in 0.7.0 and the new GPU feature comes in. Instead of writing straight to disk, the data is written to the GPU and as xtool reads and processes the next batch of data, the GPU will be writing to the disk making sure that the processing aspects of xtool are not slowed down.

When decoding, xtool can sometimes use the ram for the duplicated streams resulting in high memory usage when installing a repack, usually when you're installing a game, the GPU isn't doing anything so these duplicated streams are stored on the GPU, this is done to reduce memory usage. 75% of the allocated GPU is dedicated to this while 25% is dedicated to caching data from srep+lolz/lzma or what it is that you use where xtool would be reading data in advance to reduce bottlenecks even more.

TLDR; how to enable this feature? just add -g75p when encoding/decoding (can be both, up to you)

How to know if it's working? Check in task manager and you should see xtool utilising the GPU with the Engine "Copy".

Benchmarks

0.7.1

XTool is created by Razor12911

Streams: 1415315 / 1415335
Time: 00:07:20 (CPU 00:44:37)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

0.7.2

XTool is created by Razor12911

NVIDIA GeForce GTX 1060 6GB (4.50 GB loaded)
Streams: 1415315 / 1415335
Time: 00:05:31 (CPU 00:46:09)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

There was a 30% speed improvement, your mileage may vary but only use this feature if you repack on HDD. There won't much speed gains on SSD.

OpenCL was used to achieve this so if your PC does not have the library in system32 folder, you should place the dll near xtool.

If feature does not work as intended then you must understand why I keep some features undocumented.

[Masquerade]

Nothing short of incredible, always love seeing XTool updates. Thanks again Razor!

[infovs]

Thank you Razor12911 for your incredible work! Two short questions if it is not too much trouble...how to use undocumented option introduced in 0.70?
- added optimize option to speed up the decoding process for zstd and oodle codecs
Looking at source it is used by adding parameter -o ...but not sure where, at decoding or encoding... I tried used it at both places, precomp some oodle data -> decoding speed was exactly the same as before without -o parameter :\
Also, is there any chance or parameter that can make xtool use and utilize srep64.exe instead of renamed srep.exe?
Thanks in advance, again, big thanks for everything!

[Razor12911]

Originally Posted by infovs

Thank you Razor12911 for your incredible work! Two short questions if it is not too much trouble...how to use undocumented option introduced in 0.70?
- added optimize option to speed up the decoding process for zstd and oodle codecs
Looking at source it is used by adding parameter -o ...but not sure where, at decoding or encoding... I tried used it at both places, precomp some oodle data -> decoding speed was exactly the same as before without -o parameter :\
Also, is there any chance or parameter that can make xtool use and utilize srep64.exe instead of renamed srep.exe?
Thanks in advance, again, big thanks for everything!

Hi, optimise was added but not documented simply because the conditions needed for the benefits to even be noticeable are obscure, essentially the game Returnal was the reason this feature was added. The game used leviathan which takes a long time to process, to put it into perspective, if a stream was compressed using level 9 and let's say it takes 20 seconds to process, what optimise does is try lesser compression settings that give the same output, like level 8 which takes half the time on average compared to level 9 so that's about 10 seconds, 50% gain in speed, it does this again and again until the compression setting no longer gives the same output. As a result, some level 9 compressed streams of Returnal could be compressed with level 6, some level 7.

With this in mind, the installation should be faster right? Well yes it is but I wasn't impressed by the improvements so I left it undocumented.

Benchmark on pakchunk2_s8-WindowsNoEditor.pak of Returnal

without -o

XTool is created by Razor12911

Time: 00:22:17 (CPU 02:12:41)

with -o

XTool is created by Razor12911

Time: 00:18:50 (CPU 01:52:02)

We got about 15% speed improvement. You might see this as a win, but precompression took 32 mins with -o than 23 mins without -o so 40% more time for precompression.

As for srep/srep64, just let 32-bit programs die.

[janb14]

Hello everyone,
first of all thank you Razor12911 and eveyone else who made this possible.
Im currently experimenting with xtool and noticed no dedicated handling of ogg files. Im trying to compress some of my minecraft backups where mods with ogg sound files contribute a significant share.
I have already extraced some and compressed them with standalone oggre. Now i want to include them the same way panker1992 (i recreated his wavpack config and it worked ;D) did but im not really a hex guru. Did anyone already have a working ogg config ?
So far ive found the Signature but im not sure what to fill in for the rest of the ini values:

[Stream1]
Name=ogg
Codec=oggre
BigEndian=0
Signature=0x5367674f
Structure=Signature(4),FileSize(4),FileType(4),Stream
StreamOffset=-12
CompressedSize=FileSize + 8
DecompressedSize=0

Originally Posted by panker1992

Hello, it's been ages since i had an error !!

This Came directly from Xtool documentation
Xtool version used 6.2

this is my wav.ini
[Stream1]
Name=wav
Codec=wavpack
BigEndian=0
Signature=0x46464952
Structure=Signature(4),FileSize(4),FileType(4),Str eam
StreamOffset=-12
CompressedSize=FileSize + 8
DecompressedSize=0
Condition1=FileType = 0x45564157
Condition2=FileSize >= 4096

this is my xtool.ini
[wavpack]
Encode=wavpack.exe -hh -x4 <filein>.wav <fileout>.wv
Decode=wvunpack.exe <filein>.wv <fileout>.wav

i have been unable to make this even detect the sample FLAC internal codec works and so any other
NOTE: i use this as standalone and not in FA for testing purposes

[Masquerade]

^^
I'd advise against using XTool with oggre because oggre has built in deduplication that will not be working when each ogg is compressed individually with XTool.