XTool 2020 (Main Project)

[Razor12911]

Update available

Changes

- added depth info functions
- added support for oodle 2.9.0+ functions
- fixed data patching bug
- updated oodle codec
- updated command line parser

Changes between 0.3.9 to 0.3.11

  ES_R11 (0.3.11)
  - fixed x86 build bugs
  - fixed config multi-threading bug
  - fixed resource management bug
  - fixed deduplication bug

  ES_R10 (0.3.10)
  - minor bug fixes
  - added diff tolerance parameter (--diff=)
  - fixed plugin database bug
  - fixed lz4 codec bug
  - updated oodle codec
  - updated library structure
  - added resource management
  - added direct use encryption codecs
  - added embedded deduplication feature (--dedup) [makes temps during encoding]

@Cesar82

I updated the command line parser

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp { -moption} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

If you use "-mxtool:zlib:ue4,m1,k0x115E..." via Freearc, this is passed to xtool as "-mzlib+ue4:m1:k0x115E..." which should allow you to use it in your project.

[Razor12911]

@Everyone

The oodle precompressor in xtool is now more aggressive (means slower), I'll add aggression parameter to control how much time it should spend finding more and more streams however this only applies to kraken, mermaid, selkie and hydra. Leviathan is still problematic and you'll have to rely on plugins for data compressed using this codec.

Here are a few benchmarks that show what I am talking about:

0.99 GB (1,066,131,456 bytes) > 1.52 GB (1,638,079,260 bytes) [took 2 minutes, 9 seconds] (xtool 0.3.9)
0.99 GB (1,066,131,456 bytes) > 1.84 GB (1,977,644,853 bytes) [took 4 minutes, 56 seconds] (xtool 0.3.12)
0.99 GB (1,066,131,456 bytes) > 1.53 GB (1,649,308,007 bytes) [took 8 minutes, 52 seconds] (oo2reck)

As you can see, version 0.3.12 is now slower but it found more streams than 0.3.9 or the oodle precompressor side project.

This is a sample from Cyberpunk 2077 and since the game does have a plugin these are the results if you had use the plugin

0.99 GB (1,066,131,456 bytes) > 2.26 GB (2,432,780,012 bytes) [took 41 seconds] (plugin)

So what do these results mean? Well the newer version of xtool traded speed for more ratio in an attempt to beat the side project while still being faster. Also if a plugin for a specific game is created, it shows that the plugins will constantly be superior to the universal precompressor as it's not only faster but produces better results.

In conclusion, try to run tests with other games to see if the new xtool produces better results than oo2rec and if a plugin for a game exists then it's best to just use it.

[:( Sad8669]

I am now even more fired up for FIFA.

Thanks for these updates.

[Razor12911]

For FIFA and all these other frostbite games, 0.3.13 update is needed

[:( Sad8669]

Crash Bandicoot 4 Comparison - For zlib streams

0.3.12

Compressed 1 file, 19,665,315 => 91,532,563 bytes. Ratio 465.45%
Compression time: cpu 0.05 sec/real 2.77 sec = 2%. Speed 7.11 mB/s
All OK

0.3.9

Compressed 1 file, 19,665,315 => 98,920,152 bytes. Ratio 503.02%
Compression time: cpu 0.06 sec/real 2.80 sec = 2%. Speed 7.03 mB/s
All OK

Those FIFA's after 19 are such a pain...

[Razor12911]

Originally Posted by :( Sad8669

Crash Bandicoot 4 Comparison - For zlib streams

0.3.12

Compressed 1 file, 19,665,315 => 91,532,563 bytes. Ratio 465.45%
Compression time: cpu 0.05 sec/real 2.77 sec = 2%. Speed 7.11 mB/s
All OK

0.3.9

Compressed 1 file, 19,665,315 => 98,920,152 bytes. Ratio 503.02%
Compression time: cpu 0.06 sec/real 2.80 sec = 2%. Speed 7.03 mB/s
All OK

Those FIFA's after 19 are such a pain...

must be deduplication, check command line

FIFA after 19 isn't a problem, I don't have samples for these games so I am limited by the little samples that I have.

[Cesar82]

Originally Posted by Razor12911

@Cesar82
I updated the command line parser

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp { -moption} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

If you use "-mxtool:zlib+ue4,m1,k0x115E..." via Freearc, this is passed to xtool as "-mxtool:zlib+ue4:m1:k0x115E..." which should allow you to use it in your project.

@Razor12911, I don't know if another method besides ue4 has parameters. But supposing yes, how would I use a combination of 2 methods in which each one contains parameters?
Assuming that zlib had a parameter -m3 for example (just to specify the debt):

For example, assuming that zlib had a parameter -m3 (just to specify the debt) would it look like this?

"-mxtool:zlib,m3+ue4,m1,k0x115E..."

Should the arc.ini file be used in this way only { -moption}?
to use zlib+ue4 (Borderlands 3) it would not be necessary to send the -d1 parameter to XTool.
How can I pass or not the -d1 if I use the command like the arc.ini information reported above?

[Razor12911]

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp { -moption} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

-mxtool:zlib,m3:ue4,m1,k0x115E...

If you plan to add -d1 option via command line, you then have to modify the arc.ini to something like

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp { -option} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

-mxtool:d1:mzlib,m3:mue4,m1,k0x115E

which equates to -d1 -mzlib,m3 -mue4,m1,k0x115E
then -d1 -mzlib+ue4:m1:k0x115E when it reaches xtool

[:( Sad8669]

Might hook you up with some samples, once i get to testing it. If you would like.

Yes it was the dedup.

Without dedup :

Compressed 1 file, 19,665,315 => 98,946,941 bytes. Ratio 503.15%
Compression time: cpu 0.00 sec/real 4.12 sec = 0%. Speed 4.77 mB/s
All OK

[KaktoR]

Bethesda plugin does not work with xtool 0.3.12

0.3.9
Compressed 1 file, 373,427,426 => 532,225,912 bytes. Ratio 142.52%
Compression time: cpu 0.25 sec/real 4.74 sec = 5%. Speed 78.73 mB/s
All OK

0.3.12
Compressed 1 file, 373,427,426 => 373,427,610 bytes. Ratio 100.00%
Compression time: cpu 0.41 sec/real 2.22 sec = 18%. Speed 168.52 mB/s
All OK

Yes I have replaced bsa.dll with the new one.

[21h2]

Originally Posted by Razor12911

XTool is program made specifically repackaging games by providing a full suite of useful features such as data precompression, archiving, encryption and etc.

With that being said, nothing restricts it from being used on everyday files such as documents, pictures and media but with few limitations.

Read the documentation to find out how it works and how to use it.

Link for older releases.

i still dont understand how i can zip thinks with it

[Razor12911]

Update available

Changes

- updated lz4 codec
- updated library structure
- updated depth info functions
- updated depth feature

[kast1450]

XTool LZ4:
8:20:52 PM - Overall input size: 6.54 GB
8:20:52 PM - Overall output size: 3.91 GB
8:20:52 PM - Overall conversion time: 00:00:52

XTool 2020 LZ4:
8:22:33 PM - Overall input size: 6.54 GB
8:22:33 PM - Overall output size: 6.54 GB
8:22:33 PM - Overall conversion time: 00:01:02

Is this normal or a bug? I'm using the same settings with the same dataset yet one LZ4 gives good compression and the other gives no compression.

[ScOOt3r]

If i wanted to use Oodle with xtool
would this be correct with the latest verison of xtool

[External compressor:xtool]
header = 0
packcmd = xtool\xtool.exe precomp -moodle -c256mb -t22 --dbase --dedup - - <stdin> <stdout>

cause it doesnt work for me... get a disc full error

thanks
ScOOt3r

[Razor12911]

Originally Posted by kast1450

XTool LZ4:
8:20:52 PM - Overall input size: 6.54 GB
8:20:52 PM - Overall output size: 3.91 GB
8:20:52 PM - Overall conversion time: 00:00:52

XTool 2020 LZ4:
8:22:33 PM - Overall input size: 6.54 GB
8:22:33 PM - Overall output size: 6.54 GB
8:22:33 PM - Overall conversion time: 00:01:02

Is this normal or a bug? I'm using the same settings with the same dataset yet one LZ4 gives good compression and the other gives no compression.

xtool hasn't had a working lz4 codec.

Originally Posted by ScOOt3r

If i wanted to use Oodle with xtool
would this be correct with the latest verison of xtool

[External compressor:xtool]
header = 0
packcmd = xtool\xtool.exe precomp -moodle -c256mb -t22 --dbase --dedup - - <stdin> <stdout>

cause it doesnt work for me... get a disc full error

thanks
ScOOt3r

there is no codec with the name oodle however there is kraken, mermaid and selkie.

Edit: the parameter you used will require about 6GB ram if no streams are found and it will require even more if they are found

(256mb x 22 threads) + 256mb = 5,888MB ram usage and if the average inflation ratio is 200%, you're looking at 12GB+ ram usage by xtool. I'm letting you know before hand that the parameter you have input will make xtool use a lot of memory.