XTool 2020 (Main Project)

[elit]

Razor I think you mentioned in the past that -mzlib is able to detect deflate? I recall mentioning back then that it did not work for me. So here I am attaching set of png images that I just accidentally tested and did not work. Both -mreflate as well as previous ztool:high does work.

pngs.zip

[Razor12911]

-mzlib can detect deflate streams, but similarly to my previous post regarding leviathan, leviathan streams are detected however detection and being able to process them are two different things.

XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 000000000000A046 (41031)
[0] Actual zlib stream found at 0000000000000053 (40928 >> 87680)

[0] Processing streams on block 0000000000000000 to 000000000000A046 (41031)
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l11,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l12,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l13,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l14,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l15,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l16,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l17,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l18,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l19,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l21,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l22,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l23,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l24,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l25,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l26,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l27,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l28,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l29,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l31,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l32,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l33,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l34,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l35,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l36,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l37,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l38,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l39,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l41,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l42,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l43,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l44,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l45,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l46,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l47,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l48,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l49,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l51,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l52,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l53,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l54,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l55,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l56,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l57,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l58,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l59,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l61,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l62,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l63,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l64,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l65,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l66,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l67,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l68,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l69,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l71,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l72,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l73,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l74,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l75,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 3072) using l76,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l77,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l78,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l79,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l81,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l82,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l83,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l84,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l85,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l86,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l87,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l88,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l89,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l91,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l92,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l93,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l94,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l95,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l96,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l97,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l98,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l99,w15 has failed

Streams: 0/1
Time: 00:00:00 (00:00:00)
Memory: 128 MB (128 MB)

I tried to precompress one of the png files and as you can see, the method -mzlib was able to detect the deflate stream but then it failed to process it.

[KaktoR]

Originally Posted by Razor12911

Update available

Changes

- removed leviathan codec restriction

Notes

Xtool can "actually" detect leviathan streams however the only way it can process them is if they are divisible by the block size used by the new oodle compressors which is 262144, if a stream decompressed size cannot be divided by this value leaving no remainder then no way of predicting the stream size (at least that I know of) is possible hence why plugins are the only ones that were allowed to use the leviathan codec.

-mleviathan

If you are not satisfied with the results produced then find a way to make a plugin for proper support for that particular game.

It's better then nothing I guess, especially if you're not familiar with creating plugins, so thanks for the update

Streams: 354/415
Time: 00:01:18 (00:01:17)
Memory: 423 MB (423 MB)

64,1mb >> 352mb

[Masquerade]

Thanks Razor, this will be good for when the generator is working.

[elit]

Originally Posted by Razor12911

-mzlib can detect deflate streams, but similarly to my previous post regarding leviathan, leviathan streams are detected however detection and being able to process them are two different things.

Thanks, so on more technical ground what is to be expected going forward regarding this? Specifically why is old reflate able to work without issues and mzlib not, is it matter of time/rework or is this a implementation limitation that is to be expected from mzlib, as compare to reflate and will not change in future?

[elit]

I cannot inflate zst(zstd) file from Paper Mario Origami King with xtool. With zstd.exe it does unpack just fine:

zstdproblem.png

I was so desperate that I actually downloaded and compiled ALL available zstd versions from github! I am attaching those dll's, zstd.exe and a .zst file itself:
zstdproblem.7z

PS(could it be due to window size - 48k ?)

[Razor12911]

Originally Posted by Gehrman

https://github.com/emoose/re4-resear...build-0.1a.zip

Sorry for late response but here's an example.

in the generate folder, you'll get the example of how database would be created.

xtool.exe generate -mre4lfs "game\*" "game\*" lfs.xtl

You'll notice that the unpacked data and the original data is the same, that's because this tool accepts game files as they are and doesn't need to extract additional streams.

Note: Chunk size is should be the same size as the largest stream/file

then in precompress folder is where you need to transfer the generated database file from generate folder to precompress folder (lfs.xtl)

Look in xtool.ini to see how the program should be configured based on the information you provided.

[re4lfs]
Encode=re4lfs.exe <filein>.lfs <fileout>.pack
Decode=re4lfs.exe <filein>.pack <fileout>.lfs

Results of the samples you provided

Compressed 5 files, 2,853,066 => 6,511,766 bytes. Ratio 228.24%
Compression time: cpu 0.02 sec/real 1.06 sec = 1%. Speed 2.70 mB/s

[Razor12911]

Originally Posted by elit

Thanks, so on more technical ground what is to be expected going forward regarding this? Specifically why is old reflate able to work without issues and mzlib not, is it matter of time/rework or is this a implementation limitation that is to be expected from mzlib, as compare to reflate and will not change in future?

there are several implementations of the deflate algorithm, it's very flexible. Think of using lzma to compress something, obviously there are compression levels that it comes with 1-9. -mzlib implementation is based around this idea that a user who uses this would have used a common setting such as compression level so for precompression it will decompress the data and then try to recompress it using these common levels, but of course lzma has methods that people use for other purposes like for stronger compression such as "lzma:ultra:a1:mfbt4:d158m: fb273:mc1000000000:lc8" mzlib cannot process such, similarly to deflate streams in png images as the way deflate set in such a way that it's good compressing pixels of an image but also make it quicker for loading/viewing. Reflate and preflate have no problem processing these as they used an advanced method for precompressing them.

You might say that mzlib is therefore useless, actually it still has its uses. mzlib is faster than both reflate and preflate because of its simple implementation and incurring no overhead furthermore mzlib produces better results because there is no additional header information file produced hence why I've preached several times to mix both zlib with reflate or preflate, if zlib fails to process the stream it just passes the stream to reflate/preflate for it to process it.

mzlib:
Compressed 1 file, 86,347,072 => 444,743,922 bytes. Ratio 515.07%
Compression time: cpu 0.16 sec/real 13.37 sec = 1%. Speed 6.46 mB/s

mreflate,l9:
Compressed 1 file, 86,347,072 => 445,179,589 bytes. Ratio 515.57%
Compression time: cpu 0.14 sec/real 14.33 sec = 1%. Speed 6.03 mB/s

mpreflate:
Compressed 1 file, 86,347,072 => 366,023,129 bytes. Ratio 423.90%
Compression time: cpu 0.14 sec/real 10.25 sec = 1%. Speed 8.42 mB/s

mzlib+reflate,l9:
Compressed 1 file, 86,347,072 => 444,941,384 bytes. Ratio 515.29%
Compression time: cpu 0.06 sec/real 11.74 sec = 1%. Speed 7.35 mB/s

mzlib+preflate:
Compressed 1 file, 86,347,072 => 444,744,743 bytes. Ratio 515.07%
Compression time: cpu 0.11 sec/real 11.23 sec = 1%. Speed 7.69 mB/s

As you can see, zlib perform faster when alone but better when you hook it up with reflate/preflate and preflate fails to process some streams. I've given people options, all up to them how they use xtool.

Originally Posted by elit

I cannot inflate zst(zstd) file from Paper Mario Origami King with xtool. With zstd.exe it does unpack just fine:

Attachment 31395

I was so desperate that I actually downloaded and compiled ALL available zstd versions from github! I am attaching those dll's, zstd.exe and a .zst file itself:
Attachment 31394

PS(could it be due to window size - 48k ?)

use --verbose mode to see which levels are producing the closest result to the original stream.

[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 17101) using l1 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 16535) using l2 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 16351) using l3 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15784) using l4 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15711) using l5 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15493) using l6 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15462) using l7 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15462) using l8 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15444) using l9 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15442) using l10 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15438) using l11 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15437) using l12 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15428) using l13 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14817) using l14 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14733) using l15 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14689) using l16 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l17 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l18 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l19 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l20 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l21 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l22 has failed

level 17 is close but level 19 is commonly used, then set -mzstd,l19

[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l19 has failed
[0] - Patched stream at 0000000000000000 (14692 >> 14688) [28] successfully

Compressed 1 file, 14,692 => 48,239 bytes. Ratio 328.34%
Compression time: cpu 0.00 sec/real 0.55 sec = 0%. Speed 0.03 mB/s

Used libzstd130.dll

[elit]

Originally Posted by Razor12911

level 17 is close but level 19 is commonly used, then set -mzstd,l19

Thank you Razor! But wait, why it did not tried that level automatically? Is xtool trying only some levels here as well? Like up to 9, maybe due to speed?

If that is the case, may I suggest that it try to do all of them - up to certain mb of data(user defined). For instance first 100-200mb all levels and if nothing found drop to only speedier levels as is now etc.?

[Razor12911]

Dude it tried all levels, look at the log via verbose (level 1-22) and none of them produced crc perfect stream and you can even see that with level 19 the original stream was 14692 which was decompressed to 48079 and recompressed to 14688.

Xtool applies patches using xdelta only if you have specified what level was used else for zstd as an example which has 22 levels, each stream would have to be recompressed 22 times and 22 patches will exist and then how would xtool decide what level was used, by looking at which one produced the smaller diff file? because if you look at the log again level 17, 18, 19, 20, 21 and 22, yep 6 different levels produced the same result so how would xtool decide what the best level is? Even if you do it for the first 100-200mb, you may even find that zstd compression on some data even taps out at level 10 and any level higher than this will produce the same result. Xtool will still be confused af as to which level is the best which is why I left it to the user.

Xtool in such a scenario will just trust the user specified level and only help out if any of the streams fail to be properly processed by applying xdelta otherwise it will not apply any patches.

Edit: Verbose mode was added just so that the user can see why xtool fails and if there are patterns, like for example this stream "14692 >> 48079 >> 14688", with 4 bytes difference, I assume that all the streams in this game will show a pattern of 4 bytes also missing, which can mean when the game was compressed the developers turned on checksum mode for zstd, meaning a hash sized 4 byte was added for all the streams just to verify if the data is not corrupt in anyway and xtool uses zstd in default mode meaning checksum option is disabled hence why they produce 4 bytes less.

[Gehrman]

Originally Posted by Razor12911

Sorry for late response but here's an example.

in the generate folder, you'll get the example of how database would be created.

xtool.exe generate -mre4lfs "game\*" "game\*" lfs.xtl

You'll notice that the unpacked data and the original data is the same, that's because this tool accepts game files as they are and doesn't need to extract additional streams.

Note: Chunk size is should be the same size as the largest stream/file

then in precompress folder is where you need to transfer the generated database file from generate folder to precompress folder (lfs.xtl)

Look in xtool.ini to see how the program should be configured based on the information you provided.

[re4lfs]
Encode=re4lfs.exe <filein>.lfs <fileout>.pack
Decode=re4lfs.exe <filein>.pack <fileout>.lfs

Results of the samples you provided

Compressed 5 files, 2,853,066 => 6,511,766 bytes. Ratio 228.24%
Compression time: cpu 0.02 sec/real 1.06 sec = 1%. Speed 2.70 mB/s

Thank Razor12911
But strangely, it does not work with files larger than 15.9 MB.
this sample file

[Razor12911]

Originally Posted by Razor12911

Note: Chunk size is should be the same size as the largest stream/file

generate -c##

[Gehrman]

Originally Posted by Razor12911

generate -c##

good work

@echo off
xtool.exe generate -mre4lfs -c512mb "game\*" "game\*" lfs.xtl 
pause

Thank you for this wonderful tool

[elit]

@Razor now I understand, thank you! Its clear that internal knowledge of xtool is still important. All make sense now although it also means my "lz scan" cannot be a simple fire & forget proof for all corner cases(well it could but I feel it would be too much to go with each level separately within the script + all the lib versions etc., maybe compiled binary with proper output report would do). It is fine though, as long as users are aware of this.

-------------------------------------------------------------------------------------------------------------------------------

Unrelated to above, I want to give one advice to all users regarding zstd(possibly lz4 as well). For best results it may be important to try all lib versions even if you found one working. During my switch games repacking I found for example that nearby versions and sometimes far versions may get better result(or pickup again) and by results not only I mean size but also speed. ZSTD have a habit to slow down to crawl(~1mb/s) when used on wrong lib version or can have as much as 2-3x speed difference on nearby versions where all of them can inflate very close to!

I dont have data to show you anymore, but just to imagine it was something like this. Lets say we have a random file of 8Mb, then:

libzstd130.dll > 10000k > 6min
libzstd131.dll > 13000k > 5min
libzstd132.dll > 16000k > 3min
libzstd133.dll > 16400k > 1min
libzstd134.dll > 16500k > 20s
libzstd135.dll > 16550k > 15s
libzstd136.dll > 8000k > 4min (back to original size)
...                           (here all following versions did nothing)
libzstd144.dll > 16200k > 1min
libzstd145.dll > 16620k > 13s
libzstd146.dll > 8700k > 4min

This is just example, not exact but yes something like that actually happened.

[elit]

Maybe I found a way to pinpoint zstd level. After you scan with 'verbose' you can see there are several levels same. However, using ':l17' to ':l22'(without 'verbose') separately further reveal which level is real. Only with l19 it shows this:
foundzstdlevel.png
All other levels remain '0/1' even if they show same repacked size in '--verbose' scan. After this it's only matter of figuring out lib version. In this case versions working were 114, 120, 130 and 131. Probably on bigger files time would start showing difference, but I have seen these giving same result before(but also not always) so once they match, any of them should be safe.