XTool - Successor of ZTool

[ZakirAhmad]

xtool v 0.7 Tried it on a file from AC orgins

it didnt expanded.
sorry to bother oodle was missing from arc.ini setting.
this sorted the issue.

afr expaned it to 163.15% while xtool to 162.91% xtool
restored it in only 14sec while afr took 18 seconds in my system

Now my only problem is with Dunia engine. i wrote dat file address in xtool.ini still no change.

BTW is farcry primal and James Camerons avatar supported.

[KaktoR]

Yep, didn't expand with :lzo

Maybe wrong codec in arc.ini? lzo2.dll is in place

Edit: You have to use

:oodle

for Origins

DataPC.forge (ACO)

xtool v0.7 oodle: 273.088.512 Bytes => 574.853.120 Bytes
AFR:              273.088.512 Bytes => 574.849.024 Bytes

From this example xtool and AFR have pretty much the same compression speed (AFR is a little bit faster, but just a few secs)

[elit]

Originally Posted by Razor12911

Update available
In terms of the Dunia engine, I advise that you use the latest liblz4.dll to avoid some streams being skipped due to their major difference, this could be because even I myself am not sure if I used the correct parameters for recompression, if you have an idea of what is used, please comment. (I used lz4hc, level 9)

I dont know if you want to go this route, but the way I did my tests when developing lz4 packer for "Raiders of the Broken Planet" game was that I used both original and self-modified quickbms script for unpacking game archive first. I unpacked thus twice - for each.

In modified script only difference was that I disabled decompression by taking out "comtype" command and used "log" instead of "clog" to dump files raw & compressed.

So I had both compressed and decompressed individual files from archive. Then it was only matter of finding right/best cmp settings through hex editor and binary comparison(first on few files and then all) to get best setting. And even wrong(but close) settings were still compatible with 80%+ of all archive.

[ShivShubh]

Originally Posted by elit

I dont know if you want to go this route, but the way I did my tests when developing lz4 packer for "Raiders of the Broken Planet" game was that I used both original and self-modified quickbms script for unpacking game archive first. I unpacked thus twice - for each.

In modified script only difference was that I disabled decompression by taking out "comtype" command and used "log" instead of "clog" to dump files raw & compressed.

So I had both compressed and decompressed individual files from archive. Then it was only matter of finding right/best cmp settings through hex editor and binary comparison(first on few files and then all) to get best setting. And even wrong(but close) settings were still compatible with 80%+ of all archive.

So by saying all that what were you trying to say by this "I dont know if you want to go this route" ? That Razor should find the correct parameters ? I don't think he has that much of an interest in far cry 5 or wants to waste time doing all that.

[Razor12911]

Originally Posted by Sergey3695

https://yadi.sk/d/WNKW73Ht3UpyVN
data.cpk (393 mb)
NARUTO SHIPPUDEN Ultimate Ninja STORM 2

[External compressor:xprecomp]
header = 0
packcmd = xtool.exe e:precomp:c32mb,t1:crilayla - - <stdin> <stdout>
unpackcmd = xtool.exe d:precomp:c32mb,t1:crilayla - - <stdin> <stdout>

Extracting archive: data.arc
Extracting data.cpk (413123988 bytes)
ERROR: file _TEST\data.cpk failed CRC check
:confused:

i'll check :)

[Edison007]

compr:

afr:v0+srep:m5f:l512:d512m:hash-+lolz:d64m - 37,6 MB (39 463 094 bytes)
xtool+srep:m5f:l512:d512m:hash-+lolz:d64m  - 37,8 MB (39 671 049 bytes)

dec time/speed:

cls-afr, v019 (x86)
t1 - 199.28 sec; ~  5.37 mB/s
t7 -  44.07 sec; ~ 24.29 mB/s
t8 -  42.89 sec; ~ 24.96 mB/s

xtool_x86, v07
t1 - 237.07 sec; ~  4.52 mB/s
t7 -  53.84 sec; ~ 19.88 mB/s
t8 -  51.19 sec; ~ 20.91 mB/s

xtool_x64, v07
t1 - 210.85 sec; ~  5.08 mB/s
t7 -  49.35 sec; ~ 21.69 mB/s
t8 -  46.91 sec; ~ 22.82 mB/s

1'070'530'560 ->   afr: 1'767'925'585 bytes
              -> xtool: 1'768,021'857 bytes

i7-4700MQ, 8gb ram, ram-disk, win7x64.

[shazzla]

Similar happens on Agony's .PAK files(zlib,v0.7, v0.6 works good)
First file processed well,but the second one looks like simply copied.
Cant upload files because they are more than 10 gigs. :/

[elit]

Originally Posted by doofoo24

i think it may be better for Razor12911 if he develop the tool separately like pzlib/plz4/pzstd/plzo for games...
less of a headache

How exactly is that less headache? This way at least he can re-use a lot of same code/routines for each compressor instead of duplicating and maintaining separate projects with a lot of same code.

[Razor12911]

Update available

Changes

- fixed crilayla bug on naruto games
- fixed issue detecting lzo2a, lzo1c streams

@doofoo24
I have to agree with elit, It's more headache separating the codecs. An example would be ztool itself, normally when I make changes to pzlib, if it's not related to precompression itself, I had to make the same changes in plz4, plzo... sometimes I forget doing this and end up with different sources of each, in one, a bug was fixed then another bug appears in another, now there are bugs all over the codecs, different bugs, you end up trying to fix one and another pops up and so forth. The idea of putting everything together means one code, if one codec is affected, then all will be affected then I'll know the source of the bug.

Originally Posted by Edison007

compr:

afr:v0+srep:m5f:l512:d512m:hash-+lolz:d64m - 37,6 MB (39 463 094 bytes)
xtool+srep:m5f:l512:d512m:hash-+lolz:d64m  - 37,8 MB (39 671 049 bytes)

dec time/speed:

cls-afr, v019 (x86)
t1 - 199.28 sec; ~  5.37 mB/s
t7 -  44.07 sec; ~ 24.29 mB/s
t8 -  42.89 sec; ~ 24.96 mB/s

xtool_x86, v07
t1 - 237.07 sec; ~  4.52 mB/s
t7 -  53.84 sec; ~ 19.88 mB/s
t8 -  51.19 sec; ~ 20.91 mB/s

xtool_x64, v07
t1 - 210.85 sec; ~  5.08 mB/s
t7 -  49.35 sec; ~ 21.69 mB/s
t8 -  46.91 sec; ~ 22.82 mB/s

1'070'530'560 ->   afr: 1'767'925'585 bytes
              -> xtool: 1'768,021'857 bytes

i7-4700MQ, 8gb ram, ram-disk, win7x64.

I wonder if there is still room for improvement.

[Razor12911]

Originally Posted by doofoo24

@Razor12911 xtool 0.7 seems to skip file's test it on ME A with zstd and before with lzo on ac 1...
xtool 0.6 work fine...
any idea ?
i will test with 0.8...

I didn't touch zstd codec in between the updates

ztool_08

Process ID       : 12240
Thread ID        : 19080
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          24.500s
Kernel Time      :           0.734s
Process Time     :          25.234s
Clock Time       :           7.764s

Working Set      :          193016 KB
Paged Pool       :             114 KB
Nonpaged Pool    :              13 KB
Pagefile         :          314972 KB
Page Fault Count : 311084

IO Read          :           43332 KB (in             683 reads )
IO Write         :          101353 KB (in            1584 writes)
IO Other         :               3 KB (in             178 others)

ztool_06

Process ID       : 12748
Thread ID        : 14488
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          24.890s
Kernel Time      :           1.125s
Process Time     :          26.015s
Clock Time       :           8.160s

Working Set      :          177480 KB
Paged Pool       :             115 KB
Nonpaged Pool    :              12 KB
Pagefile         :          229352 KB
Page Fault Count : 316978

IO Read          :           43262 KB (in             677 reads )
IO Write         :          101353 KB (in            1584 writes)
IO Other         :               5 KB (in             270 others)

[Razor12911]

Originally Posted by Edison007

compr:

afr:v0+srep:m5f:l512:d512m:hash-+lolz:d64m - 37,6 MB (39 463 094 bytes)
xtool+srep:m5f:l512:d512m:hash-+lolz:d64m  - 37,8 MB (39 671 049 bytes)

dec time/speed:

cls-afr, v019 (x86)
t1 - 199.28 sec; ~  5.37 mB/s
t7 -  44.07 sec; ~ 24.29 mB/s
t8 -  42.89 sec; ~ 24.96 mB/s

xtool_x86, v07
t1 - 237.07 sec; ~  4.52 mB/s
t7 -  53.84 sec; ~ 19.88 mB/s
t8 -  51.19 sec; ~ 20.91 mB/s

xtool_x64, v07
t1 - 210.85 sec; ~  5.08 mB/s
t7 -  49.35 sec; ~ 21.69 mB/s
t8 -  46.91 sec; ~ 22.82 mB/s

1'070'530'560 ->   afr: 1'767'925'585 bytes
              -> xtool: 1'768,021'857 bytes

i7-4700MQ, 8gb ram, ram-disk, win7x64.

Ran a test using the samples I got from forum. Very small input but got these results:

Input: 278 MB (291,605,504 bytes)
DataPC_Map_Menu.forge (ACI)
DataPC_LoadingRoom2.forge (ACI)
DataPC.forge (ACII)
DataPC.forge (ACIV)

command line:
xtool_x86: d:precomp:t# %1.out1 %1.res
afr19_x86: d -v0 -t# %1.out2 %1.res

Thread(1) = 1 result:
xtool: 17.898s, 18.050s, 17.901s, 18.056s, 17.942s
afr19: 17.853s, 17.875s, 18.173s, 17.865s, 17.879s

Thread(s) = 2 result:
xtool: 9.120s, 9.156s, 9.035s, 9.067s, 9.008s
afr19: 9.394s, 9.356s, 9.318s, 9.248s, 9.291s

Thread(s) = 3 result:
xtool: 6.325s, 6.412s, 6.249s, 6.412s, 6.264s
afr19: 6.603s, 6.511s, 6.696s, 6.517s, 6.691s

Not sure in terms of results here, I ran test on ramdisk using my old core 2 extreme, I didn't include t4 since those are mostly irrelevant.

As for final output after compression, I'd be taking a gamble when I say it's headers since they include stuff like what codec was used since xtool is not just made for one game engine but I'll check in terms of the number of streams found by xtool vs afr

[Razor12911]

Originally Posted by doofoo24

tried xtool 0.8 and 0.7 on multiple files at once like shazzla First file processed well,but the second one looks like simply copied.
lz4 / lzo /zlib

upload files and command line used

[pakrat2k2]

ALWAYS choose disable smilies, when posting/ replying ( checkbox below message box )

pain to have to come in & clean up posts & quotes.

[Edison007]

Not sure in terms of results here

It's normal, because i using differents decode methods in cls and exe. In cls i use some tricks, which allow me to get a higher speed. exe-decoder, yes, slower.

I'll check in terms of the number of streams found by xtool vs afr

I think, that numbers of streams found is the same. But maybe in AFR the output container is more successfully made

I wonder if there is still room for improvement.

Yep, but I'm not interested in this engine anymore.

[KaktoR]

Originally Posted by Edison007

Yep, but I'm not interested in this engine anymore.

So, what's your next clue?