XTool 2020 (Main Project)

[dixen]

Originally Posted by Razor12911

could you try without deduplication, too many points where an error could occur.

Of course...waiting
UPD.
Yes without dedup - the test was completed successfully and xt used about 500-600 mb RAM (XT20 R5)

Tested 11 files, 20,205,910,501 => 21,896,534,144 bytes. Ratio 92.28%
Testing time: cpu 68.59 sec/real 365.42 sec = 19%. Speed 59.92 mB/s
All OK

[elit]

You may want to be interested in this.

I was trying to inflate cpk file from Diablo 3 from nintendo switch. Normaly these are crilayla but gfs detected lz. So I tried old ztool, some older xtool(date modified says 26.april 2019) and xtool2009R3.

For reference I tried single file "Act1.cpk" of 814mb size. GFS detected potential of inflation of up to ~1800mb.

Ztool could not inflate past few 10's of mb if any, regardless of setting like m2, m3 etc.

Later xtool2009R3 depended on codec, most were at about 100mb extra only, except reflate which did go to 1.1gb. However final packed size after srep+xlzma was about same which make me think of possible huge overheat?

Finally, that older xtool with :high setting was able to inflate to.. 1.7gb!!
I tried to compress that and got about ~40mb better compression.

General final compression after codecs was around 740mb+, this one went to ~704mb. I wonder if this was real inflated data to 1.7gb or too much overheat. I must also say that older xtool was able to do it only with :high option which unlike ztool is not even know or documented and probably unofficial. Also it took very long time compare to say ztool:high, it seems to work differently.

I have uploaded the file act1.cpk for further research if you are interested:
https://megaup.net/1zEg1/Act1.cpk

[amin fear]

Is there any incompatibility between "CIU 3.x+ZTOOL+XTOOL" extraction process & new AMD CPU series ?

Anybody have tested ZTOOL with below CPU ?

AMD Ryzen™ 9 3900X "12-Core 3.8 GHz" (4.6 GHz Max Boost) Socket AM4 105W

My installer created with "CIU 3.x+SrepInsidex64+ZToolx64+LOLZx64" works pretty well on ALL of mid range CPUs(Corei3+Corei5+Corei7) but when extracting on NEW AMD Ryzen™ 9 3900X, installer stuck on "0.0" percent progress bar & "ZTool.exe" won't launch on task manager!!!

Is "ZTool.exe" incompatible with new 12-Core CPUs ?

Thanks @Razor12911

[KaktoR]

Did you try to unpack with bat file`? If this works then the problem could be inside CIU code somewhere.

[Razor12911]

my tools can handle up to 256 threads, if you have problems it's mainly because you're using ztool instead of the newer xtool which had MT bugs fixed.

[amin fear]

Originally Posted by KaktoR

Did you try to unpack with bat file`? If this works then the problem could be inside CIU code somewhere.

Thanks @KaktoR for batch script! i tried that unpack script but it gives me some errors ! can you please take a look at them :

I have a lot of HDD space [200GB] ,but still gives me errors !

https://imgur.com/a/v2DOBcn

I have compressed my files in below names & put them in "OUTPUT" folder of DiskSpan_UNPACK script:

Data1.Bin.001
Data1.Bin.002
Data1.Bin.003

using below compression method :

pzlib:m2+srep:m3f+lolz:dtb1:mtt1:mt4:d128m:mc1023

I only modified archive name in "_DiskSpan_UNPACK (Test Only).bat" , should i change other parameters ? for example compression switch !

[KaktoR]

Try this

Regedit
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System
EnableLUA > 0
ConsentPromptBehaviorAdmin > 0
FilterAdministratorToken > 0

Restart Computer

[Razor12911]

Update available

Changes

- added temporary libdunia codec (thanks ProFrager and FitGirl)

[Masquerade]

Thanks Razor / ProFrager / FitGirl for the new dunia codec!

Here's a test on primal_main.dat (6.32GB)

Compressed 1 file, 6,796,379,826 => 8,579,219,400 bytes. Ratio 126.23%
Compression time: cpu 6.58 sec/real 299.11 sec = 2%. Speed 22.72 mB/s
All OK

Working fast and good.

[dixen]

WD Legion

london.dat
shadersobj.dat
common.dat
patch.dat
london_preload.dat
london_cache.dat

13 gb > 17.6 gb

Log not saved)) Sorry

[Masquerade]

Having a few problems decompressing with the fcp option, I have tried a few different unpackcmds:

[External compressor: fcp]
header = 0
unpackcmd = xtool decode --dbase=fcp.xtl - - <stdin> <stdout>

[External compressor: fcp]
header = 0
unpackcmd = xtool decode - - <stdin> <stdout>

[External compressor: fcp]
header = 0
unpackcmd = xtool decode -mfcp - - <stdin> <stdout>

Yes, fcp.xtl and liblz4.dll are with xtool.exe

Errors like:

G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed   0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed   0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed   0%
ERROR: write error (disk full?) in compression algorithm fcp

This is just on 1 file with no extra compressors.

Is there something wrong with my unpackcmd?

[Masquerade]

Update: srep+lolz is smaller than fcp+srep+lolz

[shazzla]

Guys ,how can i set the

--dedup=FC.bin

parameter in -m chain ?
is it possible ?

[elit]

So I was dealing with crilayla in Persona 5 cpk's. Normally for these I just use bms but for the first time I got errors in some archives. So i tried your older xtool that still had it(in fact that's why I kept it). And man.. this thing is fantastic! I bring it because you said you dropped it because of speed. Well I don't know if I can change your mind but let me try.

I tested 4.4gb cpk with :t4 on my 4960k 4.2ghz, which is pretty normal today. Inflating took about 16min and re-encoding about 14min. This gives the speed of around 4.5-5.5mb\s. Look, this is not bad at all! There are lot of crazy people out there abusing lzma/lolz with outright stupid settings like mc1000+ that can slow you to crawl with no benefits, yet unlike those this tool is extremely beneficial.

Maybe you meant that for decompression its too slow. Well, repacking cpk back with single threaded bms is just as slow if not more + all the work around it, I did it this way till now. I even had to use xdelta because bms compressor write something into header that I have not seen in single game yet(other than zeroes, I think it was offset 16 and ~6 bytes long string).

So your cri xtool is excellent, work well and have reasonable speed including de/compression - considering alternatives. Also future CPU cores are only going to be more, not less.

This is my 2 cents, of course you do what you feel like. But there is no other cri tool like this and compression is probably used second most often after deflate. There already are plenty of deflate tools and libraries, even original ztool is just perfectly fine for this. Its those other codecs that would be beneficial. Crilayla, maybe also lzss and certainly yaz0 to name few.

In fact you already completed the work, why abandon it? So for the love of god I beg you, put it back in latest version!

PS(Zstd and lz4 are temporary and will stop working after time because of their retarded design. Not sure about lzo(that is actually a question, why is standard lzo not working in dunia and unreal engines and need special treatment? I thought its a stable design like zlib?). But crilayla, lzss, yaz0 and such should remain compatible.

[elit]

-grittibanzli inflate completely randomly. Sometimes tested 20mb inflated zip file is 34mb, other times 31.6mb, another time 33mb atd. exactly same setting and just repeating same command on console.

-zlib codec did not inflated zip file at all!(I thought it was supposed to handle deflate?)

-preflate and reflate both did to same ~27mb, but preflate took ~6sec while reflate did it in ~1sec.

Looks like good old reflate is still the best zlib codec to stick.


EDIT(and ztool is also able to inflate to 27mb without :high or :m2 !) <<UPDATE(BS, scrap that claim, need m2 or high indeed)
EDIT2(ok I checked manual again and learned about zlib+reflate combination benefits. Maybe this is best way. It took ~3sec compared to ~1sec on deflate alone though. Not sure how big those benefits are in real application yet.)
EDIT3(have to explicitly state 'mb' in -c option instead of just 'm', e.g. -c128m vs -c128mb. No big deal but used to ztool, was more flexible. PS: I am really liking the idea of zlib+reflate in one go, before I used to pass twice in test to find out which one to use.)