XTool 2020 (Main Project)

[Razor12911]

Update available

Changes

- fixed command line parser bug
- updated library support

[github]

Hi thanks for the great work.

With xtool0.12 I have

[External compressor:xzlib,xZLib]
header = 0
packcmd = "pre\xt12\xtool" erecomp:t75p,c128m:zlib - - <stdin> <stdout>
unpackcmd= pre\xt12\xtool drecomp:t4 - - <stdin> <stdout>

On a testing file set mostly zip and 7z files, it could precomp it well

;xzlib:
;Compressed 139 files, 822,152,142 => 1,096,061,919 bytes. Ratio 133.32%
;Compression time: cpu 0.81 sec/real 31.35 sec = 3%. Speed 26.23 mB/s

Then I could add srep+FL2 to compress it with a good ratio.

With the recent release xtool 2020 from xtool_0.3.8.7z,

with a setting of

[External compressor:xzlib2,xZLib2]
header = 0
packcmd = "pre\XTool_2020\XTool" precomp:zlib:c128m,t75p - - <stdin> <stdout>
unpackcmd= "pre\XTool_2020\XTool" decode:t4 - - <stdin> <stdout>

It will not do the similar precomp process, output is

;xzlib2
;Compressed 139 files, 822,152,142 => 822,151,338 bytes. Ratio 100.00%
;Compression time: cpu 0.53 sec/real 2.23 sec = 24%. Speed 369.24 mB/s

Do I have to adjust/add the parameter to have the similar results?

Edit:

I guess the default values for zlib might have changed. And on some aspects the new version is performing much more like a normal average setting.


For a set of .dat files 800MB, tried to compressing with arc

srep+FL2 compressed it to 414300KB, takes 32 seconds.

xtool_0.12 erecomp:t75p,c128m:zlib+srep+FL2 compressed to 412000KB takes a big 55 seconds. the zlib codec here is working very hard on something and found a little bit to more to compress further. But it added 23 seconds up on (srep+FL2)

xtool_0.12 erecomp:t75p,c128m:zstd+srep+FL2 got 414000KB takes 36 seconds, it adds 4 seconds up on (srep+FL2)

xtool_0.3.8 erecomp:t75p,c128m:zlib+srep+FL2 got to 414000KB takes 33 seconds, it adds 1 seconds up on (srep+FL2)

xtool_0.3.8 erecomp:t75p,c128m:zstd+srep+FL2 got to 414000KB takes 33 seconds , it adds 1 seconds up on (srep+FL2)

[Razor12911]

Update available

Changes

- fixed future stream bug

@github

The command line has changed, check the example to see how it's used.

Furthermore, the old xtool used reflate without the knowledge of the user. The new one doesn't so if you have a scenario where the old xtool gives better results, then you may need to combine zlib with reflate or preflate.

[DiCaPrIo]

small test on WDL

[github]

Thank you. So with xtool 2020 I should use zlib+reflate -d or zlib+preflate -d
on command line with this it does detect many streams

.\xtool.exe precomp -mzlib+reflate -d3 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
14992 streams
.\xtool.exe precomp -mzlib+preflate -d5 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
2603 streams

how to I pass the -d level in the arc.ini?
does xtool0.12/2020 support the writing of {options} in arc.ini?

I tried

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1:d3 - - <stdin> <stdout>

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1,d3 - - <stdin> <stdout>

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1 {options} - - <stdin> <stdout>

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1 -d3 - - <stdin> <stdout>

Those doesn't make any difference to the one without d3 option

[Razor12911]

are you precompressing something that requires depth level to be set to 3 or higher? as for choosing between reflate or preflate, you need to know how each works as they have their advantages and disadvantages.

.\xtool.exe precomp -mzlib+reflate -d3 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
14992 streams
.\xtool.exe precomp -mzlib+preflate -d5 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
2603 streams

which should explain their differences seen here. What I would point out as the main difference, reflate supports streams with no tail while preflate doesn't so if a special deflate encoder was used or if a stream had been cut off, preflate will be unable to process such streams but reflate is able to handle these.

as for {option} via freearc, I'd first say you need to understand the command line syntax of xtool. The old xtool and new xtool use different syntax.

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp -mzlib -c32mb -t100p - - <stdin> <stdout>

Ideally this would become

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp {option} - - <stdin> <stdout>

then used as -mxtool:mzlib:c32mb:t100p from freearc.

[github]

Thank you for the example.

I have it setup up like this

;zlib zstd,oodle,reflate,preflate
;calling example: xtool:mzlib or xtool:mzlib:c128m:t100p-1 not xtool:mzlib+zstd
[External compressor:xtool]
header = 0
default = -c128m -t100p-1
packcmd = "pre\XTool_2020\XTool" precomp {options} - - <stdin> <stdout>
unpackcmd= "pre\XTool_2020\XTool" decode:t4 - - <stdin> <stdout>

; calling example zlib:c128m:t100p-1
[External compressor:zlib,zstd,oodle,reflate,preflate]
header = 0
default = -c128m -t100p-1
packcmd = "pre\XTool_2020\XTool" precomp -m{compressor} {options} - - <stdin> <stdout>
unpackcmd= "pre\XTool_2020\XTool" decode:t4 - - <stdin> <stdout>

However both setups can not to pass in like xtool:mzlib+zstd since + is parsed by freearc, but I guess it's rarely needed to pass in to methods.

I asked about passing in the -d option, since there are up to 10, just saying to test it so I used 3 or 5, and in my case I do need to use -d3 or -d5 to get the same result as xtool 0.12

[Razor12911]

Originally Posted by github

However both setups can not to pass in like xtool:mzlib+zstd since + is parsed by freearc, but I guess it's rarely needed to pass in to methods.

This true.

I asked about passing in the -d option, since there are up to 10, just saying to test it so I used 3 or 5, and in my case I do need to use -d3 or -d5 to get the same result as xtool 0.12

I wonder though, A depth of 3-5. I asked before if there is such data you have on hand because this is basically the same as having data, compress it using zlib, the output, compress that using zlib, do this at least 5 times. That's what depth is.

A depth of 1-2 is usually enough on everyday files such as pictures and documents as these are usually shipped in a zip file, that zip file is compressed using deflate and if it contains pdfs or png images then, that's when you need to use depth otherwise, don't touch it as you'll be increasing precompression time unnecessarily and have no gains in ratio.

[Masquerade]

Hello Razor, is there anything that can be done to cap the memory usage of xtool at a certain amount?

I am aware of a cmem variable in arc.ini packcmd (only small knowledge) however I was wondering if there's any other ways about this.

I find on larger workloads/workloads with high amounts of streams, xtool fills up my RAM and completely crashes my PC.

I am working with 16GB memory. Is there anything that can be done without changing chunk size?

Currently I'm capping threads and using $$arc$$ instead of stdio.

[Razor12911]

-lm

[Masquerade]

thank you for implementing such a feature, I thought it got removed when the project was started (I remember it being in 2019 version).

Thanks again.

Compressed 1 file, 12,630,693,406 => 54,088,313,477 bytes. Ratio 428.23%
Compression time: cpu 66.05 sec/real 1078.31 sec = 6%. Speed 11.71 mB/s
All OK

[Snake288]

Hello Razor,Masquerade is there anything that can be done to cap the memory usage of xtool at a certain amount?
Can you give an example of how we will apply this method

[Razor12911]

Originally Posted by Snake288

Hello Razor,Masquerade is there anything that can be done to cap the memory usage of xtool at a certain amount?
Can you give an example of how we will apply this method

you can't cap the memory usage of the program, you can only reduce it. When you are compressing, usually each thread gets its own chunk. Using -lm parameter makes all thread use 1 chunk. As for memory usage, that's highly dependent on the user and the data they are precompressing, if you set very high chunk sizes, be prepared for high memory usage. When decompressing, xtool tries to not use more than 512 MB ram automatically unless if there are large streams or if the recompressing library requires more memory.

[doofoo24]

did quick test on mad max with/out --dedup=xtool.bin to se mem usage...
with --dedup size to 55gb
without --dedup size 57.1gb
but memory dec the same around 6gb.
*note only test with mzlib+srep

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib -c128mb -t12 --dbase --dedup=xtool.bin - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>


[External compressor:srep]
header = 0
packcmd = srep.exe -m3f -d1g -a2 $$arcdatafile$$.tmp - <stdout>

[FitGirl]

Mad Max is a bad example for dedup testing. They use really huge deflate chunks with level 1 compression, so the amount of dupe chunks is not very big, unlike the duplicated data in those chunks when decompressed, that's why srep is good solution here.