Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #301  
Old 31-01-2021, 03:38
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,649
Thanks: 2,028
Thanked 10,446 Times in 2,214 Posts
Razor12911 is on a distinguished road
Update available

Changes

- fixed command line parser bug
- updated library support
Reply With Quote
The Following 5 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (31-01-2021), ffmla (31-01-2021), Gehrman (22-04-2021), ScOOt3r (31-01-2021), vint56 (31-01-2021)
Sponsored Links
  #302  
Old 03-02-2021, 14:53
github github is offline
Registered User
 
Join Date: Jan 2021
Location: Australia
Posts: 23
Thanks: 1
Thanked 3 Times in 3 Posts
github is on a distinguished road
Hi thanks for the great work.

With xtool0.12 I have

Quote:
[External compressor:xzlib,xZLib]
header = 0
packcmd = "pre\xt12\xtool" erecomp:t75p,c128m:zlib - - <stdin> <stdout>
unpackcmd= pre\xt12\xtool drecomp:t4 - - <stdin> <stdout>
On a testing file set mostly zip and 7z files, it could precomp it well
Quote:
;xzlib:
;Compressed 139 files, 822,152,142 => 1,096,061,919 bytes. Ratio 133.32%
;Compression time: cpu 0.81 sec/real 31.35 sec = 3%. Speed 26.23 mB/s
Then I could add srep+FL2 to compress it with a good ratio.

With the recent release xtool 2020 from xtool_0.3.8.7z,

with a setting of

Quote:
[External compressor:xzlib2,xZLib2]
header = 0
packcmd = "pre\XTool_2020\XTool" precomp:zlib:c128m,t75p - - <stdin> <stdout>
unpackcmd= "pre\XTool_2020\XTool" decode:t4 - - <stdin> <stdout>
It will not do the similar precomp process, output is

Quote:
;xzlib2
;Compressed 139 files, 822,152,142 => 822,151,338 bytes. Ratio 100.00%
;Compression time: cpu 0.53 sec/real 2.23 sec = 24%. Speed 369.24 mB/s
Do I have to adjust/add the parameter to have the similar results?

Edit:

I guess the default values for zlib might have changed. And on some aspects the new version is performing much more like a normal average setting.


For a set of .dat files 800MB, tried to compressing with arc

srep+FL2 compressed it to 414300KB, takes 32 seconds.

xtool_0.12 erecomp:t75p,c128m:zlib+srep+FL2 compressed to 412000KB takes a big 55 seconds. the zlib codec here is working very hard on something and found a little bit to more to compress further. But it added 23 seconds up on (srep+FL2)

xtool_0.12 erecomp:t75p,c128m:zstd+srep+FL2 got 414000KB takes 36 seconds, it adds 4 seconds up on (srep+FL2)

xtool_0.3.8 erecomp:t75p,c128m:zlib+srep+FL2 got to 414000KB takes 33 seconds, it adds 1 seconds up on (srep+FL2)

xtool_0.3.8 erecomp:t75p,c128m:zstd+srep+FL2 got to 414000KB takes 33 seconds , it adds 1 seconds up on (srep+FL2)

Last edited by github; 03-02-2021 at 15:13.
Reply With Quote
The Following User Says Thank You to github For This Useful Post:
Razor12911 (04-02-2021)
  #303  
Old 04-02-2021, 00:47
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,649
Thanks: 2,028
Thanked 10,446 Times in 2,214 Posts
Razor12911 is on a distinguished road
Update available

Changes

- fixed future stream bug

@github

The command line has changed, check the example to see how it's used.

Furthermore, the old xtool used reflate without the knowledge of the user. The new one doesn't so if you have a scenario where the old xtool gives better results, then you may need to combine zlib with reflate or preflate.
Reply With Quote
The Following 4 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (04-02-2021), dixen (04-02-2021), KaktoR (04-02-2021), ZAZA4EVER (04-02-2021)
  #304  
Old 04-02-2021, 05:03
DiCaPrIo DiCaPrIo is offline
Registered User
 
Join Date: Apr 2017
Location: Don't Know
Posts: 50
Thanks: 90
Thanked 49 Times in 30 Posts
DiCaPrIo is on a distinguished road
Wdl

small test on WDL
Attached Images
File Type: png wdl.PNG (29.7 KB, 380 views)
Reply With Quote
The Following 2 Users Say Thank You to DiCaPrIo For This Useful Post:
ffmla (04-02-2021), Razor12911 (04-02-2021)
  #305  
Old 04-02-2021, 07:56
github github is offline
Registered User
 
Join Date: Jan 2021
Location: Australia
Posts: 23
Thanks: 1
Thanked 3 Times in 3 Posts
github is on a distinguished road
Thank you. So with xtool 2020 I should use zlib+reflate -d or zlib+preflate -d
on command line with this it does detect many streams

Quote:
.\xtool.exe precomp -mzlib+reflate -d3 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
14992 streams
.\xtool.exe precomp -mzlib+preflate -d5 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
2603 streams
how to I pass the -d level in the arc.ini?
does xtool0.12/2020 support the writing of {options} in arc.ini?

I tried

Quote:
packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1:d3 - - <stdin> <stdout>

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1,d3 - - <stdin> <stdout>

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1 {options} - - <stdin> <stdout>

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1 -d3 - - <stdin> <stdout>
Those doesn't make any difference to the one without d3 option
Reply With Quote
  #306  
Old 04-02-2021, 17:57
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,649
Thanks: 2,028
Thanked 10,446 Times in 2,214 Posts
Razor12911 is on a distinguished road
are you precompressing something that requires depth level to be set to 3 or higher? as for choosing between reflate or preflate, you need to know how each works as they have their advantages and disadvantages.

Code:
.\xtool.exe precomp -mzlib+reflate -d3 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
14992 streams
.\xtool.exe precomp -mzlib+preflate -d5 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
2603 streams
which should explain their differences seen here. What I would point out as the main difference, reflate supports streams with no tail while preflate doesn't so if a special deflate encoder was used or if a stream had been cut off, preflate will be unable to process such streams but reflate is able to handle these.

as for {option} via freearc, I'd first say you need to understand the command line syntax of xtool. The old xtool and new xtool use different syntax.

Code:
[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp -mzlib -c32mb -t100p - - <stdin> <stdout>
Ideally this would become

Code:
[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp {option} - - <stdin> <stdout>
then used as -mxtool:mzlib:c32mb:t100p from freearc.
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
ffmla (04-02-2021)
  #307  
Old 05-02-2021, 20:07
github github is offline
Registered User
 
Join Date: Jan 2021
Location: Australia
Posts: 23
Thanks: 1
Thanked 3 Times in 3 Posts
github is on a distinguished road
Thank you for the example.

I have it setup up like this

Quote:
;zlib zstd,oodle,reflate,preflate
;calling example: xtool:mzlib or xtool:mzlib:c128m:t100p-1 not xtool:mzlib+zstd
[External compressor:xtool]
header = 0
default = -c128m -t100p-1
packcmd = "pre\XTool_2020\XTool" precomp {options} - - <stdin> <stdout>
unpackcmd= "pre\XTool_2020\XTool" decode:t4 - - <stdin> <stdout>
Quote:
; calling example zlib:c128m:t100p-1
[External compressor:zlib,zstd,oodle,reflate,preflate]
header = 0
default = -c128m -t100p-1
packcmd = "pre\XTool_2020\XTool" precomp -m{compressor} {options} - - <stdin> <stdout>
unpackcmd= "pre\XTool_2020\XTool" decode:t4 - - <stdin> <stdout>

However both setups can not to pass in like xtool:mzlib+zstd since + is parsed by freearc, but I guess it's rarely needed to pass in to methods.

I asked about passing in the -d option, since there are up to 10, just saying to test it so I used 3 or 5, and in my case I do need to use -d3 or -d5 to get the same result as xtool 0.12

Last edited by github; 05-02-2021 at 20:29.
Reply With Quote
  #308  
Old 06-02-2021, 13:22
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,649
Thanks: 2,028
Thanked 10,446 Times in 2,214 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by github View Post
However both setups can not to pass in like xtool:mzlib+zstd since + is parsed by freearc, but I guess it's rarely needed to pass in to methods.
This true.

Quote:
I asked about passing in the -d option, since there are up to 10, just saying to test it so I used 3 or 5, and in my case I do need to use -d3 or -d5 to get the same result as xtool 0.12
I wonder though, A depth of 3-5. I asked before if there is such data you have on hand because this is basically the same as having data, compress it using zlib, the output, compress that using zlib, do this at least 5 times. That's what depth is.

A depth of 1-2 is usually enough on everyday files such as pictures and documents as these are usually shipped in a zip file, that zip file is compressed using deflate and if it contains pdfs or png images then, that's when you need to use depth otherwise, don't touch it as you'll be increasing precompression time unnecessarily and have no gains in ratio.
Reply With Quote
  #309  
Old 10-02-2021, 11:45
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 779
Thanks: 184
Thanked 855 Times in 417 Posts
Masquerade is on a distinguished road
Hello Razor, is there anything that can be done to cap the memory usage of xtool at a certain amount?

I am aware of a cmem variable in arc.ini packcmd (only small knowledge) however I was wondering if there's any other ways about this.

I find on larger workloads/workloads with high amounts of streams, xtool fills up my RAM and completely crashes my PC.

I am working with 16GB memory. Is there anything that can be done without changing chunk size?

Currently I'm capping threads and using $$arc$$ instead of stdio.

Last edited by Masquerade; 10-02-2021 at 11:51.
Reply With Quote
  #310  
Old 10-02-2021, 11:57
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,649
Thanks: 2,028
Thanked 10,446 Times in 2,214 Posts
Razor12911 is on a distinguished road
-lm
Attached Images
File Type: png Capture3.PNG (32.8 KB, 295 views)
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
ffmla (11-02-2021), Masquerade (10-02-2021), sajmon83 (19-02-2021)
  #311  
Old 10-02-2021, 12:56
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 779
Thanks: 184
Thanked 855 Times in 417 Posts
Masquerade is on a distinguished road
thank you for implementing such a feature, I thought it got removed when the project was started (I remember it being in 2019 version).

Thanks again.

Code:
Compressed 1 file, 12,630,693,406 => 54,088,313,477 bytes. Ratio 428.23%
Compression time: cpu 66.05 sec/real 1078.31 sec = 6%. Speed 11.71 mB/s
All OK
Reply With Quote
  #312  
Old 10-02-2021, 14:19
Snake288 Snake288 is offline
Registered User
 
Join Date: Nov 2018
Location: Paris
Posts: 174
Thanks: 89
Thanked 31 Times in 25 Posts
Snake288 is on a distinguished road
Hello Razor,Masquerade is there anything that can be done to cap the memory usage of xtool at a certain amount?
Can you give an example of how we will apply this method
Reply With Quote
  #313  
Old 10-02-2021, 19:57
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,649
Thanks: 2,028
Thanked 10,446 Times in 2,214 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Snake288 View Post
Hello Razor,Masquerade is there anything that can be done to cap the memory usage of xtool at a certain amount?
Can you give an example of how we will apply this method
you can't cap the memory usage of the program, you can only reduce it. When you are compressing, usually each thread gets its own chunk. Using -lm parameter makes all thread use 1 chunk. As for memory usage, that's highly dependent on the user and the data they are precompressing, if you set very high chunk sizes, be prepared for high memory usage. When decompressing, xtool tries to not use more than 512 MB ram automatically unless if there are large streams or if the recompressing library requires more memory.

Last edited by Razor12911; 10-02-2021 at 19:59.
Reply With Quote
  #314  
Old 11-02-2021, 20:24
doofoo24 doofoo24 is offline
Registered User
 
Join Date: Nov 2016
Location: canada
Posts: 408
Thanks: 138
Thanked 460 Times in 227 Posts
doofoo24 is on a distinguished road
did quick test on mad max with/out --dedup=xtool.bin to se mem usage...
with --dedup size to 55gb
without --dedup size 57.1gb
but memory dec the same around 6gb.
*note only test with mzlib+srep

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib -c128mb -t12 --dbase --dedup=xtool.bin - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>


[External compressor:srep]
header = 0
packcmd = srep.exe -m3f -d1g -a2 $$arcdatafile$$.tmp - <stdout>
Reply With Quote
  #315  
Old 12-02-2021, 19:14
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 192
Thanks: 17
Thanked 232 Times in 125 Posts
FitGirl is on a distinguished road
Mad Max is a bad example for dedup testing. They use really huge deflate chunks with level 1 compression, so the amount of dupe chunks is not very big, unlike the duplicated data in those chunks when decompressed, that's why srep is good solution here.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 15:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 18:16



All times are GMT -7. The time now is 00:52.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com