Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #166  
Old 18-07-2020, 09:25
doofoo24 doofoo24 is offline
Registered User
 
Join Date: Nov 2016
Location: canada
Posts: 408
Thanks: 138
Thanked 456 Times in 227 Posts
doofoo24 is on a distinguished road
tested on just cause2 arc files...
default setting nothing change "ProcProfile64.exe xtool.exe precomp -mzlib -c16mb -t100p-1 %1 %1.out"
system ryzen 3900x with 32gb
*is there arc.ini setting ?
Attached Images
File Type: png Untitled.png (22.4 KB, 264 views)
Reply With Quote
The Following User Says Thank You to doofoo24 For This Useful Post:
Razor12911 (18-07-2020)
Sponsored Links
  #167  
Old 18-07-2020, 13:12
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 255
Thanks: 304
Thanked 292 Times in 135 Posts
dixen is on a distinguished road
Rayman Orgins
bundle_pc.pak (1.9 gb)

Quote:
Process ID : 8952
Thread ID : 4228
Process Exit Code: 0
Thread Exit Code : 0

User Time : 149.796s
Kernel Time : 4.375s
Process Time : 154.171s
Clock Time : 74.923s

Working Set : 372312 KB
Paged Pool : 115 KB
Nonpaged Pool : 12 KB
Pagefile : 462572 KB
Page Fault Count : 147577

IO Read : 2046448 KB (in 31978 reads )
IO Write : 3038303 KB (in 47474 writes)
IO Other : 6 KB (in 297 others)
PS. tested on my 2nd pc (fx-4100, 2 cores, 4 threads).
Reply With Quote
The Following User Says Thank You to dixen For This Useful Post:
Razor12911 (18-07-2020)
  #168  
Old 18-07-2020, 16:05
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,942 Times in 2,140 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by doofoo24 View Post
*is there arc.ini setting ?
Code:
[External compressor:xzlib]
header    = 0
packcmd   = xtool.exe precomp -c16mb -t100p-1 -mzlib - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p-1 - - <stdin> <stdout>
here's preflate test, test on all zlib compressed games + dishonored and doom
Attached Files
File Type: 7z xtool_preflate_test.7z (892.4 KB, 39 views)
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
COPyCAT (23-08-2020), doofoo24 (18-07-2020), Harsh ojha (18-07-2020)
  #169  
Old 18-07-2020, 17:04
doofoo24 doofoo24 is offline
Registered User
 
Join Date: Nov 2016
Location: canada
Posts: 408
Thanks: 138
Thanked 456 Times in 227 Posts
doofoo24 is on a distinguished road
tested on Dishonored2 with c16m and 512m and set -t100p-1...
work fine on ryzen 3900x and 32gb use all core and ram...

42.1gb to 57.2gb with c16mb
42.1gb to 59.8gb with c512mb

*used on all game file...

tested unpacking with inno
i get ERROR: archive data corrupted...
Reply With Quote
The Following User Says Thank You to doofoo24 For This Useful Post:
Razor12911 (18-07-2020)
  #170  
Old 18-07-2020, 19:09
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,942 Times in 2,140 Posts
Razor12911 is on a distinguished road
I must have messed up stdio mode during decoding but here is the fixed version.
Attached Files
File Type: 7z xtool_preflate_test_fix.7z (891.5 KB, 55 views)

Last edited by Razor12911; 18-07-2020 at 19:12.
Reply With Quote
The Following 5 Users Say Thank You to Razor12911 For This Useful Post:
COPyCAT (23-08-2020), dixen (19-07-2020), doofoo24 (19-07-2020), Harsh ojha (18-07-2020), shazzla (19-07-2020)
  #171  
Old 19-07-2020, 02:58
doofoo24 doofoo24 is offline
Registered User
 
Join Date: Nov 2016
Location: canada
Posts: 408
Thanks: 138
Thanked 456 Times in 227 Posts
doofoo24 is on a distinguished road
now dec work super fast but i noticed it cpu usage around 50% to 70% of threads unlike compress use 90% to 100% when set to

unpackcmd = xtool.exe decode -t100p-1 - - <stdin> <stdout>

disk write using ssd nvme mp510 write after burst 400 to 500 mb/s...
nvme bottleneck ?
need ps5 ssd to get the max
Reply With Quote
  #172  
Old 19-07-2020, 16:42
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,942 Times in 2,140 Posts
Razor12911 is on a distinguished road
you noticed how much memory xtool uses when compressing compared to when decompressing? that's why. xtool uses less memory when decoding and because of this it can't use up the entire cpu because there are times in between where there is nothing to process from memory. I haven't implemented faster code but in future I will.
Reply With Quote
  #173  
Old 19-07-2020, 16:56
panker1992's Avatar
panker1992 panker1992 is offline
Registered User
 
Join Date: Oct 2015
Location: Always Somewhere
Posts: 515
Thanks: 112
Thanked 739 Times in 289 Posts
panker1992 is on a distinguished road
I talked to you about preflate quite some time ago

glad to see you have made something out of it, will be returning to testing as soon as i am back home!!


have produced several videos on youtube on advanced compression past weeks.
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows
My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45.
Reply With Quote
  #174  
Old 20-07-2020, 22:06
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,942 Times in 2,140 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by panker1992 View Post
I talked to you about preflate quite some time ago
talking to me about it isn't the same as implementing it

@everyone

here's the xtool with reflate support.

I'll once again explain what's the difference. There are 3 deflate precompressors in xtool: zlib, reflate and preflate.

The old xtool/ztool/pzlib all had zlib+reflate but they had so many errors so I added preflate as an alternative if you have issues when you use reflate.

There are other things to this of course. I added all 3 because sometimes one has an advantage such as speed or better size output. Which is up to you to pick.

reflate cannot be used at the same time as preflate or the other way round.

but zlib can be used with reflate or preflate (pick one) like this -mzlib+reflate or -mzlib+preflate

here are my results on games that use the "normal" deflate configuration

using -mzlib:
Code:
Compressed 1 file, 1,138,892,800 => 2,611,771,763 bytes. Ratio 229.33%
Compression time: cpu 1.34 sec/real 55.54 sec = 2%. Speed 20.51 mB/s
using -mreflate:
Code:
Compressed 1 file, 1,138,892,800 => 2,613,343,052 bytes. Ratio 229.46%
Compression time: cpu 1.28 sec/real 86.86 sec = 1%. Speed 13.11 mB/s
using -mpreflate
Code:
Compressed 1 file, 1,138,892,800 => 2,613,650,264 bytes. Ratio 229.49%
Compression time: cpu 1.34 sec/real 90.68 sec = 1%. Speed 12.56 mB/s
Attached Files
File Type: 7z xtool_reflate_test.7z (1.89 MB, 62 views)
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
COPyCAT (23-08-2020), dixen (21-07-2020), shazzla (21-07-2020)
  #175  
Old 20-07-2020, 22:17
BLACKFIRE69's Avatar
BLACKFIRE69 BLACKFIRE69 is offline
Registered User
 
Join Date: Mar 2019
Location: In the Hell
Posts: 308
Thanks: 253
Thanked 870 Times in 210 Posts
BLACKFIRE69 is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
talking to me about it isn't the same as implementing it

@everyone

here's the xtool with reflate support.

I'll once again explain what's the difference. There are 3 deflate precompressors in xtool: zlib, reflate and preflate.

The old xtool/ztool/pzlib all had zlib+reflate but they had so many errors so I added preflate as an alternative if you have issues when you use reflate.

There are other things to this of course. I added all 3 because sometimes one has an advantage such as speed or better size output. Which is up to you to pick.

reflate cannot be used at the same time as preflate or the other way round.

but zlib can be used with reflate or preflate (pick one) like this -mzlib+reflate or -mzlib+preflate

here are my results on games that use the "normal" deflate configuration

using -mzlib:
Code:
Compressed 1 file, 1,138,892,800 => 2,611,771,763 bytes. Ratio 229.33%
Compression time: cpu 1.34 sec/real 55.54 sec = 2%. Speed 20.51 mB/s
using -mreflate:
Code:
Compressed 1 file, 1,138,892,800 => 2,613,343,052 bytes. Ratio 229.46%
Compression time: cpu 1.28 sec/real 86.86 sec = 1%. Speed 13.11 mB/s
using -mpreflate
Code:
Compressed 1 file, 1,138,892,800 => 2,613,650,264 bytes. Ratio 229.49%
Compression time: cpu 1.34 sec/real 90.68 sec = 1%. Speed 12.56 mB/s
Razor,
it would be nice if you could update the "Game File Scanner" with the new xtool.
Reply With Quote
The Following User Says Thank You to BLACKFIRE69 For This Useful Post:
Harsh ojha (23-07-2020)
  #176  
Old 21-07-2020, 06:15
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 255
Thanks: 304
Thanked 292 Times in 135 Posts
dixen is on a distinguished road
Dishonored. Death of the outsider (yes, again)

game1.resources
game1_001.resources
game1_patch.resources

-mpreflate

Quote:
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 3 files, 9,663,676,416 => 13,842,292,926 bytes. Ratio 143.24%
Compression time: cpu 12.19 sec/real 498.98 sec = 2%. Speed 19.37 mB/s
All OK
Unpack

Quote:
Extracted 3 files, 13,842,292,926 => 9,663,676,416 bytes. Ratio 143.24%
Extraction time: cpu 21.22 sec/real 217.84 sec = 10%. Speed 44.36 mB/s
All OK
Reply With Quote
The Following User Says Thank You to dixen For This Useful Post:
Razor12911 (23-07-2020)
  #177  
Old 23-07-2020, 10:22
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,942 Times in 2,140 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by BLACKFIRE69 View Post
Razor,
it would be nice if you could update the "Game File Scanner" with the new xtool.
I actually have a better idea

Quote:
Originally Posted by dixen View Post
Dishonored. Death of the outsider (yes, again)

game1.resources
game1_001.resources
game1_patch.resources

-mpreflate



Unpack
try with reflate
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
shazzla (23-07-2020)
  #178  
Old 23-07-2020, 10:28
BLACKFIRE69's Avatar
BLACKFIRE69 BLACKFIRE69 is offline
Registered User
 
Join Date: Mar 2019
Location: In the Hell
Posts: 308
Thanks: 253
Thanked 870 Times in 210 Posts
BLACKFIRE69 is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
I actually have a better idea
sounds great
Reply With Quote
  #179  
Old 23-07-2020, 12:58
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 255
Thanks: 304
Thanked 292 Times in 135 Posts
dixen is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
try with reflate
-mreflate

Quote:
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 3 files, 9,663,676,416 => 14,323,974,019 bytes. Ratio 148.22%
Compression time: cpu 11.94 sec/real 273.23 sec = 4%. Speed 35.37 mB/s
All OK
Quote:
Extracting 3 files, 9,663,676,416 bytes. Processed 92.1%
ERROR: CRC failed in "unpacked\game1.resources". File is broken.
Test on DOOM - later..

Last edited by dixen; 23-07-2020 at 13:42.
Reply With Quote
The Following User Says Thank You to dixen For This Useful Post:
Razor12911 (23-07-2020)
  #180  
Old 23-07-2020, 14:43
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 255
Thanks: 304
Thanked 292 Times in 135 Posts
dixen is on a distinguished road
DOOM 2016

snap_gameresources.resources
gameresources.resources

-mxtool 2019 (R3)

Quote:
Extracted 2 files, 5,513,073,128 => 11,189,482,143 bytes. Ratio 49.27%
Extraction time: cpu 20.17 sec/real 1522.22 sec = 1%. Speed 7.35 mB/s
All OK
-mreflate

Decompress

Quote:
Compressed 2 files, 11,189,482,143 => 21,441,692,530 bytes. Ratio 191.62%
Compression time: cpu 13.69 sec/real 811.30 sec = 2%. Speed 13.79 mB/s
All OK
Unpack

Quote:
Extracting 2 files, 11,189,482,143 bytes. Processed 78.6%
ERROR: CRC failed in "unpacked\gameresources.resources". File is broken.
arc.exe: wclose: invalid argument (Bad file descriptor)
79.0%
-mpreflate

Quote:
Compressed 2 files, 11,189,482,143 => 11,189,490,830 bytes. Ratio 100.00%
Compression time: cpu 13.23 sec/real 1371.54 sec = 1%. Speed 8.16 mB/s
All OK

Last edited by dixen; 23-07-2020 at 15:06.
Reply With Quote
The Following 2 Users Say Thank You to dixen For This Useful Post:
Razor12911 (23-07-2020), sanekbest1 (24-07-2020)
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump



All times are GMT -7. The time now is 02:52.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com