Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #226  
Old 02-11-2020, 20:06
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 286
Thanks: 350
Thanked 327 Times in 148 Posts
dixen is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
could you try without deduplication, too many points where an error could occur.
Of course...waiting
UPD.
Yes without dedup - the test was completed successfully and xt used about 500-600 mb RAM (XT20 R5)
Quote:
Tested 11 files, 20,205,910,501 => 21,896,534,144 bytes. Ratio 92.28%
Testing time: cpu 68.59 sec/real 365.42 sec = 19%. Speed 59.92 mB/s
All OK

Last edited by dixen; 02-11-2020 at 20:50.
Reply With Quote
The Following User Says Thank You to dixen For This Useful Post:
Razor12911 (02-11-2020)
Sponsored Links
  #227  
Old 03-11-2020, 11:09
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
You may want to be interested in this.

I was trying to inflate cpk file from Diablo 3 from nintendo switch. Normaly these are crilayla but gfs detected lz. So I tried old ztool, some older xtool(date modified says 26.april 2019) and xtool2009R3.

For reference I tried single file "Act1.cpk" of 814mb size. GFS detected potential of inflation of up to ~1800mb.

Ztool could not inflate past few 10's of mb if any, regardless of setting like m2, m3 etc.

Later xtool2009R3 depended on codec, most were at about 100mb extra only, except reflate which did go to 1.1gb. However final packed size after srep+xlzma was about same which make me think of possible huge overheat?

Finally, that older xtool with :high setting was able to inflate to.. 1.7gb!!
I tried to compress that and got about ~40mb better compression.

General final compression after codecs was around 740mb+, this one went to ~704mb. I wonder if this was real inflated data to 1.7gb or too much overheat. I must also say that older xtool was able to do it only with :high option which unlike ztool is not even know or documented and probably unofficial. Also it took very long time compare to say ztool:high, it seems to work differently.

I have uploaded the file act1.cpk for further research if you are interested:
https://megaup.net/1zEg1/Act1.cpk

Last edited by elit; 04-11-2020 at 03:55.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
oltjon (08-11-2020)
  #228  
Old 09-11-2020, 04:41
amin fear amin fear is offline
Registered User
 
Join Date: Feb 2012
Location: Inferno
Posts: 98
Thanks: 78
Thanked 54 Times in 29 Posts
amin fear is on a distinguished road
Is there any incompatibility between "CIU 3.x+ZTOOL+XTOOL" extraction process & new AMD CPU series ?

Anybody have tested ZTOOL with below CPU ?

AMD Ryzen™ 9 3900X "12-Core 3.8 GHz" (4.6 GHz Max Boost) Socket AM4 105W

My installer created with "CIU 3.x+SrepInsidex64+ZToolx64+LOLZx64" works pretty well on ALL of mid range CPUs(Corei3+Corei5+Corei7) but when extracting on NEW AMD Ryzen™ 9 3900X, installer stuck on "0.0" percent progress bar & "ZTool.exe" won't launch on task manager!!!

Is "ZTool.exe" incompatible with new 12-Core CPUs ?

Thanks @Razor12911
__________________
Paint me white so i can invisibly fight in the light...paint me black so i can hide my tears in the shadow of your heart.
Reply With Quote
  #229  
Old 09-11-2020, 07:43
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,557
Thanks: 935
Thanked 5,802 Times in 2,143 Posts
KaktoR is on a distinguished road
Did you try to unpack with bat file`? If this works then the problem could be inside CIU code somewhere.
__________________
Haters gonna hate
Reply With Quote
  #230  
Old 09-11-2020, 16:15
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,630
Thanks: 2,003
Thanked 10,318 Times in 2,198 Posts
Razor12911 is on a distinguished road
my tools can handle up to 256 threads, if you have problems it's mainly because you're using ztool instead of the newer xtool which had MT bugs fixed.
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
amin fear (10-11-2020)
  #231  
Old 10-11-2020, 05:13
amin fear amin fear is offline
Registered User
 
Join Date: Feb 2012
Location: Inferno
Posts: 98
Thanks: 78
Thanked 54 Times in 29 Posts
amin fear is on a distinguished road
Quote:
Originally Posted by KaktoR View Post
Did you try to unpack with bat file`? If this works then the problem could be inside CIU code somewhere.
Thanks @KaktoR for batch script! i tried that unpack script but it gives me some errors ! can you please take a look at them :

I have a lot of HDD space [200GB] ,but still gives me errors !

Code:
https://imgur.com/a/v2DOBcn
I have compressed my files in below names & put them in "OUTPUT" folder of DiskSpan_UNPACK script:

Code:
Data1.Bin.001
Data1.Bin.002
Data1.Bin.003
using below compression method :

Code:
pzlib:m2+srep:m3f+lolz:dtb1:mtt1:mt4:d128m:mc1023
I only modified archive name in "_DiskSpan_UNPACK (Test Only).bat" , should i change other parameters ? for example compression switch !
__________________
Paint me white so i can invisibly fight in the light...paint me black so i can hide my tears in the shadow of your heart.
Reply With Quote
  #232  
Old 10-11-2020, 08:12
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,557
Thanks: 935
Thanked 5,802 Times in 2,143 Posts
KaktoR is on a distinguished road
Try this

Code:
Regedit
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System
EnableLUA > 0
ConsentPromptBehaviorAdmin > 0
FilterAdministratorToken > 0

Restart Computer
__________________
Haters gonna hate
Reply With Quote
The Following 2 Users Say Thank You to KaktoR For This Useful Post:
amin fear (10-11-2020), Hota (29-11-2020)
  #233  
Old 10-11-2020, 18:45
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,630
Thanks: 2,003
Thanked 10,318 Times in 2,198 Posts
Razor12911 is on a distinguished road
Update available

Changes

- added temporary libdunia codec (thanks ProFrager and FitGirl)
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
78372 (12-11-2020), Hota (29-11-2020), ZAZA4EVER (12-11-2020)
  #234  
Old 11-11-2020, 00:24
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 731
Thanks: 160
Thanked 764 Times in 382 Posts
Masquerade is on a distinguished road
Thanks Razor / ProFrager / FitGirl for the new dunia codec!

Here's a test on primal_main.dat (6.32GB)

Code:
Compressed 1 file, 6,796,379,826 => 8,579,219,400 bytes. Ratio 126.23%
Compression time: cpu 6.58 sec/real 299.11 sec = 2%. Speed 22.72 mB/s
All OK
Working fast and good.
Reply With Quote
  #235  
Old 11-11-2020, 05:38
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 286
Thanks: 350
Thanked 327 Times in 148 Posts
dixen is on a distinguished road
WD Legion

london.dat
shadersobj.dat
common.dat
patch.dat
london_preload.dat
london_cache.dat

13 gb > 17.6 gb

Log not saved)) Sorry
Reply With Quote
  #236  
Old 12-11-2020, 07:56
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 731
Thanks: 160
Thanked 764 Times in 382 Posts
Masquerade is on a distinguished road
Having a few problems decompressing with the fcp option, I have tried a few different unpackcmds:

Code:
[External compressor: fcp]
header = 0
unpackcmd = xtool decode --dbase=fcp.xtl - - <stdin> <stdout>
Code:
[External compressor: fcp]
header = 0
unpackcmd = xtool decode - - <stdin> <stdout>
Code:
[External compressor: fcp]
header = 0
unpackcmd = xtool decode -mfcp - - <stdin> <stdout>
Yes, fcp.xtl and liblz4.dll are with xtool.exe

Errors like:

Code:
G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed   0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed   0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed   0%
ERROR: write error (disk full?) in compression algorithm fcp
This is just on 1 file with no extra compressors.

Is there something wrong with my unpackcmd?
Reply With Quote
  #237  
Old 12-11-2020, 10:24
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 731
Thanks: 160
Thanked 764 Times in 382 Posts
Masquerade is on a distinguished road
Update: srep+lolz is smaller than fcp+srep+lolz
Reply With Quote
  #238  
Old 21-11-2020, 07:51
shazzla shazzla is offline
Registered User
 
Join Date: Nov 2010
Location: Hunnia
Posts: 222
Thanks: 408
Thanked 86 Times in 63 Posts
shazzla is on a distinguished road
Guys ,how can i set the

--dedup=FC.bin

parameter in -m chain ?
is it possible ?
Reply With Quote
  #239  
Old 04-12-2020, 09:30
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
So I was dealing with crilayla in Persona 5 cpk's. Normally for these I just use bms but for the first time I got errors in some archives. So i tried your older xtool that still had it(in fact that's why I kept it). And man.. this thing is fantastic! I bring it because you said you dropped it because of speed. Well I don't know if I can change your mind but let me try.

I tested 4.4gb cpk with :t4 on my 4960k 4.2ghz, which is pretty normal today. Inflating took about 16min and re-encoding about 14min. This gives the speed of around 4.5-5.5mb\s. Look, this is not bad at all! There are lot of crazy people out there abusing lzma/lolz with outright stupid settings like mc1000+ that can slow you to crawl with no benefits, yet unlike those this tool is extremely beneficial.

Maybe you meant that for decompression its too slow. Well, repacking cpk back with single threaded bms is just as slow if not more + all the work around it, I did it this way till now. I even had to use xdelta because bms compressor write something into header that I have not seen in single game yet(other than zeroes, I think it was offset 16 and ~6 bytes long string).

So your cri xtool is excellent, work well and have reasonable speed including de/compression - considering alternatives. Also future CPU cores are only going to be more, not less.

This is my 2 cents, of course you do what you feel like. But there is no other cri tool like this and compression is probably used second most often after deflate. There already are plenty of deflate tools and libraries, even original ztool is just perfectly fine for this. Its those other codecs that would be beneficial. Crilayla, maybe also lzss and certainly yaz0 to name few.

In fact you already completed the work, why abandon it? So for the love of god I beg you, put it back in latest version!

PS(Zstd and lz4 are temporary and will stop working after time because of their retarded design. Not sure about lzo(that is actually a question, why is standard lzo not working in dunia and unreal engines and need special treatment? I thought its a stable design like zlib?). But crilayla, lzss, yaz0 and such should remain compatible.

Last edited by elit; 04-12-2020 at 15:47.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
Gehrman (27-08-2021)
  #240  
Old 04-12-2020, 12:35
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
-grittibanzli inflate completely randomly. Sometimes tested 20mb inflated zip file is 34mb, other times 31.6mb, another time 33mb atd. exactly same setting and just repeating same command on console.

-zlib codec did not inflated zip file at all!(I thought it was supposed to handle deflate?)

-preflate and reflate both did to same ~27mb, but preflate took ~6sec while reflate did it in ~1sec.

Looks like good old reflate is still the best zlib codec to stick.


EDIT(and ztool is also able to inflate to 27mb without :high or :m2 !) <<UPDATE(BS, scrap that claim, need m2 or high indeed)
EDIT2(ok I checked manual again and learned about zlib+reflate combination benefits. Maybe this is best way. It took ~3sec compared to ~1sec on deflate alone though. Not sure how big those benefits are in real application yet.)
EDIT3(have to explicitly state 'mb' in -c option instead of just 'm', e.g. -c128m vs -c128mb. No big deal but used to ztool, was more flexible. PS: I am really liking the idea of zlib+reflate in one go, before I used to pass twice in test to find out which one to use.)

Last edited by elit; 04-12-2020 at 13:54.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 15:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 18:16



All times are GMT -7. The time now is 07:11.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com