Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #271  
Old 27-12-2020, 10:31
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Exactly! But.. I doubt I am doing anything wrong, I mean:
Code:
-mcustom6/$gamelz=xtool:mcp2077+srp64+xlzma:lc8
*.archive = $gamelz group.
So *.archive = FA -mxtool:mcp2077+srp64+xlzma:lc8
or xtool precomp -t100p -mcp2077 to be precise
Code:
[External compressor:xtool]
header = 0
packcmd = _EC\xtool\xtool precomp -t100p {options} - - <stdin> <stdout> 
 unpackcmd = _EC\xtool\xtool decode -t100p - - <stdin> <stdout>
Code:
srp64          = srep64:m3f:c256:mem8g:m64k:b16m:a2
procdefault    = srp64+xlzma:lc8
Code:
-mcustom6/$gamelz=xtool:mcp2077+procdefault
Anyone?(PS its latest 2012r2)
EDIT(restarted without t100p, but I don't think that's it: xtool.png
How you guys getting ~10mb/s is beyond me, are you tarring or doing individual files? As it may matter..)
EDIT2(speed down to 0.6mb/s, estimation ~22h+ and slowing down. Really, this ain't right but tool appear to be working correct not hanging! Can anyone try: latest xtool version with cp2077 plugin and on tarred archives through FA not individual files? I am sure there is something wrong in one of those variables.)
PS(I forgot to note that game is gog with 1.06 update, that means 9gb 1.05 was also applied, maybe it changed oodle structure? I will postpone compression until solution/cause is clear.)

Last edited by elit; 27-12-2020 at 11:19.
Reply With Quote
Sponsored Links
  #272  
Old 27-12-2020, 11:52
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
So I just quickly tried oo2rect. That one seem to be working ok! I am going to use it on whole game and will see. I also tried xtool with -mkraken instead cp2077 plugin and also on individual archives. Speed issue remains. It does seem to have problem no matter what, while oo2rect seems fine.

EDIT:
oo2rect is same slow, after ~1h about 20gb and 0.xxmb/s speed. At this point I am pretty sure problem is not xtool nor oo2rect or plugin, but a game. If this was not a case with original version 1.03, try to update to 1.06, I think they changed data enough to become problem. For reference, files of concern are:
basegame_3_nightcity.archive
basegame_3_nightcity_gi.archive

Last edited by elit; 27-12-2020 at 14:17.
Reply With Quote
  #273  
Old 28-12-2020, 03:09
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Alright so I left it overnight. I can confirm it to work correctly but it took 11h:40min to compress, from that about ~10h due to xtool oodle. Archive tested ok and decompression was much quicker, compression is that slow though.
Reply With Quote
  #274  
Old 28-12-2020, 03:34
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,557
Thanks: 935
Thanked 5,802 Times in 2,143 Posts
KaktoR is on a distinguished road
No Idea what's wrong, but for me it's working just fine.

I use GOG version 1.06, along with an Ryzen 5 2600 and 16GB RAM and latest xtool version posted a few days ago.

I can test the whole game folder if you wish, but I can tell you the results will be just fine and nothing wrong.

Xtool settings I use:
Code:
[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp -mcp2077 -c128mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>
Code:
basegame_3_nightcity.archive

Compressed 1 file, 7,958,224,896 => 22,714,137,974 bytes. Ratio 285.42%
Compression time: cpu 9.34 sec/real 704.44 sec = 1%. Speed 11.30 mB/s
All OK

Extracted 1 file, 22,714,137,974 => 7,958,224,896 bytes. Ratio 285.42%
Extraction time: cpu 8.11 sec/real 333.60 sec = 2%. Speed 23.86 mB/s
All OK
__________________
Haters gonna hate

Last edited by KaktoR; 28-12-2020 at 03:38.
Reply With Quote
The Following 3 Users Say Thank You to KaktoR For This Useful Post:
Cesar82 (07-01-2021), elit (28-12-2020), ffmla (28-12-2020)
  #275  
Old 28-12-2020, 10:29
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Hm, you have 6 core with hyperthreading. Maybe -t100p for you utilize all 12 virtual cores. You may try -t4 and also no --dbase to see if it affect speed that much. Also no need to test all archives only 2 files I mentioned above is enough. I wonder if --dbase or 12 virtual cores make for such difference. BTW thanks for the info above, it's interesting to know.
Reply With Quote
  #276  
Old 28-12-2020, 23:30
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,630
Thanks: 2,003
Thanked 10,318 Times in 2,198 Posts
Razor12911 is on a distinguished road
i5 4590, 4 Cores (4 Threads)

Code:
[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp -mcp2077 -c128mb -t100p - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>
Code:
basegame_3_nightcity.archive

Compressed 1 file, 7,958,224,896 => 22,919,265,672 bytes. Ratio 287.99%
Compression time: cpu 9.91 sec/real 1538.09 sec = 1%. Speed 5.17 mB/s
All OK

Tested 1 file, 22,919,265,672 => 7,958,224,896 bytes. Ratio 287.99%
Testing time: cpu 6.45 sec/real 528.38 sec = 1%. Speed 15.06 mB/s
All OK
My concern is the difference in ratio, --dbase should not have an impact on ratio just the speed. Some bug I guess...

Last edited by Razor12911; 29-12-2020 at 00:01.
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
elit (29-12-2020)
  #277  
Old 07-01-2021, 05:47
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 731
Thanks: 160
Thanked 764 Times in 382 Posts
Masquerade is on a distinguished road
Here's an interesting one.

ZZ1.dat - 175MB from Steel Division 2.


XTool 2020 (zlib)
Code:
Compressed 1 file, 183,623,680 => 183,661,889 bytes. Ratio 100.02%
Compression time: cpu 0.16 sec/real 26.08 sec = 1%. Speed 7.04 mB/s
All OK
Xtool v12 (zlib)
Code:
Compressed 1 file, 183,623,680 => 226,689,420 bytes. Ratio 123.45%
Compression time: cpu 0.27 sec/real 26.18 sec = 1%. Speed 7.01 mB/s
All OK
XTool v12 hangs inifintely when during decompression - the file just keeps getting bigger and bigger.

GFS Detects zlib streams, whereas Drop Scan does not:





https://anonfiles.com/FcY1Q060p9/ZZ_1_dat
Reply With Quote
  #278  
Old 07-01-2021, 05:50
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,557
Thanks: 935
Thanked 5,802 Times in 2,143 Posts
KaktoR is on a distinguished road
Make headerless+force detection

PS: The anonfiles is shit. Always slow connection here (185kb/s)
__________________
Haters gonna hate
Reply With Quote
  #279  
Old 07-01-2021, 06:02
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,557
Thanks: 935
Thanked 5,802 Times in 2,143 Posts
KaktoR is on a distinguished road
@Masquerade
Use reflate codec

Code:
Compressed 1 file, 183,623,680 => 227,188,297 bytes. Ratio 123.72%
Compression time: cpu 0.09 sec/real 5.12 sec = 2%. Speed 35.87 mB/s
All OK

Extracted 1 file, 227,188,297 => 183,623,680 bytes. Ratio 123.72%
Extraction time: cpu 0.13 sec/real 1.07 sec = 12%. Speed 171.41 mB/s
All OK
__________________
Haters gonna hate
Reply With Quote
The Following 3 Users Say Thank You to KaktoR For This Useful Post:
dixen (07-01-2021), Masquerade (07-01-2021), shazzla (07-01-2021)
  #280  
Old 07-01-2021, 18:59
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,630
Thanks: 2,003
Thanked 10,318 Times in 2,198 Posts
Razor12911 is on a distinguished road
When you use zlib method in Xtool v12 or older variants of xtool, they automatically used reflate when you placed the libraries each time zlib fails, the new xtool does not do any of this. The user is the one who needs to do this by themselves because what I have noticed is people putting reflate libraries near xtool and they never know when xtool uses them or when it doesn't. This way the user knows what the issues is when decompression fails each time when they try to figure out what the actual problem is.

I also did point out that the best method for precompressing anything that is zlib/deflate compressed, use -mzlib along with either reflate/preflate so you get the best results.
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
Ele (07-01-2021), Masquerade (08-01-2021)
  #281  
Old 08-01-2021, 05:14
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 731
Thanks: 160
Thanked 764 Times in 382 Posts
Masquerade is on a distinguished road
Razor12911,
I did use zlib+preflate, I simply forgot about reflate.

After using relate, everything works fine.
Reply With Quote
  #282  
Old 08-01-2021, 20:00
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,630
Thanks: 2,003
Thanked 10,318 Times in 2,198 Posts
Razor12911 is on a distinguished road
Update available

Changes

- updated library support
- updated command line parser
- included x86 build
- fixed depthing issues

Notes

I have added 32-bit build as requested but you'll have to use x86 libraries or find a matching pair of x86/x64 libraries. You can get these from github of a project under the releases section.
The versioning is brought back to the format of 1.0.0 instead of the build number as requested.
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
ffmla (09-01-2021), kenzo34 (09-01-2021), ZAZA4EVER (09-01-2021)
  #283  
Old 09-01-2021, 02:31
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 731
Thanks: 160
Thanked 764 Times in 382 Posts
Masquerade is on a distinguished road
@Razor12911

Thanks for this new method... incredible!

Code:
Compressing 1 file, 17,530,888,054 bytes
  Compressing TSCGame-WindowsNoEditor.pak
Compressed 1 file, 17,530,888,054 => 28,460,607,223 bytes. Ratio 162.35%
Compression time: cpu 18.42 sec/real 624.55 sec = 3%. Speed 28.07 mB/s
All OK
On the sample I sent to you yesterday.
Reply With Quote
  #284  
Old 09-01-2021, 08:03
Cesar82's Avatar
Cesar82 Cesar82 is offline
Registered User
 
Join Date: May 2011
Location: Brazil
Posts: 781
Thanks: 1,291
Thanked 1,647 Times in 562 Posts
Cesar82 is on a distinguished road
@Razor12911, thanks for the 32-bit version, it will be very useful to include in CIU.
1) Also like to inform you that it is being detected as a false positive by the KIS anti-virus (I don't know if there is anything that can be done).
P.S: The 64-bit version has not been captured.

2) You could share libraries compatible with the 32-bit version to be fully compatible and avoid errors due to incompatibilities.
3) I wonder if I have 3 games (collection) and each game uses XTool with different methods to compress.
Assuming that only the libraries needed for compression (nothing more than necessary) for each Data.bin file are together with XTool.exe at the time of compressing.
To perform the decompression, can I have other libraries that were not used to compress the Data.bin file together with XTool (Like: Have 1 library next to XTool when compressing and 5 when decompressing)?
Thanks!
Reply With Quote
  #285  
Old 09-01-2021, 08:34
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,630
Thanks: 2,003
Thanked 10,318 Times in 2,198 Posts
Razor12911 is on a distinguished road
1) some interesting stuff,
x86: https://www.virustotal.com/gui/file/...296d/detection
x64: https://www.virustotal.com/gui/file/...7dfe/detection

the code is actually the same so I don't know...

2) Yes the issue is also the plugins which also have to shipped with 32-bit version, who still uses 32-bit system anyways?
Capture2.PNG

3) Yes, shouldn't be a problem.

I'll ship x86 libraries but any incompatibility issues should fall on the end user.
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (09-01-2021), ffmla (09-01-2021)
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 15:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 18:16



All times are GMT -7. The time now is 07:37.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com