Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #136  
Old 15-08-2020, 15:26
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,636
Thanks: 2,013
Thanked 10,343 Times in 2,202 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by FitGirl View Post
I suspect it depends on number of duplicated streams
you answered yourself

but as I said, I have a few ideas to reduce the memory requirements like getting rid of streams from memory that have restored the duplicated streams. I am not sure if srep does this already but if it doesn't then it should be a bonus. GPU memory usage is another thing that will be added but in future. I need start adding support for other codecs, especially the external ones. What I plan is for people to write their own plugins regardless whether they can write code or not. If you can figure out a header structure of a game, just write and ini file and give it to xtool and it will do the rest. This is mostly for lz4 games, encrypted ones or ones that require something special. I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)

Here's an example of Saints Row IV Remastered, instead of making another side project, you just write this and give it to xtool.

Code:
[stream1]
Codec=lz4hc:l10 // lz4hc compression used, level 10
Header=00BADBEE0FEEDBEE // this is the magic bytes
Structure={Header(8)} {CSize(4)} {DSize(4)} {Stream} // this is the header structure
Condition1=DSize > CSize // the conditions for xtool to accept the stream
Condition2=CSize > 64
Condition3=CSize < 16 x 1024 x 1024
Quote:
Maybe it's possible for xtool to report it after initial decompression is done?
this I can do


@everyone

here are some benchmarks, I finally written the decompression code, though it needs some tweaking before I can post update

normal xtool
Code:
Compressed 1 file, 320,782,304 => 108,018,914 bytes. Ratio 33.67%
Compression time: cpu 0.47 sec/real 22.64 sec = 2%. Speed 14.17 mB/s

Tested 1 file, 108,018,914 => 320,782,304 bytes. Ratio 33.67%
Testing time: cpu 0.38 sec/real 14.91 sec = 3%. Speed 21.51 mB/s
+ memory database
+ deduplication
Code:
Compressed 1 file, 320,782,304 => 107,858,828 bytes. Ratio 33.62%
Compression time: cpu 0.44 sec/real 15.98 sec = 3%. Speed 20.08 mB/s

Tested 1 file, 107,858,828 => 320,782,304 bytes. Ratio 33.62%
Testing time: cpu 0.44 sec/real 9.32 sec = 5%. Speed 34.41 mB/s
thus far, the output is slightly better for more speed

Last edited by Razor12911; 15-08-2020 at 16:06.
Reply With Quote
The Following 11 Users Say Thank You to Razor12911 For This Useful Post:
BLACKFIRE69 (15-08-2020), Cesar82 (15-08-2020), dixen (16-08-2020), ffmla (17-08-2020), FitGirl (15-08-2020), Gehrman (16-08-2020), Harsh ojha (15-08-2020), L0v3craft (15-08-2020), Masquerade (16-08-2020), shazzla (15-08-2020), ZAZA4EVER (16-08-2020)
Sponsored Links
  #137  
Old 15-08-2020, 18:06
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 191
Thanks: 16
Thanked 231 Times in 124 Posts
FitGirl is on a distinguished road
Configuration feature is great! It would be so much easier to repack some obscure formats.
Reply With Quote
  #138  
Old 16-08-2020, 02:04
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 296
Thanks: 356
Thanked 332 Times in 152 Posts
dixen is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)
For LEGO series games this would be fine)
Reply With Quote
  #139  
Old 16-08-2020, 13:03
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 191
Thanks: 16
Thanked 231 Times in 124 Posts
FitGirl is on a distinguished road
Quote:
Originally Posted by dixen View Post
For LEGO series games this would be fine)
It will never work with LEGO series, cause there are no public compressors for algos used in those games, and the decompressors used by Luigi are in binary (ripped from DLL) format, so they can only be extracted, not packed back.
Reply With Quote
  #140  
Old 16-08-2020, 16:05
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,636
Thanks: 2,013
Thanked 10,343 Times in 2,202 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by FitGirl View Post
Any predictions on RAM requirements for both imported/memory DBs?
back on the SEACREST input from Need For Speed Hot Pursuit

I ran a memory usage test as you asked and these were the results.

xtool-virtual-memory.tmp = 328 MB
Decompression memory is 22 mb. 80,927 matches = 1,294,832 bytes = 0.10% of file

= total 350 MB (with deduplication)

Decompression memory is 1485 mb. 147,537 matches = 2,360,592 bytes = 0.18% of file

= total 1485 MB (without deduplication)

NOTE: Xtool uses memory of its own and this was not factored, about 190 MB
so if you do the math, that's 350+190 = about 540 MB total memory while if xtool doesn't remove deduplicates and lets srep all the work that's 1485+190 = 1675 MB

540 MB vs 1675 MB memory usage, which is better?

when 540 MB ram was used, xtool was decoding at 88.91 MB/s
and when 1675 MB ram was used, xtool decoded at 27.45 MB/s

so yes more speed and less memory usage, potato pcs should benefit greatly from this (like mine )
Reply With Quote
The Following 5 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (16-08-2020), Gehrman (27-08-2021), Harsh ojha (16-08-2020), Mortal Lord (16-08-2020), shazzla (17-08-2020)
  #141  
Old 17-08-2020, 01:43
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 747
Thanks: 168
Thanked 791 Times in 394 Posts
Masquerade is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
xtool-virtual-memory.tmp = 328 MB
Decompression memory is 22 mb. 80,927 matches = 1,294,832 bytes = 0.10% of file

= total 350 MB (with deduplication)
I know you don't have much data to go on, but how does xtool+dedup compare against xtool+srep?
Reply With Quote
The Following User Says Thank You to Masquerade For This Useful Post:
shazzla (17-08-2020)
  #142  
Old 17-08-2020, 04:51
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 191
Thanks: 16
Thanked 231 Times in 124 Posts
FitGirl is on a distinguished road
Quote:
Originally Posted by Masquerade View Post
I know you don't have much data to go on, but how does xtool+dedup compare against xtool+srep?
The idea of new xtool is not replace srep. It's purpose is not to recompress duplicate chunks, but just copy them, which makes less CPU load and faster overall decompression. You still need to run srep after xtool, cause it will deal with unpacked duplicated chunks.
Reply With Quote
The Following User Says Thank You to FitGirl For This Useful Post:
shazzla (17-08-2020)
  #143  
Old 20-08-2020, 02:12
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 747
Thanks: 168
Thanked 791 Times in 394 Posts
Masquerade is on a distinguished road
Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB



So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.
Reply With Quote
The Following User Says Thank You to Masquerade For This Useful Post:
Razor12911 (21-08-2020)
  #144  
Old 20-08-2020, 04:20
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 191
Thanks: 16
Thanked 231 Times in 124 Posts
FitGirl is on a distinguished road
Quote:
Originally Posted by Masquerade View Post
Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB



So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.
Afair Scribblenauts is encrypted. You are lucky to always choose encrypted games for your tests
Reply With Quote
  #145  
Old 20-08-2020, 05:14
Harsh ojha's Avatar
Harsh ojha Harsh ojha is offline
Registered User
 
Join Date: May 2019
Location: INDIA
Posts: 78
Thanks: 504
Thanked 72 Times in 36 Posts
Harsh ojha is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
so yes more speed and less memory usage, potato pcs should benefit greatly from this (like mine )
Same here potato pc 😅

FitGirl Repack take more HHD space for installing game ex. Far Cry 3 duology
Need 17 GB HHD space free
But the game Size was 10 GB ( language english.
Without FC3 mapEditor )

*Sorry for my bad English as English is not my mother tongue 😅*
__________________
Video Creator
Discord - Harsh_Ojha_748 #8782
Reply With Quote
  #146  
Old 20-08-2020, 13:36
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 747
Thanks: 168
Thanked 791 Times in 394 Posts
Masquerade is on a distinguished road
Quote:
Originally Posted by FitGirl View Post
You are lucky to always choose encrypted games for your tests
Sucks to be me I guess. I don't knowingly do this.
Reply With Quote
  #147  
Old 21-08-2020, 15:01
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,636
Thanks: 2,013
Thanked 10,343 Times in 2,202 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Masquerade View Post
Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB



So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.
hmm, final size after compression?
Reply With Quote
  #148  
Old 21-08-2020, 15:12
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 747
Thanks: 168
Thanked 791 Times in 394 Posts
Masquerade is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
hmm, final size after compression?
With xtool r12: 1.38GB

I feel slightly ashamed for not thinking the data could be encrypted.

Didn't test to make a full archive with 2020, I will do this if you wish.

Every time I put xtool 2020 up against xtool r12, r12 always produces a larger size. Both run at 128m chunk size.
Reply With Quote
  #149  
Old 21-08-2020, 16:08
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,636
Thanks: 2,013
Thanked 10,343 Times in 2,202 Posts
Razor12911 is on a distinguished road
and the final size of 2020?
Reply With Quote
  #150  
Old 21-08-2020, 22:38
shazzla shazzla is offline
Registered User
 
Join Date: Nov 2010
Location: Hunnia
Posts: 223
Thanks: 410
Thanked 86 Times in 63 Posts
shazzla is on a distinguished road
@Razor12911 :

Few days ago i started playing with XTool 2008. Great job ! Thanks !
But i found this :

Frostpunk - On the edge (Is its streams ZLIB? It must be IMHO)

It cant precompress those .dat files. (For example ,common.dat and languages.dat were tested).
Common.dat nearly a 600 MB file,languages.dat is a much smaller.
In both case ,the output is the same.
Older generation of XTool (AFAIR V0.9) precompresses commond.dat well to ~1.5 GB, languages.dat ,too.
XTOOLs (older) setting was : 128mb
XTOOLs (2008) setting was : 128mb ,d0 and d9 ,after this 256mb. (doesnt matter)

In my other test i faced with this (dont remember what game,maybe Hellbound):
XTool 2008 inflation was 2 gigs AFAIR ,compressed with LOLZ ,after that compressed the game without XTOOL2008 and the same LOLZ parameters and it was smaller with some MBs ! (maybe a special case ,dont know)

ATM im away from my computer for the next...lot of days ,i cant send example files.

Keep up the good work !
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 15:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 18:16



All times are GMT -7. The time now is 14:19.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com