Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #121  
Old 20-01-2020, 11:37
doofoo24 doofoo24 is offline
Registered User
 
Join Date: Nov 2016
Location: canada
Posts: 408
Thanks: 138
Thanked 460 Times in 227 Posts
doofoo24 is on a distinguished road
Attached Images
File Type: gif ezgif.com-add-text.gif (956.8 KB, 614 views)
Reply With Quote
The Following 9 Users Say Thank You to doofoo24 For This Useful Post:
78372 (20-01-2020), devil777 (20-01-2020), dixen (20-01-2020), Gehrman (24-01-2020), Harsh ojha (21-01-2020), Razor12911 (20-01-2020), ShivShubh (21-01-2020), Simorq (20-01-2020), ZAZA4EVER (30-01-2020)
Sponsored Links
  #122  
Old 10-05-2020, 01:05
traxgaming traxgaming is offline
Registered User
 
Join Date: May 2020
Location: india
Posts: 8
Thanks: 5
Thanked 1 Time in 1 Post
traxgaming is on a distinguished road
What is Precomp ?

What is Precomp the Files ?
What is the advantage of it ?
How to use it?
What are the methods use it ?
Reply With Quote
  #123  
Old 06-08-2020, 18:51
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,646
Thanks: 2,026
Thanked 10,417 Times in 2,211 Posts
Razor12911 is on a distinguished road
Finally an update after a year (08/2019 - 08/2020).

Project is restarted and I hope all goes well this time.

Xtool will be getting frequent updates from now onward since the base code is solid and can allow for new features to be added.

If you have been participating in the [Dev]Xtool thread, thanks a lot. I have managed to fix a lot of bugs and came to a conclusion. Use reflate at your own risk, an alternative has been added which is preflate but this cannot process all streams so choose wisely.

Changes from the Dev Xtool

Added stats monitoring (not visible when using xtool with FA via stdio mode)
Added depth setting (finding streams within streams)
Fixed low memory mode
Speed improvements (when dealing with files with a lot of streams)

Plans for next update
Add stream deduplication

Last edited by Grumpy; 07-08-2020 at 03:54.
Reply With Quote
The Following 20 Users Say Thank You to Razor12911 For This Useful Post:
78372 (06-08-2020), Andu21 (07-08-2020), ARTEMiS14 (08-08-2020), COPyCAT (23-08-2020), devil777 (06-08-2020), DiCaPrIo (07-08-2020), dixen (07-08-2020), doofoo24 (07-08-2020), ffmla (06-08-2020), FitGirl (07-08-2020), Gehrman (16-08-2020), giothothan (08-08-2020), Grumpy (06-08-2020), Harsh ojha (06-08-2020), Masquerade (07-08-2020), PsYcHo_RaGE (08-08-2020), rambohazard (07-08-2020), shazzla (06-08-2020), vint56 (06-08-2020), ZAZA4EVER (07-08-2020)
  #124  
Old 07-08-2020, 02:42
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 310
Thanks: 368
Thanked 339 Times in 159 Posts
dixen is on a distinguished road
Dishonored. Death of the Outsider

Unpack

Quote:
Tested 3 files, 13,824,493,421 => 9,663,676,416 bytes. Ratio 143.06%
Testing time: cpu 8.91 sec/real 239.61 sec = 4%. Speed 40.33 mB/s
All OK
Reply With Quote
The Following 3 Users Say Thank You to dixen For This Useful Post:
doofoo24 (07-08-2020), Razor12911 (09-08-2020), shazzla (10-08-2020)
  #125  
Old 09-08-2020, 15:30
panker1992's Avatar
panker1992 panker1992 is offline
Registered User
 
Join Date: Oct 2015
Location: Always Somewhere
Posts: 527
Thanks: 113
Thanked 753 Times in 294 Posts
panker1992 is on a distinguished road
From My POV,

Precompression is give/take antibalanced situation.

Where i see it if you want a game decompressed 100% go ahead and decompress manually all then pack them,
and waste time and space when unpacking to do the same function reversed.

If we can get 98% os the files inputed decompressed we can go ahead and call it a day

2% is withing the margin of error
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows
My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45.
Reply With Quote
The Following User Says Thank You to panker1992 For This Useful Post:
Razor12911 (09-08-2020)
  #126  
Old 09-08-2020, 20:53
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,646
Thanks: 2,026
Thanked 10,417 Times in 2,211 Posts
Razor12911 is on a distinguished road
I was supposed to post an update on the project 2 days ago but then I was greeted by this cloudflare error each time I tried to open the site.

35.PNG

So I just thought the site was down little did I know that either my ISP or the forum blocked my IP address because I used VPN and the site opened.

Perhaps I spend too much time on the forum
Reply With Quote
  #127  
Old 10-08-2020, 04:41
Grumpy's Avatar
Grumpy Grumpy is offline
Moderator
 
Join Date: Jun 2004
Location: Australia
Posts: 5,658
Thanks: 1,169
Thanked 1,771 Times in 769 Posts
Grumpy is on a distinguished road
@Razor12911

I quite often get that error when updating the Conversion Index, I just figure it takes to long to update so much text each time ... when I do encounter the error I just open the Index in another window and I find it has updated.
__________________
Can't find a Game Conversion? Check the 'Conversion INDEX'
Reply With Quote
The Following User Says Thank You to Grumpy For This Useful Post:
Razor12911 (11-08-2020)
  #128  
Old 13-08-2020, 07:24
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 776
Thanks: 177
Thanked 849 Times in 414 Posts
Masquerade is on a distinguished road
Unhappy

Hello, following guidance from the GFS thread, I tried the zlib+preflate method for the dynamic streams located in Automobilista 2, however the files haven't been precompressed. Here is the result:

Code:
Compressed 1,438 files, 27,663,572,443 => 27,663,586,865 bytes. Ratio 100.00%
Compression time: cpu 27.78 sec/real 728.72 sec = 4%. Speed 37.96 mB/s
All OK
This makes me think that this method is not correct for this game.

XTool is doing SOMETHING though, since there is CPU and RAM usage, just as if it was processing normal zlib streams.

Any thoughts? I can upload some of the files if anyone is interested.
Reply With Quote
  #129  
Old 13-08-2020, 07:57
shazzla shazzla is offline
Registered User
 
Join Date: Nov 2010
Location: Hunnia
Posts: 223
Thanks: 411
Thanked 86 Times in 63 Posts
shazzla is on a distinguished road
I dont have this game,but are you sure its zlib?! (Ok,gfs.....)
Could it be Oodle?! I dont know,yust a question!
Reply With Quote
  #130  
Old 13-08-2020, 09:59
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 776
Thanks: 177
Thanked 849 Times in 414 Posts
Masquerade is on a distinguished road
Quote:
Originally Posted by shazzla View Post
I dont have this game,but are you sure its zlib?! (Ok,gfs.....)
Could it be Oodle?! I dont know,yust a question!
There are oodle libs in game folder, but this game is definitely zlib (as shown in GFS, with dynamic stream detection enabled). Oodle scan detects nothing.
Reply With Quote
  #131  
Old 13-08-2020, 10:35
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,620
Thanks: 954
Thanked 5,901 Times in 2,182 Posts
KaktoR is on a distinguished road
Maybe files are crypted?
__________________
Haters gonna hate
Reply With Quote
  #132  
Old 13-08-2020, 10:35
shazzla shazzla is offline
Registered User
 
Join Date: Nov 2010
Location: Hunnia
Posts: 223
Thanks: 411
Thanked 86 Times in 63 Posts
shazzla is on a distinguished road
If there r oodle libs then those streams r oodle streams imho.
Doesnt matter what scanners says its zlib.
Try with different oodle libs.
Again : imho. But maybe im very wrong...

@Kaktor : +1
Reply With Quote
  #133  
Old 13-08-2020, 11:35
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 776
Thanks: 177
Thanked 849 Times in 414 Posts
Masquerade is on a distinguished road
Unhappy Oh dear

Quote:
Originally Posted by KaktoR View Post
Maybe files are crypted?
The files are definitely encrypted. Sorry for the problems...

500mb sample ---> 450mb when 7z compressed.
Reply With Quote
  #134  
Old 13-08-2020, 16:40
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,646
Thanks: 2,026
Thanked 10,417 Times in 2,211 Posts
Razor12911 is on a distinguished road
Next update will have two new features, database and stream deduplication and here are some benchmarks as to what these features can do.

Input is Need for Speed Hot Pursuit\SEACREST

pzlib v2
Compressed 225 files, 3,253,110,016 => 859,932,872 bytes. Ratio 26.43%
Compression time: cpu 5.73 sec/real 300.50 sec = 2%. Speed 10.83 mB/s

pzlib v3
fails, no idea why

xtool2018 (v012)
Compressed 225 files, 3,253,110,016 => 722,889,785 bytes. Ratio 22.22%
Compression time: cpu 5.06 sec/real 188.09 sec = 3%. Speed 17.30 mB/s

xtool2019 (1908_R6)
Compressed 225 files, 3,253,110,016 => 722,934,572 bytes. Ratio 22.22%
Compression time: cpu 4.89 sec/real 189.14 sec = 3%. Speed 17.20 mB/s

xtool2020
Compressed 225 files, 3,253,110,016 => 722,927,890 bytes. Ratio 22.22%
Compression time: cpu 4.86 sec/real 165.26 sec = 3%. Speed 19.68 mB/s

xtool2020 with deduplication
Compressed 225 files, 3,253,110,016 => 719,194,176 bytes. Ratio 22.11%
Compression time: cpu 4.08 sec/real 154.45 sec = 3%. Speed 21.06 mB/s

xtool2020 with deduplication + memory database
Compressed 225 files, 3,253,110,016 => 719,195,766 bytes. Ratio 22.11%
Compression time: cpu 4.20 sec/real 75.00 sec = 6%. Speed 43.38 mB/s

xtool2020 with deduplication + imported database
Compressed 225 files, 3,253,110,016 => 719,194,905 bytes. Ratio 22.11%
Compression time: cpu 4.50 sec/real 61.77 sec = 7%. Speed 52.66 mB/s

precomp
Compressed 225 files, 3,253,110,016 => 754,538,183 bytes. Ratio 23.19%
Compression time: cpu 6.69 sec/real 547.05 sec = 1%. Speed 5.95 mB/s

In precomp's defence, it uses disk instead of memory to work and stream type was never narrowed to only focus on deflate streams.

All tests ran using 4 threads with 32mb chunk size

NOTE: This input has repeated streams, some games have similar characteristics therefore xtool stores a database in memory to remember the configuration and then removes these repeated streams to give more speed.
It's free speed, there is no penalty in performance if you use the new options at all times however, some tests need to be run due to the possibility of collisions in the dictionary used...

This project has come a long way

Last edited by Razor12911; 13-08-2020 at 16:53.
Reply With Quote
The Following 20 Users Say Thank You to Razor12911 For This Useful Post:
78372 (13-08-2020), ARTEMiS14 (13-08-2020), Cesar82 (13-08-2020), COPyCAT (23-08-2020), DedSec (15-08-2020), DiCaPrIo (14-08-2020), dixen (15-08-2020), ff2009 (15-08-2020), ffmla (13-08-2020), FitGirl (14-08-2020), Gehrman (16-08-2020), Gupta (14-08-2020), Harsh ojha (13-08-2020), L0v3craft (13-08-2020), Mortal Lord (14-08-2020), Perykles (15-08-2020), PsYcHo_RaGE (14-08-2020), shazzla (13-08-2020), vint56 (13-08-2020), ZAZA4EVER (13-08-2020)
  #135  
Old 14-08-2020, 08:31
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 192
Thanks: 17
Thanked 232 Times in 125 Posts
FitGirl is on a distinguished road
Cool update, waiting impatiently
Any predictions on RAM requirements for both imported/memory DBs? I suspect it depends on number of duplicated streams, but it would be good beforehand to know what additional RAM will be required upon recompression. Maybe it's possible for xtool to report it after initial decompression is done?
Reply With Quote
The Following 2 Users Say Thank You to FitGirl For This Useful Post:
Cesar82 (14-08-2020), shazzla (14-08-2020)
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 15:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 18:16



All times are GMT -7. The time now is 23:31.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com