#121
|
|||
|
|||
|
The Following 9 Users Say Thank You to doofoo24 For This Useful Post: | ||
78372 (20-01-2020), devil777 (20-01-2020), dixen (20-01-2020), Gehrman (24-01-2020), Harsh ojha (21-01-2020), Razor12911 (20-01-2020), ShivShubh (21-01-2020), Simorq (20-01-2020), ZAZA4EVER (30-01-2020) |
Sponsored Links |
#122
|
|||
|
|||
What is Precomp ?
What is Precomp the Files ?
What is the advantage of it ? How to use it? What are the methods use it ? |
#123
|
||||
|
||||
Finally an update after a year (08/2019 - 08/2020).
Project is restarted and I hope all goes well this time. Xtool will be getting frequent updates from now onward since the base code is solid and can allow for new features to be added. If you have been participating in the [Dev]Xtool thread, thanks a lot. I have managed to fix a lot of bugs and came to a conclusion. Use reflate at your own risk, an alternative has been added which is preflate but this cannot process all streams so choose wisely. Changes from the Dev Xtool Added stats monitoring (not visible when using xtool with FA via stdio mode) Added depth setting (finding streams within streams) Fixed low memory mode Speed improvements (when dealing with files with a lot of streams) Plans for next update Add stream deduplication Last edited by Grumpy; 07-08-2020 at 04:54. |
The Following 20 Users Say Thank You to Razor12911 For This Useful Post: | ||
78372 (06-08-2020), Andu21 (07-08-2020), ARTEMiS14 (08-08-2020), COPyCAT (23-08-2020), devil777 (06-08-2020), DiCaPrIo (07-08-2020), dixen (07-08-2020), doofoo24 (07-08-2020), ffmla (06-08-2020), FitGirl (07-08-2020), Gehrman (16-08-2020), giothothan (08-08-2020), Grumpy (06-08-2020), Harsh ojha (06-08-2020), Masquerade (07-08-2020), PsYcHo_RaGE (08-08-2020), rambohazard (07-08-2020), shazzla (06-08-2020), vint56 (06-08-2020), ZAZA4EVER (07-08-2020) |
#124
|
|||
|
|||
Dishonored. Death of the Outsider
Unpack Quote:
|
The Following 3 Users Say Thank You to dixen For This Useful Post: | ||
#125
|
||||
|
||||
From My POV,
Precompression is give/take antibalanced situation. Where i see it if you want a game decompressed 100% go ahead and decompress manually all then pack them, and waste time and space when unpacking to do the same function reversed. If we can get 98% os the files inputed decompressed we can go ahead and call it a day 2% is withing the margin of error
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45. |
The Following User Says Thank You to panker1992 For This Useful Post: | ||
Razor12911 (09-08-2020) |
#126
|
||||
|
||||
I was supposed to post an update on the project 2 days ago but then I was greeted by this cloudflare error each time I tried to open the site.
35.PNG So I just thought the site was down little did I know that either my ISP or the forum blocked my IP address because I used VPN and the site opened. Perhaps I spend too much time on the forum |
#127
|
||||
|
||||
@Razor12911
I quite often get that error when updating the Conversion Index, I just figure it takes to long to update so much text each time ... when I do encounter the error I just open the Index in another window and I find it has updated.
__________________
Can't find a Game Conversion? Check the 'Conversion INDEX'
|
The Following User Says Thank You to Grumpy For This Useful Post: | ||
Razor12911 (11-08-2020) |
#128
|
||||
|
||||
Hello, following guidance from the GFS thread, I tried the zlib+preflate method for the dynamic streams located in Automobilista 2, however the files haven't been precompressed. Here is the result:
Code:
Compressed 1,438 files, 27,663,572,443 => 27,663,586,865 bytes. Ratio 100.00% Compression time: cpu 27.78 sec/real 728.72 sec = 4%. Speed 37.96 mB/s All OK XTool is doing SOMETHING though, since there is CPU and RAM usage, just as if it was processing normal zlib streams. Any thoughts? I can upload some of the files if anyone is interested. |
#129
|
|||
|
|||
I dont have this game,but are you sure its zlib?! (Ok,gfs.....)
Could it be Oodle?! I dont know,yust a question! |
#130
|
||||
|
||||
There are oodle libs in game folder, but this game is definitely zlib (as shown in GFS, with dynamic stream detection enabled). Oodle scan detects nothing.
|
#131
|
||||
|
||||
Maybe files are crypted?
__________________
Haters gonna hate
|
#132
|
|||
|
|||
If there r oodle libs then those streams r oodle streams imho.
Doesnt matter what scanners says its zlib. Try with different oodle libs. Again : imho. But maybe im very wrong... @Kaktor : +1 |
#133
|
||||
|
||||
Oh dear
The files are definitely encrypted. Sorry for the problems...
500mb sample ---> 450mb when 7z compressed. |
#134
|
||||
|
||||
Next update will have two new features, database and stream deduplication and here are some benchmarks as to what these features can do.
Input is Need for Speed Hot Pursuit\SEACREST pzlib v2 Compressed 225 files, 3,253,110,016 => 859,932,872 bytes. Ratio 26.43% Compression time: cpu 5.73 sec/real 300.50 sec = 2%. Speed 10.83 mB/s pzlib v3 fails, no idea why xtool2018 (v012) Compressed 225 files, 3,253,110,016 => 722,889,785 bytes. Ratio 22.22% Compression time: cpu 5.06 sec/real 188.09 sec = 3%. Speed 17.30 mB/s xtool2019 (1908_R6) Compressed 225 files, 3,253,110,016 => 722,934,572 bytes. Ratio 22.22% Compression time: cpu 4.89 sec/real 189.14 sec = 3%. Speed 17.20 mB/s xtool2020 Compressed 225 files, 3,253,110,016 => 722,927,890 bytes. Ratio 22.22% Compression time: cpu 4.86 sec/real 165.26 sec = 3%. Speed 19.68 mB/s xtool2020 with deduplication Compressed 225 files, 3,253,110,016 => 719,194,176 bytes. Ratio 22.11% Compression time: cpu 4.08 sec/real 154.45 sec = 3%. Speed 21.06 mB/s xtool2020 with deduplication + memory database Compressed 225 files, 3,253,110,016 => 719,195,766 bytes. Ratio 22.11% Compression time: cpu 4.20 sec/real 75.00 sec = 6%. Speed 43.38 mB/s xtool2020 with deduplication + imported database Compressed 225 files, 3,253,110,016 => 719,194,905 bytes. Ratio 22.11% Compression time: cpu 4.50 sec/real 61.77 sec = 7%. Speed 52.66 mB/s precomp Compressed 225 files, 3,253,110,016 => 754,538,183 bytes. Ratio 23.19% Compression time: cpu 6.69 sec/real 547.05 sec = 1%. Speed 5.95 mB/s In precomp's defence, it uses disk instead of memory to work and stream type was never narrowed to only focus on deflate streams. All tests ran using 4 threads with 32mb chunk size NOTE: This input has repeated streams, some games have similar characteristics therefore xtool stores a database in memory to remember the configuration and then removes these repeated streams to give more speed. It's free speed, there is no penalty in performance if you use the new options at all times however, some tests need to be run due to the possibility of collisions in the dictionary used... This project has come a long way Last edited by Razor12911; 13-08-2020 at 17:53. |
The Following 20 Users Say Thank You to Razor12911 For This Useful Post: | ||
78372 (13-08-2020), ARTEMiS14 (13-08-2020), Cesar82 (13-08-2020), COPyCAT (23-08-2020), DedSec (15-08-2020), DiCaPrIo (14-08-2020), dixen (15-08-2020), ff2009 (15-08-2020), ffmla (13-08-2020), FitGirl (14-08-2020), Gehrman (16-08-2020), Gupta (14-08-2020), Harsh ojha (13-08-2020), L0v3craft (13-08-2020), Mortal Lord (14-08-2020), Perykles (15-08-2020), PsYcHo_RaGE (14-08-2020), shazzla (13-08-2020), vint56 (13-08-2020), ZAZA4EVER (13-08-2020) |
#135
|
|||
|
|||
Cool update, waiting impatiently
Any predictions on RAM requirements for both imported/memory DBs? I suspect it depends on number of duplicated streams, but it would be good beforehand to know what additional RAM will be required upon recompression. Maybe it's possible for xtool to report it after initial decompression is done? |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
[Dev]XTool | Razor12911 | Conversion Tutorials | 180 | 23-10-2020 07:26 |
Project Cars Digital Edition (3xDVD5) (srep+lzma) | GTX590 | PC Games - CD/DVD Conversions | 10 | 28-08-2017 09:34 |
Project IGI Anthology 1xCD700 CIUV2 2039 | mausschieber | PC Games - CD/DVD Conversions | 0 | 24-07-2017 16:12 |
Space Channel 5 Part 2 Translation Project | Christuserloeser | DC Games | 0 | 21-06-2004 19:16 |