FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   XTool 2020 (Main Project) (https://fileforums.com/showthread.php?t=102832)

doofoo24 20-01-2020 11:37

1 Attachment(s)
:cool:

traxgaming 10-05-2020 01:05

What is Precomp ?
 
What is Precomp the Files ?
What is the advantage of it ?
How to use it?
What are the methods use it ?

Razor12911 06-08-2020 18:51

Finally an update after a year (08/2019 - 08/2020).

Project is restarted and I hope all goes well this time.

Xtool will be getting frequent updates from now onward since the base code is solid and can allow for new features to be added.

If you have been participating in the [Dev]Xtool thread, thanks a lot. I have managed to fix a lot of bugs and came to a conclusion. Use reflate at your own risk, an alternative has been added which is preflate but this cannot process all streams so choose wisely.

Changes from the Dev Xtool

Added stats monitoring (not visible when using xtool with FA via stdio mode)
Added depth setting (finding streams within streams)
Fixed low memory mode
Speed improvements (when dealing with files with a lot of streams)

Plans for next update
Add stream deduplication

dixen 07-08-2020 02:42

Dishonored. Death of the Outsider

Unpack

Quote:

Tested 3 files, 13,824,493,421 => 9,663,676,416 bytes. Ratio 143.06%
Testing time: cpu 8.91 sec/real 239.61 sec = 4%. Speed 40.33 mB/s
All OK

panker1992 09-08-2020 15:30

From My POV,

Precompression is give/take antibalanced situation.

Where i see it if you want a game decompressed 100% go ahead and decompress manually all then pack them,
and waste time and space when unpacking to do the same function reversed.

If we can get 98% os the files inputed decompressed we can go ahead and call it a day :D

2% is withing the margin of error :D

Razor12911 09-08-2020 20:53

1 Attachment(s)
I was supposed to post an update on the project 2 days ago but then I was greeted by this cloudflare error each time I tried to open the site.

Attachment 27668

So I just thought the site was down little did I know that either my ISP or the forum blocked my IP address because I used VPN and the site opened.

Perhaps I spend too much time on the forum :D

Grumpy 10-08-2020 04:41

@Razor12911

I quite often get that error when updating the Conversion Index, I just figure it takes to long to update so much text each time ... when I do encounter the error I just open the Index in another window and I find it has updated. ;)

Masquerade 13-08-2020 07:24

Hello, following guidance from the GFS thread, I tried the zlib+preflate method for the dynamic streams located in Automobilista 2, however the files haven't been precompressed. Here is the result:

Code:

Compressed 1,438 files, 27,663,572,443 => 27,663,586,865 bytes. Ratio 100.00%
Compression time: cpu 27.78 sec/real 728.72 sec = 4%. Speed 37.96 mB/s
All OK

This makes me think that this method is not correct for this game.

XTool is doing SOMETHING though, since there is CPU and RAM usage, just as if it was processing normal zlib streams.

Any thoughts? I can upload some of the files if anyone is interested.

shazzla 13-08-2020 07:57

I dont have this game,but are you sure its zlib?! (Ok,gfs.....)
Could it be Oodle?! I dont know,yust a question!

Masquerade 13-08-2020 09:59

Quote:

Originally Posted by shazzla (Post 487399)
I dont have this game,but are you sure its zlib?! (Ok,gfs.....)
Could it be Oodle?! I dont know,yust a question!

There are oodle libs in game folder, but this game is definitely zlib (as shown in GFS, with dynamic stream detection enabled). Oodle scan detects nothing.

KaktoR 13-08-2020 10:35

Maybe files are crypted?

shazzla 13-08-2020 10:35

If there r oodle libs then those streams r oodle streams imho.
Doesnt matter what scanners says its zlib.
Try with different oodle libs.
Again : imho. But maybe im very wrong...

@Kaktor : +1

Masquerade 13-08-2020 11:35

Oh dear
 
Quote:

Originally Posted by KaktoR (Post 487403)
Maybe files are crypted?

The files are definitely encrypted. Sorry for the problems... :(

500mb sample ---> 450mb when 7z compressed.

Razor12911 13-08-2020 16:40

Next update will have two new features, database and stream deduplication and here are some benchmarks as to what these features can do.

Input is Need for Speed Hot Pursuit\SEACREST

pzlib v2
Compressed 225 files, 3,253,110,016 => 859,932,872 bytes. Ratio 26.43%
Compression time: cpu 5.73 sec/real 300.50 sec = 2%. Speed 10.83 mB/s

pzlib v3
fails, no idea why :o

xtool2018 (v012)
Compressed 225 files, 3,253,110,016 => 722,889,785 bytes. Ratio 22.22%
Compression time: cpu 5.06 sec/real 188.09 sec = 3%. Speed 17.30 mB/s

xtool2019 (1908_R6)
Compressed 225 files, 3,253,110,016 => 722,934,572 bytes. Ratio 22.22%
Compression time: cpu 4.89 sec/real 189.14 sec = 3%. Speed 17.20 mB/s

xtool2020
Compressed 225 files, 3,253,110,016 => 722,927,890 bytes. Ratio 22.22%
Compression time: cpu 4.86 sec/real 165.26 sec = 3%. Speed 19.68 mB/s

xtool2020 with deduplication
Compressed 225 files, 3,253,110,016 => 719,194,176 bytes. Ratio 22.11%
Compression time: cpu 4.08 sec/real 154.45 sec = 3%. Speed 21.06 mB/s

xtool2020 with deduplication + memory database
Compressed 225 files, 3,253,110,016 => 719,195,766 bytes. Ratio 22.11%
Compression time: cpu 4.20 sec/real 75.00 sec = 6%. Speed 43.38 mB/s

xtool2020 with deduplication + imported database
Compressed 225 files, 3,253,110,016 => 719,194,905 bytes. Ratio 22.11%
Compression time: cpu 4.50 sec/real 61.77 sec = 7%. Speed 52.66 mB/s

precomp :(
Compressed 225 files, 3,253,110,016 => 754,538,183 bytes. Ratio 23.19%
Compression time: cpu 6.69 sec/real 547.05 sec = 1%. Speed 5.95 mB/s

In precomp's defence, it uses disk instead of memory to work and stream type was never narrowed to only focus on deflate streams.

All tests ran using 4 threads with 32mb chunk size

NOTE: This input has repeated streams, some games have similar characteristics therefore xtool stores a database in memory to remember the configuration and then removes these repeated streams to give more speed.
It's free speed, there is no penalty in performance if you use the new options at all times however, some tests need to be run due to the possibility of collisions in the dictionary used...

This project has come a long way :)

FitGirl 14-08-2020 08:31

Cool update, waiting impatiently :)
Any predictions on RAM requirements for both imported/memory DBs? I suspect it depends on number of duplicated streams, but it would be good beforehand to know what additional RAM will be required upon recompression. Maybe it's possible for xtool to report it after initial decompression is done?


All times are GMT -7. The time now is 17:51.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com