FileForums - Best Compression Methods for 'Specific' Games. Q&A

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

Handyman Corporation

Generate a database for Oodle Kraken using "unreal_tournament_4_0.4.27c_paks_only.bms".

Don't use oo2reck - it seems to have a memory leak during decompression (eating up all your ram) and is insanely slow.

For the pak file, use your database with XTool dedup (requires srep.exe) and add lolz.

For the rest use rz.

2GB --> 1.3GB

Results for most TT LEGO Games

Tools used:

oggre + msc + srep + lolz

Note applicable for all titles: MSC does work on a majority of the .wav/.mp3 loose files as well as the container files used in some titles, such as the "LEGO Harry Potter" titles as well as the "LEGO Middle-Earth" titles, however it's recommend to do a bulk test on all applicable files as there are a fair few instances of CRC-mismatches and there isn't an easy means to identify this without testing. The same can be said for all applicable Vorbis audio. Newer waveform audio in the "*.ADP" container format cannot be processed currently. I haven't tested to see if the engine will accept re-coded audio in the standard and compressible PCM format, given the fact the engine used for all TT titles (bar maybe the Skywalker Saga, given its a fairly considerable engine revision) is building upon the foundations of the previous iteration, it might (?) be possible to do so. Same can probably be done for the various incompatible "*.ogg" files given the format is the same, and some chunks just don't play well with Oggre.

Files ripped: Non-English Loc., unused left-over files from the console releases & left-over demo-exclusive files.

Final notes:

- All tests were based off the latest clean GOG/Steam RIPs.

- Some decompressed sizes may be slightly larger than what others may experience as I tend to include any and all QoL fixes/mods as included offerings, and only include larger less critical mods like the soundtrack replacement mods for the various licensed titles by TT, as optional component offerings.

///////////////////////////////

LEGO City Undercover
15.20 GiB >>> 5.07 GiB

LEGO Harry Potter: Years 1-4
8.54 GiB >>> 1.77 GiB

LEGO Harry Potter: Years 5-7
12.20 GiB >>> 2.73 GiB

LEGO Indiana Jones: The Original Adventures
5.77 GiB >>> 1.41 GiB

LEGO Indiana Jones 2: The Adventure Continues
7.15 GiB >>> 1.50 GiB

LEGO Jurassic World
20.70 GiB >>> 4.52 GiB

LEGO MARVEL Super-Heroes
9.38 GiB >>> 2.30 GiB

LEGO MARVEL Avengers
22.30 GiB >>> 5.30 GiB

LEGO MARVEL Super-Heroes 2
26.73 GiB >>> 7.43 GiB

LEGO Middle-Earth: The Lord of the Rings
8.07 GiB >>> 2.24 GiB

LEGO Middle-Earth: The Hobbit
8.75 GiB >>> 2.69 GiB

LEGO Movie Videogame, The
9.79 GiB >>> 3.89 GiB

LEGO Movie Videogame, The 2
15.00 GiB >>> 3.65 GiB

LEGO Ninjago Movie Videogame, The
29.60 GiB >>> 6.09 GiB

LEGO Pirates of the Caribbean: The Videogame
11.70 GiB >>> 2.17 GiB

LEGO Star Wars - The Complete Saga
9.39 GiB >>> 1.61 GiB

LEGO Star Wars III - The Clone Wars
9.78 GiB >>> 2.10 GiB

LEGO Star Wars: The Force Awakens
28.10 GiB >>> 6.61 GiB

LEGO The Incredibles
22.70 GiB >>> 4.61 GiB

LEGO Worlds
4.36 GiB >>> 1.24 GiB

LEGO DC Batman: The Videogame
8.02 GiB >>> 1.32 GiB

LEGO DC Batman 2: DC Super Heroes
5.62 GiB >>> 1.26 GiB

LEGO DC Batman 3: Beyond Gotham
12.10 GiB >>> 3.33 GiB

LEGO DC Super-Villains
26.14 GiB >>> 6.62 GiB

Quick question for anyone who's given "Days Gone" a go, has anyone been able to process both the oodle and zlib streams present in the title's "*.pak" archives? I've been using oo2reck (since xtool seems to only be able to use one oodle library and both a "oo2core_7*" & "oo2core_8*" are required to grab all (or at least the majority) of oodle streams. I've tried using both "oo2reck + xtool_0.3.21 (zlib+reflate) + srep + lolz" as well as "xtool_0.3.21 (zlib+reflate) + oo2reck + srep + lolz" but to no avail, interestingly the latter combination results in ~15mins of memory usage with srep and disk write operations, but unfortunately also results in a -11 (corrupt archive) FreeArc decompression error code.

If anyone has had any luck or knows how to get xtool to use both libraries I'd be very much interested in any tips anyone would be willing to offer. Attached are the libraries I've found to be the most effective.

Quote:

Originally Posted by L33THAK0R (Post 500026)

Use two xtool instances with --oodle option in arc.ini like this:

Code:

[External compressor:xtool1]

header = 0

packcmd   = xtool.exe precomp -mkraken -c64mb -t100p --oodle=oo2core_7_win64.dll - - <stdin> <stdout>

unpackcmd = xtool.exe decode -t75p - - <stdin> <stdout>



[External compressor:xtool2]

header = 0

packcmd   = xtool.exe precomp -mkraken -c64mb -t100p --oodle=oo2core_8_win64.dll - - <stdin> <stdout>

unpackcmd = xtool.exe decode -t75p - - <stdin> <stdout>

Then use xtool1+xtool2+xtool:zlib

A plague tale requiem (1.4.0.0 - build 10113825)

Code:

Game Size: 47/4 GB

Method#1

Code:

xtool:c256mb:mzlib:mpreflate:dd3+4x4:lzma:a1:mfbt4:32m:fb273:mc128:lc8

Code:

30.5 GB

Method#2

Code:

xtool:mreflate:dd3+4x4:lzma:a1:mfbt4:32m:fb273:mc1024:lc8

Code:

32.3 GB

A Plague Tale: Requiem
v1.4.0.0, ripped languages from soundbanks

Code:

 10:58:07 - Selected ARC/DS method for Data1a-01.bin was: xtool:mreflate+xtool:db:dd3+4x4:lzma

 10:58:07 - Selected ARC/DS method for Data1b-01.bin was: xtool:db:dd3+4x4:lzma

-------------------------------------------------------------------------------------------------------------------------------------------

 12:16:24 - Overall input size: 47.48 GB

 12:16:24 - Overall output size: 29.86 GB (Ratio 62.89%)

 12:16:24 - Overall conversion time: 01:18:16

Dead Space Remake

Code:

Library loaded: zstd_145\libzstd.dll



Streams: 7656/7657

Time: 00:00:16 (00:01:33)

Memory: 177 MB (177 MB)



Results: 230 MB >> 478 MB

Edit:

Code:

 20:14:55 - Selected ARC/DS method for Data1a-01.bin was: xtool:mzstd:zstd_145+xtool:db:dd3+4x4:lzma

 20:14:55 - Selected ARC/DS method for Data1b-01.bin was: xtool:db:dd3+4x4:lzma

-------------------------------------------------------------------------------------------------------------------------------------------

 21:32:58 - Overall input size: 36.32 GB

 21:32:58 - Overall output size: 26.31 GB (Ratio 72.45%)

 21:32:58 - Overall conversion time: 01:18:01

-------------------------------------------------------------------------------------------------------------------------------------------

 21:57:32 - Overall decompression time: 00:23:12

 21:57:32 - All files have been successfully extracted!

Days Gone woes vol. 2

Seems like no matter how I go about attempting to pre-compress this title, I'm unable to pick up all the oodle streams, as a last resort I'm going to try unpacking the core 2 "*.pak" archives as a dirty solution and see if any gains can be had with this approach. Attached image displays my last two attempts to process all the Oodle streams with XTool (0.6.9).https://i.imgur.com/IPQmy0M.png

PowerWash Simulator [v1.1]

Use SREP v3.2 due to crash on newer versions.
Don't use XTool built in deduplication.

Code:

xtool_unity+srep+lolz

6.6GB ---> 3.3GB

Interesting results from packing significantly recoded H.264 video

A while back I took a shot at a title, "Trek to Yomi", which contains ~260mb of H.264 video, encoded at an exceedingly silly bit-rate. Linked here is a sample of my re-encoding.

The below figure should speak for itself, but after tuning ffmpeg a fair bit, I managed to get quite an extraordinary outcome, coupled with srep & lolz, a result of a 328x improvement with a near imperceivable loss in fidelity.

https://i.imgur.com/BgwlKUG.png

Has anyone else had any similar success stories? It's got me wondering how and why these algorithms were able to process these lower bit-rate frames in such an efficient manner compared to the original lossless files.

(And yes before anyone asks I did verify the archives to be valid)

For the records:

I recommend use CRF (lets say ,its constant quality) instead of targeting bitrate.

Quote:

Originally Posted by shazzla (Post 500066)

For the records:

I recommend use CRF (lets say ,its constant quality) instead of targeting bitrate.

Constant Rate Factor (CRF) encoding is pretty much mandatory I'd argue for producing any quality lossy videos. You can still use the available switches cap the maximum bit-rate as well manipulating the VBV buffer. I'd also recommend testing the various available tuning profiles (not to be confused with H.264 profiles or ffmpeg encoding presets) within ffmpeg especially if you're dealing with 2D video, as you can get some noticeable ratio improvements for a slightly longer encoding time.

Then give a shot for

Selur's Hybrid. Not ffmpeg.

I used it for encode TBs of x265 movies..

xtool (0.3.21) + srep + lolz

42.26 GiB >>> 19.4 GiB

Ripped: Non-Eng. Loc., ~10 MiB of startup videos

Note: the entire "sfpaks" folder can be deleted, I'm surprised others that have tackled this title don't seem to know it just contains CRC-perfect duplicates of data contained within the 2 primary "*.pak" archives. In my conversion I unpacked the archives to remove the need for oodle pre-compression, honestly there are only ~200 MiB of zlib/reflate streams (See Figure 1) to be processed which only results in a ~60 MiB improvement (See Figure 2) over just sticking with srep + lolz. In this form there are only ~6 MiB of Oodle streams to be processed.

Interestingly enough I found on my system, when testing for ~1hr to make sure everything worked properly, I encountered fewer stutters during game-play when using the unpacked files.

Figure 1
https://i.imgur.com/Knsc4jq.png

Figure 2
https://i.imgur.com/Hp0zUv6.png