Best Compression Methods for 'Specific' Games. Q&A

[L33THAK0R]

Source: Clean Steam Files

Version 1.10
Original Size: 27.3 GiB
Final Size: 10.705 GiB (Incl. installer)
Ripped Files: Non EN. loc.

Internal Tools used:

Xtool v0.8.9 (Codecs: reflate, zlib)
Srep
Lolz
Oggre

External Tools used:
SFK
DELZORec
MParallel

Additional notes: Happy with how this turned out, a substantial improvement over my previous results, the approx. ~200 MiB reduction can be attributed to more effective processing of the Vorbis OGG audio, as well as tighter tuning for the lolz model.

[Carldric Clement]

Zomborg v2017.3.0

Original Size: 2.21 GB
Compress Size: 378.3 MB

Method:

* rest of the files = xtool:zlib:preflate+srep:m3f:a2+exe2+lolz:d128m:mt4:mtt1:lmdf1

[L33THAK0R]

Source: Clean Steam Files

Build 17107881
Original Size: 96 GiB
Final Size: 76.6 GiB (Incl. installer)
Ripped Files: Non EN. loc.

Internal Tools used:

Xtool v0.8.9 (Codecs: reflate, zlib, unreal decrypt., oodle)
Srep
Lolz

Additional notes:

Opted to sacrifice size for speed for this title, on a 8c/16t machine install time was just over 11mins, versus my more aggressive testing which clocked in at ~45mins.

[L33THAK0R]

Details:

• Source: Clean GOG Files
• Version 1.2.0.23
• Original Size: 11 GiB
• Final Size (Without MSC) : 5.17 GiB (Incl. installer)
• Final Size (With MSC) : 4.84 GiB (Incl. installer)
• Ripped Files: None

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib, xmemcompress)
• Srep
• Lolz
• Oggre
• MSC

External Tools used:

• None

Additional notes:

I'm very inexperienced with using xmemcompress for applicable use-cases, massive thank you to Razor12911 for providing the config (attached below), to process the "*.xnb" containers, other community tools that are able to parse other XNA titles using "*.xnb" containers, weren't able to with those compressed using LZX. Attached is said config. In comparison with other published results, there's a minimum of 640MiB in potential gains left on the table, alas I can't for the life of me figure out where though. As this is the first LZX-compressed XNA title I've approached, it's entirely possible I'm just missing something obvious.

XNB Config:

[Stream1]
Name=xmemcompress
Codec=xmemcompress
BigEndian=0
Signature=0x77424E58
Structure=Signature(4),Unk1(2),CSize(4),DSize(4),Stream
StreamOffset=0
CompressedSize=CSize - 14
DecompressedSize=DSize
Condition2=CSize>=256
Condition1=DSize>CSize

[Masquerade]

^^
I wrote this config a few years ago and it still holds up for XNB titles (afair I used it on Terraria)

[Stream1]
Name=xnb
Codec=xmemcompress
BigEndian=0
Signature=0x77424E58
Structure=Signature(4),XNBFormatVersion(1),Compressed(1),FileSize(4),DSize(4),Stream
CompressedSize=FileSize-14
DecompressedSize=DSize
Condition1=Compressed = 0x81

XNB files have a byte two bytes after the 4 byte magic which is always 0x81 if the data is compressed, I was using this to simply skip uncompressed data.

Beyond this, the real magic for xcompress streams lies in xtool.ini and the parameters that get passed to the exe file.

[L33THAK0R]

Foreword: Thought I'd just post some of the more interesting projects in one large post rather than clog up the forums. Mainly to demonstrate that existing published results are able, (in some cases quite easily) to be improved outside of pure chance through non-deterministic algorithms. That being said, I'm still very much an amateur (see the "A Plague Tale: Requiem" & "Dead Island 2" entries), and despite my current knowledge and skill set, I still have plenty of room to learn and improve.

Red Dead Redemption (PC Port)

Details:

• Source: Clean Steam Files
• Build 16471194
• Original Size: 8.88 GiB
• Final Size : 5.16 GiB (Incl. Installer)
• Ripped Files: None

Internal Tools used:

• Xtool v0.5.3 (Codecs: reflate, zlib, zstd)
• Srep
• Lolz

External Tools used:

• None

Additional notes:

Unable to process audio, in the original console releases there were a few hundred MiB of Bink1 video embedded, in the form of a projector film. I was unable to isolate these videos in the PC release.

/////////////////////////////////

Dead Island 2

Details:

• Source: Clean Steam Files
• Build 20514355
• Original Size: 70.79/71.09 GiB
• Final Size : 47.90/48.20 GiB (Incl. Installer)
• Ripped Files: 2160p/1080p Video

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib, unreal, oodle)
• Srep
• Lolz

External Tools used:

• None

Additional notes:

Was lazy with this one due to juggling more interesting projects at the time, did not process Wwise audio.

/////////////////////////////////

The Darkness II

Details:

• Source: Clean Steam Files
• Build 11598
• Original Size: 6.70 GiB
• Final Size : 4.44 GiB (Incl. Installer)
• Ripped Files: Non EN Loc.

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib)
• MSC
• Srep
• Lolz

External Tools used:

• None

Additional notes:

None.

/////////////////////////////////

Dark Souls Remastered

Details:

• Source: Clean Steam Files
• Build 9694186
• Original Size: 6.86 GiB
• Final Size : 4.03 GiB (Incl. Installer)
• Ripped Files: Non EN Loc.

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib)
• MSC
• Srep
• Lolz
• Bpk

External Tools used:

• None

Additional notes:

None.

/////////////////////////////////

Dying Light - The Following (Definitive Edition)

Details:

• Source: Clean GOG Files
• Version 1.54.0
• Original Size: 27.6 GiB
• Final Size : 12.9 GiB (Incl. Installer)
• Ripped Files: Non EN Loc., Optional Main Menu Promo Videos

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib)
• Oggre
• Srep
• Lolz
• Bpk

External Tools used:

• HDiffZ

Additional notes:

To process all OGG Vorbis audio you will need to patch some archives.

/////////////////////////////////

Heavy Rain

Details:

• Source: Clean Steam Files
• Build 5236766
• Original Size: 31.70 GiB
• Final Size : 7.88 GiB (Incl. Installer)
• Ripped Files: Non EN Loc., Optional BTS Video

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib)
• MSC
• Srep
• Lolz
• Bpk

External Tools used:

• Mparallel
• SFK

Additional notes:

Audio can be processed with MSC, however not all streams are valid, you will need to generate a database and rip the extracted streams. As assets are only compressed and not encrypted, I recommend ripping the compressed streams processed seperateley, diffing the individual streams and patching these compressed streams post-install, if you don't care for the end-user's write-endurance you can just brute force it with less elegant methods. Same method applies for ripping localised content. Additional Bink1 videos are also embedded within the main data containers.

/////////////////////////////////

The Last of Us: Part I

Details:

• Source: Clean Steam Files
• Build 17034831
• Original Size: 62.8 GiB
• Final Size : 38.4 GiB (Incl. Installer)
• Ripped Files: Non EN Loc., Optional BTS Video, Credits

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib, oodle, psarc)
• Srep
• Lolz

External Tools used:

• None

Additional notes:

None.

/////////////////////////////////

Lorelei and the Laser Eyes

Details:

• Source: Clean Steam Files
• Build 15691260
• Original Size: 1.180 GiB
• Final Size : 0.535 GiB (Incl. Installer)
• Ripped Files: None

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib)
• Srep
• Lolz

External Tools used:

• None

Additional notes:

None.

/////////////////////////////////

A Plague Tale: Requiem

Details:

• Source: Clean GOG Files
• Build 56580004591411255
• Original Size: 49.4 GiB
• Final Size : 24.3 GiB (Incl. Installer)
• Ripped Files: Non EN Loc.

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib)
• Srep
• Lolz

External Tools used:

• SFK
• Mparallel
• Oggre (Wwise)

Additional notes:

PCK & BNK archives were processed, localised content is compressed within archives, rip streams using a database. See my "A Plague Tale: Innocence" results for more information.

/////////////////////////////////

Total War: Three Kingdoms

Details:

• Source: Scene Release
• Build 10356423
• Original Size: 39.5 GiB
• Final Size : 15.6 GiB (Incl. Installer)
• Ripped Files: Non EN Loc.

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib)
• Srep
• Lolz

External Tools used:

• SFK
• Mparallel
• Oggre (Wwise)

Additional notes:

Audio was processed externally, and patched post-install.

/////////////////////////////////

Trials Rising

Details:

• Source: Scene Release
• Build 10831717
• Original Size: 25.92 GiB
• Final Size : 15.83 GiB (Incl. Installer)
• Ripped Files: Non EN Loc. Videos

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib)
• Srep
• Lolz
• Bpk

External Tools used:

• None

Additional notes:

Wasn't able to extract audio for this title, it's one of the less documented "Trials" titles, tried a variety of solutions but no luck. Potentially future attempts could see some gains if audio is processed.

[wareck]

Originally Posted by Wanterlude

Star Wars: Squadrons

use zstd 1.2.0
or frostbite plugin by Razor12911, but it doesn't work for me

yes you need to modify frostbibe3.ini and add :

[Stream13]
// SWSquadrons
Name=swsquadrons
Codec=zstd
BigEndian=1
Signature=0x700F
Structure=Dict(1),DSize(3),Signature(2),CSize(2),Z StdHdr(4),Stream
StreamOffset=-4
CompressedSize=CSize
DecompressedSize=DSize
Condition1=DSize > CSize
Condition2=CSize > 64
Condition3=CSize <= 65536
Condition4=DSize <= 65536
Condition5=Dict=0
Condition6=ZStdHdr = 0x28B52FFD

then use xtool:c32mb:mfrostbite3,swsquadrons

[KaktoR]

Just use zstd without frostbite3 plugin.

-mzstd

Streams: 1221475 / 1221493
Time: 00:28:08 (CPU 03:36:37)
Duplicates: 557369 (1,97 GB) [2,75 GB >> 5,77 GB]
Srep decompression memory: 1,11 GB

Size: 31,3 GB >> 47,5 GB >> 41,8 GB >> 37,4 GB >> 22,9 GB

Done!!!

-mfrostbite3,zstd (your modified config)

Streams: 1209456 / 1209471
Time: 00:29:20 (CPU 03:45:03)
Duplicates: 545689 (1,97 GB) [2,75 GB >> 5,11 GB]
Srep decompression memory: 1,11 GB

Size: 31,3 GB >> 46,9 GB >> 41,8 GB >> 37,4 GB >> 22,9 GB

Done!!!

[L33THAK0R]

Details:

• Source: Clean Steam Files
• Build 13300582
• Original Size: 61.42 GiB
• Final Size: 45.25 GiB (Incl. Installer)
• Ripped Files: Studio credit video present on new game.

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib, oodle)
• Srep (v3.2.0)
• Lolz
• DST

External Tools used:

• None

Additional notes: Attached below are the XTool libraries (oodle being the important one), as well as a copy of Srep v3.2.0/the CLS version of srep I used for decomp, and DST (just so future users can find all the resources needed in one post). I couldn't get DST to work when I first tried this title around a year back, which it turns out was as a result of specifying an input/output in my arc.ini, rather than <stdin>/<stdout>.

I really tried to get the WEMs & BNKs processed, total savings totaled out to ~401 MiB (full results below), but I couldn't justify the total additional writes needed to patch the files back (patch WEMs > build tmp for diff > diff decomp'd output > decomp external archives) for such a small gain. More importantly I screwed up something along the way and my final output for all *.bin files was 34.4 GiB with processing vs 30.3 GiB without. I've done so many tests with this title I'll take this minor improvement and let someone else perfect this title in the future.

WEM Results:
Original Archive: 3460515fd6d930b689143a8b9a51a060.bin
Size: 2.381836 GiB
Size of Valid WEMs: 0.390625 GiB 

Original Archive: 40fb866e44aec444200cdb0c384cf5a5.bin
Size: 4.083984 GiB
Size of Valid WEMs: 0.126953 GiB 

Original Archive: 477e458b2c825633499874678a2b9ea5.bin
Size: 3.375977 GiB
Size of Valid WEMs: 0.448242 GiB 

Original Archive: 596c31fc02eca656d5ecdb546c52a7af.bin
Size: 1.516602 GiB
Size of Valid WEMs: 0.032227 GiB 

Original Archive: 7017f9bb9d52fc1c4433599203cc51b1.bin
Size: 6.795898 GiB
Size of Valid WEMs: 3.263672 GiB 

Original Archive: a53f66da5938a089f06b8a0c07c97f18.bin
Size: 0.563477 GiB
Size of Valid WEMs: 0.036133 GiB 

Original Archive: dc847b243b638c5ce539d4be7ff48007.bin
Size: 1.999023 GiB
Size of Valid WEMs: 0.05957 GiB 

Original Archive: fcd4b5918cea9a220c0da171bb60c214.bin
Size: 1.310547 GiB
Size of Valid WEMs: 0.048828 GiB 

All Savings: ~410 MiB

Videos were processed by first extracting all Bink files then rebuilding the hashed structure "<INSTALL_DIR>\movies\mono\*.bk2". Hashes can be found below (MD5):

acece703e713163e9fa1afbb4ab90b78 *sq_cs40_s20000.bk2
cc7f1ba54d38343449dde506c335f45e *sq_cs01_s00750.bk2
8cb3ad4675d07c15bd5b3d3d80667ca8 *sq_cs03_s01000.bk2
ba1ffd83f835eb433c195b41ce01c5ba *sq_cs10_s00700.bk2
e9aea7a3bdd6d07fc2a3b23dcf21f799 *sq_cs10_s90000.bk2
f6cb3f869106b223ba4a7f9aff04beb3 *sq_cs07_s00400.bk2
e0a62409c56963a1aa592a9129522617 *sq_cs01_s00900.bk2
b4a8d52213762d37e36259d7d326906f *sq_cs05_s01400.bk2
ba7e5e2e6f3fa28f71a046522f2d693d *sq_cs03_s01150.bk2
8adfc5e0c12d4592d2b27a647eb4efdd *sq_cs08_s00200.bk2
c53b0419aac17af547ed3c7f638e1958 *sq_cs08_s01080.bk2
fdc80f4a430c77d1b406707a388242f7 *sq_cs80_s00100_c004.bk2
583c15d6536b300d54021fc408b7c340 *sq_cs03_s00150.bk2
6fe7cc2224cd75370abf4fcc21ff03b6 *sq_cs80_s00100_c006.bk2
14903e6f2286f9a1fbd4a9f0047ed2d5 *sq_cs02_s00300.bk2
087dbdc8f364ca79fd16db054c16ca5e *sq_cs04_s00900.bk2
d0155f203ed7e8a36a313952a1291a09 *sq_cs80_s00100_c001.bk2
19a30487ddf45d26cd98e30c6290e70c *sq_cs80_s00100_c002.bk2
1baa25a412641509338737a87f806e06 *sq_cs04_s00700.bk2
ead776fbde6233b818688e11b86f5720 *sq_cs03_s00400.bk2
ea9612b9808a0582f651232487cffa8e *sq_cs05_s01500.bk2
0c73579bcf9db5d41088d31a48f39790 *sq_cs08_s00350.bk2
d594a38aa77a2e4028295e0116ccedbf *sq_cs06_s01100.bk2
6c4e17e46fd5dd4703c41102d656e60f *sq_cs03_s00490.bk2
74dd45ff115560379cf1726ad63587ca *sq_cs04_s01100.bk2
91a112e465106dada221417271b3fab6 *sq_cs80_s00100_c003.bk2
c96c77a4a2b6b4c38145ccb1cbec4788 *sq_cs08_s01020.bk2
242d14dcead2caa2c8bfc84491f17413 *sq_cs05_s00050.bk2
942a4cbed3d346229e34469c30a14c3d *sq_cs07_s00300.bk2
197bc090cba439a25c5d06741125af84 *sq_cs07_s00350.bk2
450e69c01a0b483524edff061ea0abf1 *sq_cs80_s00100_c007.bk2
41d0a1e1edacf4932450ff42cc14accf *sq_cs80_s00100_c005.bk2
e6ecda36874f0800885f33733aa9f561 *sq_cs06_s00360.bk2
7c84673df30c0fb76b612a9d77a9afe5 *sq_cs02_s00400.bk2
ae269228bd156da786fb1f92990f7847 *sq_cs02_s01500.bk2
3ec77daed9bf2cdd593346d7b450f638 *sq_cs02_s00700.bk2
0e55091c44cc9f405a528d0e70d66b79 *sq_cs10_s00440.bk2
9de4e36e3af2ca91e6f5c6c1a7e0ef5a *sq_cs00_s00100.bk2
9e625d144b95bd3ff9eb41f92e52df2b *sq_cs00_s00600.bk2
162c9837c0b849313e572fd5196d79b5 *sq_cs00_s00500.bk2

BPK picks up the streams but is unable to process them, it's still worth-while processing them with a final compressor of your choice.

Non-en loc. were also ripped, all relevant "*.bin" archives can be found below

93a37de4096fad67e0c99100775cce42.bin
4707c0dea95bdf7b38ed0d838965e0e6.bin
177469b03caa26c0651de930f368a3ad.bin
1dbc7dd0dcb80b83eade1ab381422ea5.bin
507b57fe3eb2170e71997df17e478a2d.bin
f70387b309fdda4e66a9d1808815252d.bin
6e0de0395842dfb82888e339230534e1.bin
73a5378183017eeacff7adcd23e66430.bin
84531a1f1930d4626bdb5f1679e94b0b.bin
7ccfce87d9887adf1689fbd07df4200a.bin
8d60c6e8d6ce70496107e588205a3054.bin
2b23c8aeeebf8df30f2caf770c9c9b1d.bin
6fb69851f3279ea051f48ee6c98c73fe.bin
51dc3286c8bb6d6139af5f7f5c0b6f59.bin
9abda4bd99f4fa01951fd153775e7e8d.bin
345184e6dd9b47d6587a8f9267a818a6.bin
9f7b09f5d311e88d090ae516e18d80df.bin
17881513a7e2a728ccd2ae7301dc9054.bin
a53f66da5938a089f06b8a0c07c97f18.bin
968bf82f34e2b499687c901a888e633a.bin
1a472f28ea1537dd3cb911fd1e18f2af.bin
a694c2e00b87cd761ee2a25b09a49ea7.bin
596c31fc02eca656d5ecdb546c52a7af.bin
3460515fd6d930b689143a8b9a51a060.bin
fcd4b5918cea9a220c0da171bb60c214.bin
477e458b2c825633499874678a2b9ea5.bin
7017f9bb9d52fc1c4433599203cc51b1.bin
732e43ff85a0e797a684c8a8c6e55627.bin
acc0d12f571a5b85f368497756cbcd3c.bin
b07acb141d08c46008235427b5c1a24a.bin
40fb866e44aec444200cdb0c384cf5a5.bin
c0306eef15c4b5122ab61a834f80d743.bin
59b95a781c9170b0d13773766e27ad90.bin
dc847b243b638c5ce539d4be7ff48007.bin

[L33THAK0R]

Details:

• Source: Clean Steam Files
• Build 10884090
• Original Size: 53 GiB
• Final Size: 16.7 GiB (Incl. installer)
• Ripped Files: Non EN Loc., ~110 MiB "Epic Online Services" installer

Internal Tools used:

• Xtool v0.8.9 (Codecs: reflate, zlib, unreal)
• Srep
• Lolz

External Tools used:

• None

Additional notes:

This title proved to be an inspiration to developing a tool to rip encrypted & compressed assets, specifically within Unreal Engine 4 titles. Currently my python script has only been tested/supports this singular title, but the results are promising. Future improvements obviously lie with processing the Wwise audio. In theory it isn't too difficult and my proof of concept tool supports ripping certain blocks to process and patch in separately, without needing to re-write entire files, but I'd already spent a good amount of time getting this far and had since lost interest in perfecting this title.

As I further develop this tool and provide additional functionality/compatibility, I'll release it publicly, however it's current state is still largely experimental. Erased files were validated by "extracting" them using FModel and examination via hex editor. A full play-through was conducted to confirm no issues persisted with said erased offsets being present.

[Masquerade]

^^
How does this process differ to using a quickbms script with bypassed clog to write encrypted data from the pak straight to the disk so you can rip it with xtool?

[KaktoR]

I also don't know either where is the difference. For this title you can extract wwise with bms script and rip the files with xtool erase command then since the wwise files aren't compressed.

[L33THAK0R]

Originally Posted by Masquerade

^^
How does this process differ to using a quickbms script with bypassed clog to write encrypted data from the pak straight to the disk so you can rip it with xtool?

It doesn't, it was just an excuse to create my own tools to solve a given problem, helps hone skills for future projects/problems.

[wareck]

Darkmoon
Data1 : xtool:c32mb:munity,lz4hc,l12:dd3+LOLZ_NORMAL (*.bundle)
Data2 : xtool:mreflate:dd3+LOLZ_NORMAL (all other files except *.bundle *.dll *.exe)
Data3 : xtool:mzlib:mreflate:dd3+razor_stdio (*.dll *.exe)

final size 4.4gb (last update and with *.bank processed)

[KaktoR]

Terra Invicta

 10:57:13 - Selected ARC/DS method for Data1a-01.bin was: xtool:munity,lz4hc,l12:lz4_193:dd3+4x4:lzma
 10:57:13 - Selected ARC/DS method for Data1b-01.bin was: xtool:dd3+4x4:lzma
-----------------------------------------------------------------------------------------------------------------------------------------
 11:40:12 - Overall input size: 21,19 GB
 11:40:12 - Overall output size: 9,83 GB (Ratio 46.40%)
 11:40:12 - Overall conversion time: 00:42:56