Best Compression Methods for 'Specific' Games. Q&A

[L33THAK0R]

Beyond Good & Evil (GOG Edition)

Files ripped:

• German, Dutch, Spanish, French, Italian localised content can be ripped, amounting to 0.96GB
• Unused pre-rendered trailer footage as well as the "Making Of" Bink1 file in the root directory can be safely removed, amounting to 0.22GB (or at the very least I haven't had any user reports of the contrary being true, I myself have only tested ~30minutes of gameplay so I can't be certain either)

2.02 GB => 520.7 MB

Method:

srep + lolz

Compression time - 30 minutes

The tricky part about this one is calculating the offsets for localised content as well as the unused videos, as this data isn't stored in a logical way, thankfully though, through my analysis and subsequent repacking of titles utilising the "Jade" engine, I can say with a fair degree of confidence that only the primary localisation is intermingled with other data (such as graphical assets, animations and game logic) within a given "*.bfX" archive, with subsequent localisations, in this case the aforementioned five, being stored in a more logical way, allowing for fewer ripped blocks to be generated.

[:( Sad8669]

Method : xt_zlib+srep:m3f:l512+lolz:normal/$criwarevideo=srep:m3f:l512+bcmx
Compression Time : 10.3 Hours
Decompression Time : ~ 55 Minutes (4T)
Initial Size : 36.8 GB
Final Size : 28.9 GB.
Ripped Files : Japanese audio.

$criwarevideo
*.usm

Note : Compressor "bcmx" is just bcm used with MTX.

[KaktoR]

Martha is Dead
v1.04

 14:50:47 - Selected ARC/DS method for Data1a-01.bin was: xtool:mmartha:1+srep_new+4x4:lzma
 14:50:47 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:lzma
-------------------------------------------------------------------------------------------------------------------------------------------
 17:47:31 - Overall input size: 17.13 GB
 17:47:31 - Overall output size: 14.05 GB (Ratio 82.01%)
 17:47:31 - Overall conversion time: 02:53:12

 18:22:38 - Overall decompression time: 00:35:01
 18:22:38 - All files have been successfully extracted!

You can speed up compression by doing it on ssd because of copying temp files.

[KaktoR]

Dune: Spice Wars

res.compressed.pak

xtool+srep+4x4:lzma
Compressed 1 file, 2,958,404,716 => 725,268,911 bytes. Ratio 24.52%
Compression time: cpu 2.50 sec/real 484.56 sec = 1%. Speed 6.11 mB/s
All OK

precomp+srep+4x4:lzma
Compressed 1 file, 2,958,404,716 => 704,372,043 bytes. Ratio 23.81%
Compression time: cpu 3.48 sec/real 564.58 sec = 1%. Speed 5.24 mB/s
All OK

wem files: 145mb -> 140mb with wemtool

[kuyhaa]

anyone, have method for .bundle file from StreamingAssets\aa\StandaloneWindows64 ?

ex : https://bayfiles.com/P90ef8b3y6/tech...ets_all_bundle

thanks

[Masquerade]

Originally Posted by kuyhaa

anyone, have method for .bundle file from StreamingAssets\aa\StandaloneWindows64 ?

You can decompress that bundle with UBT.

Not much point in trying to recompress+patch it because a patch would be almost 17mb.

[Masquerade]

The Riftbreaker [v27746]

Audio .zips:

msc+oggre+srep+lzma2

Everything else:

xzlib+srep+lolz

6.4GB ---> 4.4GB

[Carldric Clement]

Need for Speed Carbon

Original Size: 4.91 GB
Compress Size: 2.41 GB

Method:

*  any jpg files = packjpg
*  rest of files = xprecomp+srep:m3f:a2+exe2+7z:d128m:mt2

XTool Used Plugins = ZLib, Reflate, LZ4HC

[Carldric Clement]

Broforce v1131

Original Size: 457 MB
Compress Size: 289 MB

Method:

* jpg = packjpg
* rest of files = xprecomp+srep:m3f:a2+exe2+lolz:d128:mt4:mtt1:lmdf1
* asset bundle unity = xtool:zlib:unity+srep:m3f:a2+exe2+lolz:d128m:mt4:mtt1:lmdf1

Thanks to L33THAK0R for suggesting reduced size.
Update on 1 May 2022.

[L33THAK0R]

Originally Posted by Carldric Clement

Broforce v1131

Original Size: 457 MB
Compress Size: 337 MB

Method:

* jpg = packjpg
* rest of files = xprecomp+srep:m3f:a2+exe2+lolz:d128:mt4:mtt1:lmdf1

you can reduce this to ~290mb (Incl. installer) if you decompress the Unity assets.

[KaktoR]

Age of Empires II Definitive Edition
Update 61591, 1 Language, all DLC

 17:20:52 - Selected ARC/DS method for Data1a-01.bin was: precomp_mtx+srep_new+4x4:lzma
 17:20:52 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:lzma
-----------------------------------------------------------------------------------------
 21:51:31 - Overall input size: 44.71 GB
 21:51:31 - Overall output size: 24.40 GB (Ratio 54.57%)
 21:51:31 - Data files creation time: 04:09:05

Data1a = *.aoe2scenario|*.aoe2campaign|*.scx|*.png|*.aoe2spgame|*.pdf

Why precomp? Xtool is ~200mb bigger

WemTool on wem files gives some mb, not worth it

[kj911]

Quick noob decompression questions: This varying methods betterly safer from extraction process??

A: xZLib+delta+srep+lolz
B: xZlib+srep+delta+lolz

My test, its compression percentage differnece less than 0.1%. (B its smaller.)

[Masquerade]

Project CARS 3 [v1.0.0.0.0705]

• *.bff
  

  xPCARS3+srep+lolz

• *.bk2
  

  bpk (hash fail on decompression so use bpk exes and hdiff patch them)

• *.bank
  

  msc+srep+lzma2

• Everything else
  

  precomp:d0+srep+lolz

49.4GB ----> 19.1GB

[L33THAK0R]

Are there any reliable methods to discern which LZ4 library version works best with a given sample? I recently got interested in titles using LZ4, since I haven't tackled many, and thought I'd start off small with the 2013 title "Castle of Illusion", which has a singular archive containing all title assets, (decompressed archive inflates to 786 MB -> 1.20 GB, using the appropriate QuickBMSscript ), however I can't seem to figure out which library version to use, the best I've gotten so far is ~850MB. I think if I exhaust all my options I'll just look into re-importing the decompressed assets to a modified archive, since some Russian modders were able to do so, in order to apply a total conversion translation mod to the titular title in question.

[Masquerade]

Originally Posted by L33THAK0R

Are there any reliable methods to discern which LZ4 library version works best with a given sample? I recently got interested in titles using LZ4, since I haven't tackled many, and thought I'd start off small with the 2013 title "Castle of Illusion", which has a singular archive containing all title assets, (decompressed archive inflates to 786 MB -> 1.20 GB, using the appropriate QuickBMSscript )

I think you are overthinking this.

First of all, bms2xtl exists so you half of the job is already done for you. The other half of the job is about 20 minutes and a bit of guessing work

I checked this game out and it uses lz4hc, not lz4. By using lz4 precompressor, you will end up getting 21/5539 streams. By switching the codec to lz4hc alone you will instantly get 4215/5539 [liblz4 v1.7.4].

This alone would seem pretty cool, until you see that the only streams that were inflated were the small ones, so you get very little size increase. In addition to this, you still have over 1000 compressed LZ4 streams still in the file.

By digging a bit deeper into the info XTool is giving me (using verbose output), my next intention was to lock precompression at a certain level of lz4hc. I chose lz4hc level 9. This is because most of the streams previously were compressed at level 9. After making a new database and testing, we get these results:

• 5524/5539 streams [liblz4 v1.7.3, v1.7.4, v1.8.0].
• 5488/5539 streams [liblz4 v1.8.1].
• 5424/5539 streams [liblz4 v1.8.2, v1.8.3, v1.9.3].

This clearly shows that using anything newer than liblz4 v1.8.0 is going to yield extremely poor results. With the oldest 3 libraries, you will only miss 15 streams. In my opinion, this is completely acceptable. At this point, I stop testing and publish my findings in the XTool plugins thread.

We all have a lot to thank Razor12911 for.