Best Compression Methods for 'Specific' Games. Q&A

[KaktoR]

Maybe try with other versions.

Here are all I have collected over time

[hwang4]

Originally Posted by Wanterlude

NINJA GAIDEN 4

@image1.dat



sample: https://pixeldrain.com/u/NJw43Xpd
libs: https://pixeldrain.com/u/5XsLVhsL

Maybe someone has already found a solution?)

upd: https://www.patreon.com/posts/ninja-gaiden-4-141790975
thx razor12911

can't access the file !can you upload it to another site, (patreon.com is unavailable).thx!

[KaktoR]

Originally Posted by hwang4

can't access the file !can you upload it to another site, (patreon.com is unavailable).thx!

Here

[JustFun]

Sanatorium A Mental Asylum Simulator

Selected ARC/DS method for Data1a-01.bin was: 
xtool:c32mb:munity+srep_new:490g+lolz
------------------------------------------------------------------------------------------
Overall input size: 1,32 GB
Overall output size: 679,04 MB (Ratio 50.20%)

[JustFun]

Peaceful Days

Selected ARC/DS method for Data1a-01.bin was: 
srep_new:490g+lolz
------------------------------------------------------------------------------------------
Overall input size: 1,07 GB
Overall output size: 362,59 MB (Ratio 33.02%)

[L33THAK0R]

Finally taking a crack at Red Dead Redemption 2 (version 1.0.1491.50), but I've been pretty disappointed with initial results

~108 GiB (MP assets ripped) -> ~76 GiB, using oo2core_5_win64.dll with the following MD5 hash

f60dd417457e2ec9be4b28242878dbc1

I did some initial digging on the AES encryption of the "*.rpf" archives, but haven't been able to find any community tools, outside of OpenIV (closed-source, no CLI functionality to decrypt multiple archives), SparkIV (only supports GTA IV), or CodeWalker (GTA V mapping utility), that support decryption and encryption of said archives. I've reached out to multiple members of the developer team but have been met with silence. Researching prior RAGE Engine titles, of note is "Grand Theft Auto V", found a toolkit that shows promise, found here, which supports said cryptography. Very loose *maybe* here, but reading through this forum thread has given me some guidance with a starting point for dumping the AES key for RDR2, of course the methodology will likely not be identical, but it's something.

I'm hopeful that since the RPF versions between GTA V (Version 7) & RDR2 (Version 8) are quite close, I might be able to jury-rig something together.

[wareck]

data1.bin was: xtool:mmermaid:core_2.9.5:mzlib:dd3+LOLZ_NORMAL
data2.bin was: xtool:mzlib:dd3+razor_stdio

with data1 = *.ubulk and *.uexp

around 2.25gb final

[Wanterlude]

Star Wars: Squadrons

use zstd 1.2.0
or frostbite plugin by Razor12911, but it doesn't work for me

[KaktoR]

Anno 117

In Beta testing the structure looked exactly like anno 1800, but now it looks like they don't use any compression anymore.
Videos are kb2n but maybe you can rip 4K/FHD files (not tested yet).
Sounds are both pcm and oggvorb so use a mix of wemproc and msc.

  0000000000000800 1025271744 data\ui\video\4k\base\campaign\campaign_movie01_imposter_a.bk2
  000000003d1c7000 507623940  data\ui\video\4k\base\campaign\campaign_movie02_diana_snooping.bk2
  000000005b5e3000 783436492  data\ui\video\4k\base\campaign\campaign_movie03_collateral.bk2
  000000008a108000 750232692  data\ui\video\4k\base\campaign\campaign_movie04_alien.bk2
  00000000b6c82800 145519208  data\ui\video\4k\base\campaign\campaign_movie05_voadas_ambush_part1.bk2
  00000000bf74a000 382275528  data\ui\video\4k\base\campaign\campaign_movie05_voadas_ambush_part2.bk2
  00000000d63db000 394774512  data\ui\video\4k\base\campaign\campaign_movie06_voadas_defeat.bk2
  00000000edc57800 604861032  data\ui\video\4k\base\campaign\campaign_movie07_voada_backs_down.bk2
  0000000111d2f000 1166713092 data\ui\video\4k\base\campaign\campaign_movie08_a_marcia_triumph_a.bk2
  00000001575d9800 1166686676 data\ui\video\4k\base\campaign\campaign_movie08_a_marcus_triumph_a.bk2
  000000019ce7d800 1166695576 data\ui\video\4k\base\campaign\campaign_movie08_b_marcia_triumph_a.bk2
  00000001e2723800 1166675820 data\ui\video\4k\base\campaign\campaign_movie08_b_marcus_triumph_a.bk2
  0000000227fc4800 193718944  data\ui\video\4k\base\campaign\de_lucius_death_stabbed_marcia.bk2
  0000000233883800 172257676  data\ui\video\4k\base\campaign\de_lucius_death_stabbed_marcus.bk2
  000000023dccb000 184690336  data\ui\video\4k\base\campaign\de_lucius_death_stroke_marcia.bk2
  0000000248ced800 203480552  data\ui\video\4k\base\campaign\de_lucius_death_stroke_marcus.bk2
  0000000254efb800 178974240  data\ui\video\4k\base\campaign\eng_lucius_death_stabbed_marcus.bk2
  000000025f9aa800 220126412  data\ui\video\4k\base\campaign\eng_lucius_death_stroke_marcia.bk2
  000000026cb98800 219408708  data\ui\video\4k\base\campaign\eng_lucius_death_stroke_marcus.bk2
  0000000279cd7800 178839400  data\ui\video\4k\base\campaign\eng_luciusdeath_stabbed_marcia.bk2
  0000000284765800 171655440  data\ui\video\4k\base\campaign\fr_lucius_death_stabbed_marcia.bk2
  000000028eb1a000 156260236  data\ui\video\4k\base\campaign\fr_lucius_death_stabbed_marcus.bk2
  000000029801f800 196961188  data\ui\video\4k\base\campaign\fr_lucius_death_stroke_marcia.bk2
  00000002a3bf6000 203861080  data\ui\video\4k\base\campaign\fr_lucius_death_stroke_marcus.bk2
  00000002afe61000 178347444  data\ui\video\4k\base\campaign\zh_lucius_death_stabbed_marcia.bk2
  00000002ba877000 161438700  data\ui\video\4k\base\campaign\zh_lucius_death_stabbed_marcus.bk2
  00000002c426d000 185456316  data\ui\video\4k\base\campaign\zh_lucius_death_stroke_marcia.bk2
  00000002cf34a800 180489480  data\ui\video\4k\base\campaign\zh_lucius_death_stroke_marcus.bk2
  00000002d9f6b800 11190268   data\ui\video\4k\base\content\logo_ubisoft_anno_16-9_h264.bk2
  00000002daa17800 325396896  data\ui\video\4k\base\content\province_intro_albion_a.bk2
  00000002ee06a800 296100680  data\ui\video\4k\base\content\province_intro_latium_a.bk2
  00000002ffacd000 891554720  data\ui\video\4k\base\credits\credits_short.bk2
  0000000334d0e000 41389308   data\ui\video\4k\base\gamesetup\gamemodeselection_campaign.bk2
  0000000337487000 40518924   data\ui\video\4k\base\gamesetup\gamemodeselection_endless.bk2
  0000000339b2b800 40378744   data\ui\video\4k\base\gamesetup\province_intro_albion.bk2
  000000033c1ae000 40919960   data\ui\video\4k\base\gamesetup\province_intro_latium.bk2
  000000033e8b4800 122807792  data\ui\video\4k\base\mainmenu\mainmenutitle.bk2
  0000000345dd3000 15697092   data\ui\video\4k\base\pretitle\mainmenutitle_idle.bk2

  000000034e57a000 269437752  data\ui\video\fhd\base\campaign\campaign_movie01_imposter_a.bk2
  000000035e66f000 133918760  data\ui\video\fhd\base\campaign\campaign_movie02_diana_snooping.bk2
  0000000366626800 204250356  data\ui\video\fhd\base\campaign\campaign_movie03_collateral.bk2
  00000003728f0800 196746916  data\ui\video\fhd\base\campaign\campaign_movie04_alien.bk2
  000000037e492800 38187208   data\ui\video\fhd\base\campaign\campaign_movie05_voadas_ambush_part1.bk2
  00000003808fe000 101092592  data\ui\video\fhd\base\campaign\campaign_movie05_voadas_ambush_part2.bk2
  0000000386967000 103943232  data\ui\video\fhd\base\campaign\campaign_movie06_voadas_defeat.bk2
  000000038cc88000 159414536  data\ui\video\fhd\base\campaign\campaign_movie07_voada_backs_down.bk2
  0000000396490000 302145312  data\ui\video\fhd\base\campaign\campaign_movie08_a_marcia_triumph_a.bk2
  00000003a84b6000 302176148  data\ui\video\fhd\base\campaign\campaign_movie08_a_marcus_triumph_a.bk2
  00000003ba4e3800 302158596  data\ui\video\fhd\base\campaign\campaign_movie08_b_marcia_triumph_a.bk2
  00000003cc50d000 302167816  data\ui\video\fhd\base\campaign\campaign_movie08_b_marcus_triumph_a.bk2
  00000003de538800 52451352   data\ui\video\fhd\base\campaign\de_lucius_death_stabbed_marcia.bk2
  00000003e173e800 46890072   data\ui\video\fhd\base\campaign\de_lucius_death_stabbed_marcus.bk2
  00000003e43f6800 50786372   data\ui\video\fhd\base\campaign\de_lucius_death_stroke_marcia.bk2
  00000003e7466000 55142608   data\ui\video\fhd\base\campaign\de_lucius_death_stroke_marcus.bk2
  00000003ea8fd000 48100276   data\ui\video\fhd\base\campaign\eng_lucius_death_stabbed_marcus.bk2
  00000003ed6dc800 59782580   data\ui\video\fhd\base\campaign\eng_lucius_death_stroke_marcia.bk2
  00000003f0fe0000 59212996   data\ui\video\fhd\base\campaign\eng_lucius_death_stroke_marcus.bk2
  00000003f4858800 48082048   data\ui\video\fhd\base\campaign\eng_luciusdeath_stabbed_marcia.bk2
  00000003f7633800 46349796   data\ui\video\fhd\base\campaign\fr_lucius_death_stabbed_marcia.bk2
  00000003fa267800 42202820   data\ui\video\fhd\base\campaign\fr_lucius_death_stabbed_marcus.bk2
  00000003fcaa7000 53560632   data\ui\video\fhd\base\campaign\fr_lucius_death_stroke_marcia.bk2
  00000003ffdbb800 55115352   data\ui\video\fhd\base\campaign\fr_lucius_death_stroke_marcus.bk2
  000000040324b800 48056056   data\ui\video\fhd\base\campaign\zh_lucius_death_stabbed_marcia.bk2
  0000000406020000 43580724   data\ui\video\fhd\base\campaign\zh_lucius_death_stabbed_marcus.bk2
  00000004089b0000 50779544   data\ui\video\fhd\base\campaign\zh_lucius_death_stroke_marcia.bk2
  000000040ba1d800 49277904   data\ui\video\fhd\base\campaign\zh_lucius_death_stroke_marcus.bk2
  000000040e91c800 9248084    data\ui\video\fhd\base\content\logo_ubisoft_anno_16-9_h264.bk2
  000000040f1ee800 84751980   data\ui\video\fhd\base\content\province_intro_albion_a.bk2
  00000004142c2000 76607036   data\ui\video\fhd\base\content\province_intro_latium_a.bk2
  0000000418bd1000 32876304   data\ui\video\fhd\base\gamesetup\gamemodeselection_campaign.bk2
  000000041ab2b800 33100912   data\ui\video\fhd\base\gamesetup\gamemodeselection_endless.bk2
  000000041cabd000 40756988   data\ui\video\fhd\base\gamesetup\province_intro_albion.bk2
  000000041f19b800 33127392   data\ui\video\fhd\base\gamesetup\province_intro_latium.bk2
  0000000421133800 98413104   data\ui\video\fhd\base\mainmenu\mainmenutitle.bk2
  0000000426f0e800 13218008   data\ui\video\fhd\base\pretitle\mainmenutitle_idle.bk2

PS: You can still use anno7.bms script to extract the entire game.

[KaktoR]

Originally Posted by Wanterlude

Star Wars: Squadrons

use zstd 1.2.0
or frostbite plugin by Razor12911, but it doesn't work for me

It works fine, you have to use the correct config.

-mzstd
Streams: 8447 / 8447
Time: 00:00:34 (CPU 00:03:48)
Duplicates: 104 (0,00 MB) [0,00 MB >> 6,50 MB]
Srep decompression memory: 1,00 MB

Size: 411 MB >> 586 MB >> 579 MB >> 568 MB >> 344 MB

---

-mfrostbite3,zstd
Streams: 8338 / 8338
Time: 00:00:33 (CPU 00:03:46)
Duplicates: 0 (0,00 MB) [0,00 MB >> 0,00 MB]
Srep decompression memory: 2,00 MB

Size: 411 MB >> 579 MB >> 579 MB >> 568 MB >> 344 MB

The plugin skips streams which are less or equal 65536 bytes which leads to a bit better performance (see cpu time) with no impact in ratio afterwards.

[L33THAK0R]

Source: Clean GOG Files

Original Size: 41.5 GiB
Final Size: 10.032 GiB (Incl. installer)
Ripped Files: French/German loc.
Internal Tools used:

Xtool v0.8.9 (Codecs: reflate, zlib)
Srep
Lolz
Bpk

External Tools used:
SFK
MParallel
HDiffPatch
OGGRE (Wwise)

Additional notes:

PCK archives can be processed for an approx ~100 MiB gain, however a good portion of the WEMs aren't worth processing, I made a basic python script to diff them, I'm sure anyone wanting to see similar gains is able to figure out their own solution.

Old archived, PCK/BNK BMS scripts won't work with these archives, I uploaded ones that do work, these also make ripping the localised content very easy.

There were a few kb worth of zlib/reflate streams, I was processing this title and the sequel, "A Plague Tale: Requiem" (which does have significant zlib/deflate streams present) in parallel and didn't realise I used the same config for this title, you can probably save a megabyte or two in the installer size if your forgo xtool. These results were obtained using xtool's "erase" functionality, so Razor12911's hard work can still be credited with it's success!

[L33THAK0R]

Has anyone dealt with CELT audio before? Was returning to see if I could improve upon my previously published results for Mad Max, as I'm in the process of redoing my previous attempts of "Avalanche Engine" titles. In said results, I noted a strange 800MiB file with an FSB5 header, that I wasn't able to parse, turns out it's just a very large sound bank, and I believe the tools I used previously weren't able to parse files of that size. Anywho I've come to discover the audio in this sound bank is encoded as CELT audio, which I've never encountered thus far, yet alone processed. I tried running it through MSC & SAC as the tools I used had added WAVEfmt headers, but no luck.

It's a codec that was developed by the Xiph.Org Foundation, but I'm unable to find any tools published for lossless compression of this particular codec, and haven't had any luck (not that you'd expect to) with any tools developed for other Xiph.Org Foundation codecs.

Attached are some samples extracted both with/without a header, if anyone is interested.

[L33THAK0R]

Source: Clean Steam Files

Build 20516955
Original Size: 3.0 GiB
Final Size: 1.604 GiB (Incl. installer)
Ripped Files: None

Internal Tools used:

Xtool v0.8.9 (Codecs: reflate, zlib, unity (lz4 v1.8.3)
Srep
Lolz

External Tools used:
SFK

MParallel
HDiffPatch
OGGRE (Wwise)

Additional notes: All raw WEMs are valid, BNKs only about 1/4 are valid.

[KaktoR]

Kunitsu-Gami Path of the Goddess

 22:15:25 - Selected ARC/DS method for Data1a-01.bin was: xtool:dd3:l10,d128mb
 22:15:25 - Selected ARC/DS method for Data1b-01.bin was: xtool:mzstd,l11:zstd_145+xtool:mzlib:mreflate:dd3:l10,d128mb
 22:15:25 - Selected ARC/DS method for Data1c-01.bin was: xtool:dd3:l10,d128mb
-----------------------------------------------------------------------------------------------------------------------------------------
 23:55:31 - Overall input size: 28,15 GB
 23:55:31 - Overall output size: 14,64 GB (Ratio 51.99%)
 23:55:31 - Overall conversion time: 01:40:04

[wareck]

hello
There is a lot of .bank file can be processed (gain around 200mb) ...

Without processing theses files , this works:

15:46:32 - Selected ARC method for Data1a-01.bin was: poo2reck:core_2.9.10:lib_2+poo2recm:core_2.9.10:li b_2+xtool:dd3+lolz:d59:mtt1:mt16:mc1023
15:46:32 - Selected ARC method for Data1b-01.bin was: xtool:dd3+bcm
15:46:32 - Selected ARC method for Data1c-01.bin was: xtool:c32mb:d1:mreflate+xtool:dd3+razor_mtx
---------------------------------------------------------------------------------------------------------------------------------------------------------
18:57:28 - Overall input size: 9,83 GB
18:57:28 - Overall output size: 5,98 GB (Ratio 60.82%)
18:57:28 - Overall conversion time: 03:09:29
18:57:28 - Overall conversion media: 1 custom media