Best Compression Methods for 'Specific' Games. Q&A

[L33THAK0R]

Has anyone had any luck with any LZ4 library for "State of Decay 2"? Threads around the net indicate it uses some sort of "custom" LZ4 solution, however QuickBMS is able to parse it with it's built in LZ4 comtype. Currently exhausted a good collection of ~20 libraries through trials and I'm yet to pick up or detect a single stream.

Attached is a sample ".pak" archive should this compression quandary tickle anyone else's fancy.

[Masquerade]

Shame Legacy

*.pak:

ue4d:key=07E35C91100A9B400B1A87B15337BEEAC3545EEE30786F21CB2EE7283AF51743+xtool_zlib_dd5+lolz

Everything else:

razor-stdio

9.9GB ---> 6.9GB

Used CLS-UE4D library by Shegorat.
Thanks to Razor12911 and ProFrager for XTool and Lolz.
Thanks to Bulat Ziganshin for FreeArc and srep.

[KaktoR]

Anyone can help me with srep options to use? I'm getting stuck here a bit.

ERROR! Can't allocate memory: 9705 mb required

Edit: -l# option seems to be the trick.

[KaktoR]

Hitman 3 test using rpkg tool (because no plugin is available atm)

chunk0.rpkg (13.5GB)

 srep+lolz_fast
Overall input size: 14.03 GB
Overall output size: 12.10 GB (Ratio 86.29%)
Overall conversion time: 00:53:55
 
 
 srep+lolz_fast with rpkg tool
Overall input size: 15.50 GB
Overall output size: 11.41 GB (Ratio 73.61%)
Overall conversion time: 02:01:43

Input is bigger because chunk0.rpkg was extracted (lz4 variant is used). However output is smaller. But that time to process is insane and I recommend not to do this at all.

The time above was achived on a m2 ssd for a single test. Here again it is not recommended to do this kind of stuff on a ssd, even it is much faster.

Extracting all rpkg's on a HDD took barely 4 hours.
Next step is to make TAR files out of extracted files (this will take another bunch of hours), because somehow freearc seems to have problems with nearly 2 million extracted files.

*.WWEM
*.WWES

Rename them to *.WEM so you can use WemTool on them. After decoding rename them back to *.WWEM and *.WWES.
You can save approximately 10% of original size.


*.WWEV
They have "RIFF8-��WAVEfmt" but cannot do anything with them.
More info: https://fileforums.com/showpost.php?p=501212&postcount=3495

This is just a info test out of curiosity and I wanted to share some infos here.

[dixen]

Originally Posted by KaktoR

Witcher 3 v4.0 Next Gen Update

Still the same algo from before
https://fileforums.com/showthread.ph...her#post498295

-mxtool:witcher3+xtool:zlib+xtool:lz4hc,l9

Compressed 1 file, 286,247,872 => 1,249,481,045 bytes. Ratio 436.50%
Compression time: cpu 0.41 sec/real 51.41 sec = 1%. Speed 5.57 mB/s
All OK

Extracted 1 file, 1,249,481,045 => 286,247,872 bytes. Ratio 436.50%
Extraction time: cpu 0.59 sec/real 37.01 sec = 2%. Speed 7.73 mB/s
All OK

Which version xtool and liblz4.dll needed?
With W3.xtl - decompression error for me

[KaktoR]

XTool 0.6.9 + xdelta3_dll.dll + lz4_193 + doboz.dll

bms2xtl config

lz4=lz4hc:l9

[dixen]

Originally Posted by KaktoR

XTool 0.6.9 + xdelta3_dll.dll + lz4_193 + doboz.dll

bms2xtl config

lz4=lz4hc:l9

It works but..VERY slow on lz4hc streams.

[Masquerade]

Repella Fella

You can use ffmpeg to extract the vorbis audio tracks from the webm videos then make hdiffz patches to convert the oggs back into videos (note that the hdiff patches will be huge since they make up the entire video track).

Only issue is that oggre is buggy af with these oggs, so it may just be easier to use:

srep+lzma2 on the webm

precomps+srep+lolz on everything else.

Precomps = precomp stdin/stdio fork by Nicolas Comerci.

8GB > 7.2GB

I spent all day trying to get oggre working, but it will crash without much of a pattern to go off.

[KaktoR]

Originally Posted by dixen

It works but..VERY slow on lz4hc streams.

Hm, no, not for me.

Did it today with latest version.

 11:16:00 - Overall input size: 61.98 GB
 11:16:00 - Overall output size: 44.43 GB (Ratio 71.68%)
 11:16:00 - Overall conversion time: 01:58:29

[dixen]

Originally Posted by KaktoR

Hm, no, not for me.

Did it today with latest version.

 11:16:00 - Overall input size: 61.98 GB
 11:16:00 - Overall output size: 44.43 GB (Ratio 71.68%)
 11:16:00 - Overall conversion time: 01:58:29

Can you share your witcher3.xtl?

[Masquerade]

Orcs Must Die! 3 [v1.2.1.0]

*.pak:

xtool_OMD3_dd5+lolz

Everything else:

xtool_zlib+razor-stdio

12.6GB ---> 6.8GB

Requires OMD3 database and lz4 library linked here.

[Dragonis40]

Good evening, is there a way to create a .tmp file from a decompressed (or unpacked) file or batch of files in order to use hdiffz.exe and so to create a .diff file?
A lot of videogames have several files that can be unpacked using specific tools, but no way to repack them. Maybe hdiffz.exe is a solution, but it needs a .tmp file. How to create it? Thanks in advance for your help!

[Masquerade]

HDiffz does not need a .tmp file, read the usage instructions.

Old file and new file makes differential patch.

[Masquerade]

Hi,
Does anyone have any samples of LZMA compressed Unity Asset Bundles that they could share with me?
Thanks!

[KaktoR]

Diablo IV
v1.0.2, incl. HD files

 10:50:35 - Selected ARC/DS method for Data1a-01.bin was: xtool:dd3+lzma2:d96m:fb273:lc4:mc1000000
 10:50:35 - Selected ARC/DS method for Data1b-01.bin was: xtool:mzlib+xtool:mpreflate:dd3+lzma2:d96m:fb273:lc4:mc1000000
-----------------------------------------------------------------------------------------
 13:17:45 - Overall input size: 77.06 GB
 13:17:45 - Overall output size: 62.72 GB (Ratio 81.38%)
 13:17:45 - Overall conversion time: 02:27:08