Best Compression Methods for 'Specific' Games. Q&A

[KaktoR]

Originally Posted by Masquerade

ENGLISH ONLY! All unneeded language files + deluxe movies ripped.

Start size: 51.3GB

Compression method:

oo2recm+xtool_zlib+srep+lolz

Do not use XTool chunk size of greater than 128mb otherwise it will get stuck during compression. (tested on my 64GB, 16 thread PC). XTool v0.12 used. Thanks to Razor12911 for the tools.

Final Size: 37.9GB

 13:03:18 - Selected ARC/DS method for Data1a-01.bin was: oo2recm8+xtool:mzlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 13:03:18 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
----------------------------------------------------------------------------------------------------------------------------------------
 19:01:31 - Overall input size: 51.51 GB
 19:01:31 - Overall output size: 44.30 GB (Ratio 86.01%)
 19:01:31 - Data files creation time: 05:31:39

Just 200mb for lolz, bad.

Even worse that with xtool applied the size is bigger then my last repack with just oo2recm https://fileforums.com/showpost.php?...postcount=2136

[KaktoR]

They are Billions
v1.1.4

 19:48:52 - Selected ARC/SPLIT method for Data1a-01.bin was: srep_new+lolz:d64:mtt1:mt12:mc1023/$storepack=FA-MIN/$libpack=srep_new+razor_mtx/$oggretheyarebillions=oggre/$deflatetheyarebillions=xt_zlib+srep_new+lolz:d64:mtt1:mt12:mc1023
----------------------------------------------------------------------------------------------------------------------------------------
 20:30:49 - Overall input size: 5.16 GB
 20:30:49 - Overall output size: 1.84 GB (Ratio 35.70%)
 20:30:49 - Overall conversion time: 00:41:53

[KaktoR]

Battlefield 2042 test

srep+4x4:lzma
Overall input size: 246.54 MB
Overall output size: 55.90 MB (Ratio 22.67%)

oo2reck+srep+4x4:lzma (oo2reck takes very long)
Overall input size: 246.54 MB
Overall output size: 54.91 MB (Ratio 22.27%)

xtool:kraken+srep+4x4:lzma
Overall input size: 246.54 MB
Overall output size: 54.94 MB (Ratio 22.28%)

[KaktoR]

Skyrim Anniversary Edition test

Compressed 15 files, 9,955,981,643 => 16,269,399,549 bytes. Ratio 163.41%
Compression time: cpu 11.56 sec/real 320.70 sec = 4%. Speed 31.05 mB/s
All OK

Extracted 15 files, 16,269,399,549 => 9,955,981,643 bytes. Ratio 163.41%
Extraction time: cpu 17.92 sec/real 271.19 sec = 7%. Speed 36.71 mB/s
All OK

Data\Skyrim - Animations.bsa
Data\Skyrim - Interface.bsa
Data\Skyrim - Meshes0.bsa
Data\Skyrim - Meshes1.bsa
Data\Skyrim - Misc.bsa
Data\Skyrim - Shaders.bsa
Data\Skyrim - Textures0.bsa
Data\Skyrim - Textures1.bsa
Data\Skyrim - Textures2.bsa
Data\Skyrim - Textures3.bsa
Data\Skyrim - Textures4.bsa
Data\Skyrim - Textures5.bsa
Data\Skyrim - Textures6.bsa
Data\Skyrim - Textures7.bsa
Data\Skyrim - Textures8.bsa

Working bsa plugin https://fileforums.com/showpost.php?...&postcount=223
Use lz4 v1.8.3 library

PS: t100p stuck for me, so use a lower thread usage for xtool

[kj911]

Microsoft Train Simulator *.ACE files:

older:

xZLib+srep:m3f:a2+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1 :mtt1:mt4:d128m:tt32:fba4096:mc1023

newest:

xZLib+srep:m3f:a2+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1 :mtt1:mt4:d128m:tt1:oh14:gm01:lm4:fba4096:mc1023

Green its optional switches, probably testing any game. Defaults, its 00, 11 or 21 worse results than 00.
Red the magic switch! Nice compression gain! And overall usage all physical RAM! (8GB and 87% usage vs. 97-99%)
oh14 its minimally better compression. Defaults: oh12
tt1 its very fast mode, max usage: tt16 or tt32!! Higher values not better and extremely slow compression.

Difference? Old version 2.93GB! New, with tt1 (compress again with tt16 and d160m switches in last time)* 2.80GB!! ~5% compression rate gain.

UPDATE: Aaarrgh! The test package don't decompresseable! CLS-ini error or any tips?

*tt32 its probably very best results. Calculated compression time avg. ~2days! Overkill.

[Cesar82]

Could someone familiar with LOLZ parameters help me with some questions?

How is actual memory usage calculated using for multi thread used An X value for dictionary (-d)?

Considering that the supported values for dictionary are from 0 to 2047 "-d[0..2047]): 64m". If you enter a value above 2047 (such as 2050) will the default value be set to the dictionary (64m) or will the maximum value (2047m) be used?


I need this information because I created presets for LOLZ in DiskSpan GUI.
The memory and thread information will be obtained when starting the DiskSpan GUI and then starting the compression will be calculated based on the threads and RAM of the system.
It will also be informed the number of system theads replaced <thread> by the number of virtual cores (threads) of the current CPU during the conversion.
So one more question arises...
In the lolz documentation it states that the supported values are 1 to 16 thread: "mt threads(-mt[1..16]): 1"
If using a Ryzen processor 9, 5900x is 24 threads, so if you use -mt24, lolz will set it to 1 (default) or to 16 (Maximum supported).

Below is the example where it will be in LOLZ presets in DiskSpan GUI.

LOLZ_HIGH = lolz:dtb1:d<expr:0.65*memory/(thread+1)/1048576>m:mtt1:mt<thread>:mc1023

Expression example: <expr:0.65*memory/(thread+1)/1048576>

0.65    = percentage of calcule (65%)
memory  = full system memory (RAM)
thread  = number of logical cores
core    = number of Physical cores
1048576 = convert result in bytes to mb

If someone can help me with these questions, I would appreciate it!

[kj911]

Its calculated result correct?? See the linked picture from my friend laptop specifications. --> https://i.kek.sh/zUjUwL6MVZE.jpg

Example: Intel Core I3-1005G1 CPU, Win10 x64 with 8GB memory. (Total cores: 2, Total Threads: 4)

Available physical RAM 8GB --> 8589934592 byte

PreCalculate: 0.65*8589934592 = 5583457484,8
Cores --> 4+1 = 5
Final calculate: 5583457484,8/5/1048576 = 1064.96 MB?? Try it added extra /8 values? 132MB values its really.

Use -mtt1 and -mt4 mode and -d128m dictionary size, eats 5.2-5.5GB's memory from during compression.

[Cesar82]

Originally Posted by kj911

Use -mtt1 and -mt4 mode and -d128m dictionary size, eats 5.2-5.5GB's memory from during compression.

For your system it would result as if using from the command line:

LOLZ_HIGH = lolz:dtb1:d1065m:mtt1:mt4:mc1023

So lolz would use: 4 * 1065 = 4260 MB of 8192 MB.

I believe the memory usage of lolz is calculated as: theads*dictionary

Example: If using a Ryzen 5900x (24 theads) and 32 GB of RAM we would have:

LOLZ_HIGH = lolz:dtb1:d852m:mtt1:mt24:mc1023

So lolz would use: 24 * 852 = 20448 MB of 32768 MB.

It may be recommended to do the calculation using available memory rather than total system memory.

Maybe it's not just simply multiplying the dictionary by the number of threads, that's my question.
Can not understand. See the attached video.

[its_gnar_again]

Has anyone here had any luck getting Final Fantasy X/X-2 HD to decompress without any issues? The method on here is basically xtool(xzlib)+srep+lolz, but that ends up with a corrupted archive error during decompression not even half-way. I have tried the following:

• Tried different configs for LOLZ
• Tried different configs for SREP
• Tried outright replacing LOLZ with LZMA
• Tried various versions of XTool and ZTool, and different configs for them as well.

I really don't know what the problem is with this game, and trying to compress the 2 games separately still results in a corrupted error too during decompression.

[KaktoR]

Hellblade: Senua's Sacrifice Enhanced
Video documentary ripped

 17:51:27 - Selected ARC/DS method for Data1a-01.bin was: srep_new+LZMA-MT
 17:51:27 - Selected ARC/DS method for Data1b-01.bin was: bpk
----------------------------------------------------------------------------------------------------------------------------------------
 18:38:43 - Overall input size: 17.25 GB
 18:38:43 - Overall output size: 8.25 GB (Ratio 47.84%)
 18:38:43 - Data files creation time: 00:38:17

[KaktoR]

Originally Posted by its_gnar_again

Has anyone here had any luck getting Final Fantasy X/X-2 HD to decompress without any issues? The method on here is basically xtool(xzlib)+srep+lolz, but that ends up with a corrupted archive error during decompression not even half-way. I have tried the following:

• Tried different configs for LOLZ
• Tried different configs for SREP
• Tried outright replacing LOLZ with LZMA
• Tried various versions of XTool and ZTool, and different configs for them as well.

I really don't know what the problem is with this game, and trying to compress the 2 games separately still results in a corrupted error too during decompression.

xtool:reflate works just fine for me.

[External compressor:xtool]
header    = 0
default   = -c64mb -t75p
packcmd   = "XTool.exe" precomp { -option} - - <stdin> <stdout>
unpackcmd = "XTool.exe" decode -t100p - - <stdin> <stdout>

Did you test only xtool to see if it works?

[its_gnar_again]

Originally Posted by KaktoR

xtool:reflate works just fine for me.

[External compressor:xtool]
header    = 0
default   = -c64mb -t75p
packcmd   = "XTool.exe" precomp { -option} - - <stdin> <stdout>
unpackcmd = "XTool.exe" decode -t100p - - <stdin> <stdout>

Did you test only xtool to see if it works?

My xtool setup is pretty much the same from when I last tested it.

[External compressor:xtool]
header = 0
packcmd = XT\xtool.exe precomp -mzlib -c64mb -t75p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

I haven't tested JUST xtool only to see if that works, but I will get back to you here from when I do. I don't understand what the problem is on my part really.

[Grumpy]

@its_gnar_again

forgot the pass & email I used on my old account lmao

Surely you remember one of them, you've had 4 or 5 accounts since 2020

[its_gnar_again]

Originally Posted by its_gnar_again

I haven't tested JUST xtool only to see if that works, but I will get back to you.

UPDATE: I tried specifically using reflate on xtool+srep+lolz, and still get a decompression error. However, I did try using xtool only and it decompressed just fine...what could honestly be the issue? I would like to get this figured out.

[KaktoR]

Maybe check your CLS.ini settings for lolz and srep.

And make sure you're using the correct cls files for lolz and srep