Preprocessing / Optimization / Re compression For Better Compression of Games

[Gupta]

[DEFLATE/ZLIB]
1.Precomp: Precomp is a command line pre-compressor. You can use it to achieve better compression on some file-/streamtypes (works on files and streams that are compressed with Zlib or the Deflate compression method, bZip2, GIF, JPG, and MP3). Precomp tries to decompress the streams, and if they can be decompressed and "re-"compressed so that they are bit-to-bit-identical with the original stream, the decompressed stream can be used instead of the compressed one.
Links:
:Gitter Chat
:Github Repository
:Github Releases
:Universal CLS-Precomp By RAZOR12911
:CLS-PRECOMPMT By RAZOR12911
Downloads:
:PrecompInside0.31.7z

2.PZLIB V3: A fast and really very fastest re-compressor for Deflate and Zlib, developed by RAZOR12911 but has some bugs when restoring several files via stdin stdout. Hence advised to be used separately on files
Links:
:Official Thread

3.PZLIB V1: The predecessor of PZLIB V3, fast and good for games containing zlib streams compressed using default and maximum level
which have headers example GTA IV, Dragon Age etc.
Links:
:Official Thread

4.Reflate: universal deflate re-compressor
Links
:CLS-Reflate by RAZOR12911
Download:
:Reflate V1l1 (GCC Compiled)

[Stream/Specific Compressors]
1.Media Stream Compressor (MSC): bmp Packer/preprocessor for multimedia data. It currently supports the following data types: dds (raw, dxtBMP, Wav (PCM), mp3. Because This preprocessor is sharpened by working with freearc, the data processing is carried out with a single input file. The output is just one file of its own format.
Suggestion: Should be used be before rep or basically before any "compressor"
-mmsc+srep+lzma or -mprecomp+msc+srep+lzma -> correct
-mprecomp+srep+msc or -msrep+msc -> incorrect, will ruin compression
Download:
:msc0.064.7z (Contains CLS)

2.MPZ: Best Mp3 compressor Developed by Eugene Shelwein. It usually reduces file size by 15 % but on some files, it can go par 60%
Download:
:mpzapi_v1a.rar

3.PackJPG: A compression program for further compressing JPEG image files Github:

4.PackARC: packARC is an archiver program specially designed for lossless further compression of media files in the JPEG, MP3, BMP and PNM formats. Github:

5.PackMP3: A compression program for further compressing MP3 audio files Github

6.PackPNM: A compression program for further compressing PNM, BMP and Radiance HDR files Github

7.unpackJPG: Unpack only, PackJPG Github

8.Lepton: Lepton is a tool and file format for losslessly compressing JPEGs by an average of 22%.
Links
:Github repository
:Github releases
:Official Blog Post

9.Png optimize suite(Win32) (Ver 2012-06-15): set of console compressor for Pngs
Download
:pngopt-120615.7z
:Blog Post

10.OptimFrog:OptimFROG is a lossless audio compression program. Its main goal is to reduce at maximum the size of audio files while permitting bit identical restoration for all input. It is similar with the ZIP compression, but it is highly specialized to compress audio data.
Downloads:
OptimFROG_Win_x86_5100.zip
OptimFROG_Win_x64_5100.zip

11.TAK: Tom's lossless Audio Kompressor (TAK) is a lossless audio compressor which promises compression performance similar to Monkey's Audio “High” and decompression speed similar to FLAC.
Downloads:
:TAK_2.3.0.zip
Links
:HomePage

[Prerocessors]

Delta: This preprocesses data improving their further compression. It detects tables of binary records and
1) substracts successive values in columns,
2) reorder columns trying to maximize results of further compression.
should be used on raw data i.e. not processed with srep, any deflate compressor, or something other

Usage:

fazip delta infile outfile

Rep : A LZ77 compressor detects repetition in 1GB Block
Usage:

fazip rep:<dictionary size>mb infile outfile

Super-rep : Successor of rep, supports dictionaries larger than RAM available.Compression speed is 100 mb/s and decompression runs at 250 mb/s on i3-2100.
:Description of The Algo
:srep393a.7z
CLS:SrepInside0.33.7z

[Loossy Compressors]

OggEnc: A very good lossy Compressor for Waves
Links
Official Site
Downloads:
oggenc2.88-1.3.5-generic.zip
oggenc2.88-1.3.5-x64.zip

Binks Lossy Compressor: Binks Reencoding script written by PANKER1992.
Link:
Official Discussion
Downloads
Bink_Enc.7z

[Prince4]

Some knowledge collaborative like Razor's!!

Keep it up!

[Gupta]

ppx2: ppx2 allows parallel processing of arguments
Changes by me:

setting priorities of the processes
start process with new console window
verbose mode
and others i don't remembering

Here it is
ppx2.7z

For help:
ppx2 --help

The source can be found here:
https://github.com/guptaprince/ppx2

[EzzEldin16]

i have a question
i compressed a game with zlib streams found alot of them
i compressed with precompressor it packed well but unpacking didn't
is the problem with the .iss script or what i'm using wpi script by razor

[Gupta]

Are u talking about pzlib?
Then it is the pre-compressor itself.
there is no way to fix that

[ZakirAhmad]

Xdelta

How xdelta can be used to decompress Compressed files... I have seen it in crysis 3 repack of Fitgirl... The files r md5 after decompression are orginal...

[Joe Forster/STA]

Xdelta is used for patches. (You know, game X version 1.0 to game X version 1.1: version 1.0 files are the input, version 1.1 are the output.) It does compress the patch file, though. See .RTPatch for a very similar, but commercial, software.

[EzzEldin16]

@joe foster/STA can you explain how it is used precisely ??

[Joe Forster/STA]

Imagine that you're the developer of game X. You've already released version 1.0 (CD/DVD, Steam etc.) but found some bugs since (wow!). You fixed (some of) them, calling it version 1.1, but you can't re-release the whole game; you rather collect the changes into a patch. This patch can be a package of the full files that have been changed between version 1.0 and 1.1 but that can be really big if it contains many files and/or some of the files are big.

A smarter idea is collecting the actual differences between the files. (Obviously, if those files exist in both versions.) Say, you have a big data file that has an internal structure and it contains many little files. (E.g. as simple as a ZIP archive.) If you add one internal file then only that part of the big data file changes, the rest stays the same. If you fix some code and recompile the program file (executable) then, again, most of the file stays the same, only some parts of it (added code, added data, modified jump offsets, modified data pointers/references) will change.

Xdelta and .RTPatch are programs that can do a binary comparison between a bunch of "old" files (those of version 1.0) and "new" files (version 1.1) and create a patch. If your customer downloads the patch and has the "old" files, xdelta can use ("apply") the patch to convert the "old" files into the "new" ones.

It doesn't matter if the "old" and "new" files are big, if the actual differences are small then the patch will be small, too. You can create a single patch file out of the differences of multiple files, including creating a new file (that didn't exist in version 1.0 yet), deleting a file (that doesn't exist in version 1.1 anymore) or modifying a file. I'm not sure about xdelta but with .RTPatch you can also create multi-version patches (version 1.0 or 1.1 or 1.2 -> 1.3) that can modify many versions of "old" files into the "new" ones.

I've seen xdelta used in the game called "The Thing" (from 2002 or something). Most developers use .RTPatch because it probably has more features but it's extremely expensive (thousands of dollars) and closed source, while xdelta is free and open source.

[ImmortalArc]

Originally Posted by Joe Forster/STA

.RTPatch it's extremely expensive (thousands of dollars) and closed source, while xdelta is free and open source.

Whoaaa

[Gupta]

UPDATED

- New Lossy Section
- Add some Other Compressor

Guys, Please please help me further improve this thread
Thanks.

[kj911]

Introducing QOI — the Quite OK Image format. It losslessly compresses RGB and RGBA images to a similar size of PNG, while offering a 20x-50x speedup in compression and 3x-4x speedup in decompression. All single-threaded, no SIMD. It's also stupidly simple.

https://phoboslab.org/log/2021/11/qo...ge-compression

Nice preprocessors from visually losslessly recompress PNG files smaller than ~30 to 50% percentage to original ones. (Hint: compress encoded QOI files use CM/LZ* or any algo.)

Compressed PNG files to QOI format, its bigger or smaller than original PNG file and fairly compressibble than orig. PNG files. Decoded PNG files its smaller/bigger than original PNG files. Not bitperfect lossless compression. Visually lossless only.

Example:

Original test.png file: 213 745 bytes
QOI file: 398 930 byte
Decompressed PNG file to QOI file: 336 428 byte
Compressed QOI file with RAR: 96 430 byte (internal size in archive)

Example2:

pg-coral.png 265K
pg-coral.png.balz 266K // balz cannot improve png!
pg-coral.qoi 371K // seems bad? but not if further compressed
pg-coral.qoi.zip 250K // better than PNG!
pg-coral.qoi.balz 222KB // even better! total-code size (Balz + qoi + strip comments&headers) is maybe 500 lines of code.

I think if qoi used my suggestions (tiling+delta), we can get the file much smaller, maybe 200K.

Further tests with XZ:

pg-coral.qoi.xz 182KB

Again, if qoi used tile+delta, maybe we can get the file down to 150KB? (using xz)

Compiled Win XP x86 compatible EXE attached.

[Dev_0016]

Hey I need your help how fitgirls and other repacker like ( elamigos, rg ...)..
how they were compress thier files ( highly compress ) which type of tools they were use
to compress the files and make repack of them..??? will you tell me please ..

[kj911]

ALERT! S...ing MS!!!

Microsoft last month received a US patent covering modifications to a data-encoding technique called rANS, one of several variants in the Asymmetric Numeral System (ANS) family that support data compression schemes used by leading technology companies and open source projects....

https://www.theregister.com/2022/02/...ft_ans_patent/

https://patents.google.com/patent/US11234023B2/en

[kj911]

Qouting from krinkels posts, these last Optimfrog EXE file modification says its to Edison007:

4. Last but probably the most interesting. The seek parameter is the number of samples in the compressed block. The original values ​​are as follows:

default options number of samples: 
fast = 44100 * 10 
normal = 44100 * 20 
slow = 44100 * 40 
min = 44100 * 120 

mod1 
fast = 88200 * 40 
normal = 88200 * 80 
slow = 88200 * 120 
min = 88200 * 160 

mod2 
fast = 352800 * 60 
normal = 352800 * 80 
slow = 352800 * 120 
min = 352800 * 160

Actually, you can patch ofr.exe by changing them in any direction (but with an increase, the consumed RAM also grows), this can affect both for the better and for the worse. Whoever knows how to use IDA can find and change where it is, who can't wait for either a patcher or patched executables from me, I'll think about how simpler it is.

These any values increase --> its have better compression ratio?? How to its modding the EXE file?? I would replace the 2nd number with 320 instead of 160.

Also: how to methods from upgrading CLS-MSC.dll files with newest TAK/OFR/BMF libraries??