Current compressing techniques guidance

[joekkerr]

Hi, im JoeKkerr

I was actively participating in making repacks from 2010-2014.

i'm not completely new to repacking, but i remember my lack of knowledge in my end days where i wasnt able to compress or keep up with others size.

what i did was what others were doing take files > Freearc > batch file unpacking.
or some files needed precomp or a combination of precomp + srep but problem started arising when some files just werent getting reduced, even after using precomp they werent 'precompressing' or as i'd call it deflating. while groups like blackbox and kaos were able to do it. (ik it applies to only some files that can actually be deflated)

i even tried studying thier repacks and their exe's to replicate result and hence i gave up knowing more people were doing better job than me and my internet was 1MbPS with 100kb download and 25 upload.

I'm thinking of getting back into data compression. whats the current method of compressing data ? i know different games , different methods but most of them are same. i assume freearc is still the king, 0.50 version compressed better than 0.666 version back in the day.

Can anyone guide me? i was inspired by skullptura btw

[Simorq]

FreeArc 6.7

[joekkerr]

Thanks but i was asking for an elaborate explanation.

[ffmla]

Originally Posted by joekkerr

Hi, im JoeKkerr

I was actively participating in making repacks from 2010-2014.

i'm not completely new to repacking, but i remember my lack of knowledge in my end days where i wasnt able to compress or keep up with others size.

what i did was what others were doing take files > Freearc > batch file unpacking.
or some files needed precomp or a combination of precomp + srep but problem started arising when some files just werent getting reduced, even after using precomp they werent 'precompressing' or as i'd call it deflating. while groups like blackbox and kaos were able to do it. (ik it applies to only some files that can actually be deflated)

i even tried studying thier repacks and their exe's to replicate result and hence i gave up knowing more people were doing better job than me and my internet was 1MbPS with 100kb download and 25 upload.

I'm thinking of getting back into data compression. whats the current method of compressing data ? i know different games , different methods but most of them are same. i assume freearc is still the king, 0.50 version compressed better than 0.666 version back in the day.

Can anyone guide me? i was inspired by skullptura btw

You may edit the repack groups names from your post.Otherwise the forum MOD warn you..!
At first i am also got the warning.

[Spexxos]

I am also trying to get into the whole compression thing. In my case its primarily for compressing savegames. For that matter I would be interested in the current state of things aswell. I think a might be a bit ahead of you in my knowledge, JoeKkerr, and since I dont want to open up another thread with the same topic I am going to add my questions here. I am sure we both can benefit from this. Here we go:

1. What is DELTA and when to use it?
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?

Cheers

[joekkerr]

Originally Posted by Spexxos

I am also trying to get into the whole compression thing. In my case its primarily for compressing savegames. For that matter I would be interested in the current state of things aswell. I think a might be a bit ahead of you in my knowledge, JoeKkerr, and since I dont want to open up another thread with the same topic I am going to add my questions here. I am sure we both can benefit from this. Here we go:

1. What is DELTA and when to use it?
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?

Cheers

precomp is used when there is a file already compressed (i dont know the technical word) . But a file that is , if you try freearc you wont get a good result because its already compressed with low level compression, this makes high level compression lose its effectiveness. Hence, precomp..deflates the file, i.e removes the low level compression and allows it to be compressed by high level compression freearc offers.

an example would be, try RAR ing a normal text file with medium compression, then 7zip that file, you wont get any good result.
now try taht with using precomp first or on raw text file without winrar.

last time i checked (which was a long time ago) LZMA (7zip) was better than freearc for text files.
That brings us to your next question, why freearc you ask ?
Some files get better compression ratios with Freearc , some with 7zip. Mostly its freearc hands down, but some game files like Assassins creed files get better result with 7zip. Depends on game

[78372]

Originally Posted by Spexxos

I am also trying to get into the whole compression thing. In my case its primarily for compressing savegames. For that matter I would be interested in the current state of things aswell. I think a might be a bit ahead of you in my knowledge, JoeKkerr, since I dont want to open up another thread with the same topic I am going to add my questions here. I am sure we both can benefit from this. Here we go:

1. What is DELTA and when to use it?
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?

Cheers

1. DELTA is a preprocessor for executables and wav audio files. You get better ratio by using delta+lzma rather than just lzma for those files.
2. Know your data, use Razor12911's GFS for testing what to use. If your data contains zlib, use pzlib from ztool/xtool/normal pZlib v3, if contains lz4, use plz4 from ztool, if it has .forge archives from AC games, use AFR, if game is from unreal engine, use uelr and etc. etc. Use msc before anything if your data contains multimedia data.
3. Yep, external compressors, you can't use them with rar and you need to learn coding for using them with 7zip, and with FreeArc you can use them just by editing arc.ini
4. Yeah, xz is a container for LZMA2 compression which has better decompression speed and multithreading which doesn't exist in lzma
5. Yes, you get more speed, but they aren't much matured yet.
6. Check the forum, you may find more.



Joekkerr and others, if you want to know something, ask questions, don't say that I want to know everything, be specific, it will be help for others to answer.

[joekkerr]

Ok i'll be as specific as possible
I remember using precomp + srep and freearc combo on gta 5, but it failed, failed as in compression ratio was barely 70% i.e only 30% got reduced.
IIRC precomp didnt deflate much of the files.

second qn. I looked at the 'specific games technique thread' , i can understand the commands for an executable (-mx, -m2500 ,etc) but i saw some new programs for example here this line

pzlib:m3:max:c128m:t100p+srep:m2f:a2:lolz:d32 without audio file size 16.8gb.

i googled LOLZ, from my understanding, its must be similar to Freearc.
Pzlib is like precomp (correct me if im wrong)
i know srep.
So what is lolz, & why is it preferred over freearc?
am i correct about pzlib ?

Know your data, use Razor12911's GFS for testing what to use. If your data contains zlib, use pzlib from ztool/xtool/normal pZlib v3, if contains lz4, use plz4 from ztool, if it has .forge archives from AC games, use AFR, if game is from unreal engine, use uelr and etc. etc. Use msc before anything if your data contains multimedia data.

are the programs you mentioned are preprocessors like precomp ?

whats an 'msc'?

from my understanding, freearc+precomp+srep was the combo to go, whats the current ones? (i understand different for specific games, but in general ?)

and with FreeArc you can use them just by editing arc.ini

i assume you're talking about using other programs just like we do in a 'batch' file

Yeah, xz is a container for LZMA2 compression which has better decompression speed and multithreading which doesn't exist in lzma

i'll tell you what i udnerstood and then you correct me.
LZMA type of compression is from 7zip, i suppose this xz is a program (which you refer to as container) has lzma2 with decomp speed and multithread.

 wav audio files

I think using lame.exe for temporary converting them into mp3 then unpacking them into wav would be much efficient right ?

but they aren't much matured yet

as in ? unstable ?

[shazzla]

LOLZ,MSC : google for it (krinkels.org)

[78372]

GTA V contains headerless zlib streams and precomp can't detect them all. You need to use pZlib/reflate.

pZlib is like precomp.

LOLZ is like lzma, but compression gain and decompression speed is far better than lzma.

In general pzlib+srep+lolz is best for games like Mad Max, GTA V etc. with zlib/deflate streams.

About xz, that's external lzma2 implementation.

MSC is better, it's a multimedia compressor for wav/bmp/mp3 etc.

Yeah, fa011 is not that stable.

Yeah, I mentioned those ztool/xtool/pzlib/plz4 etc. are like precomp.

Now tell me, do you know how to configure freearc to use external compressors?

[78372]

Most of the games these days use to compress their games before publishing. Generally we know that a game contains audio, texture, video and etc. But we don't see well known files of these types in every games. That is because they make a compressed archive with all the files required for the game. A file that is compressed can't be compressed again. But if we unpack game files and recompress them again, in most of the cases we lost CRC and in some cases the game refuses to run. So to keep the files CRC-Perfect and achieve better compression, we need to precompress them. Precompressors decompresses contents of the game and restores them with perfect condition and 100% CRC match.

In the old days, most of the games were compressed using zlib. Zlib is a well known free and open source data compressor. Zlib uses huffman, that means if we don't precompress, we are not able to achieve a bit of compression of zlibbed files. PRECOMP was well known and the first zlib precompressor to do this stuff. In 2011, reflate - universal zlib precompressor was born in encode.ru but still precomp was mostly used because of reflate's slow speed during recompression and instability. In 2015, GTA V, a game with zlib streams without zlib headers (known as deflate method) was released, precomp was unable to precompress this game. Then, the repackers need to use reflate, which worked very well on GTA V and was able to compress it with 50% ratio. But reflate was slow and so GTA V, Mad Max and other games using reflate used to decompress very slowly. In 2016, Razor12911 released pZlib which was better than precomp in terms of speed and gave same ratio. In december 2016, we received pZlib v3(and v2 before a month, but that was buggy) which was very powerful and fast as hell. Now we have xtool which is even faster than pZlib and gives better ratio. Use it and leave precomp, it will be better.

Now games uses other methods too, they are not using zlib only, so you can't only be happy with pzlib/reflate/precomp. Like Far Cry series uses lzo and lz4, Assassin's Creed series uses lzo and oodle, Need for Speed uses lz4 and oodle, mass effect Andromeda used zstd, unreal engine games use lzo and unity uses lz4 etc. To precompress them, we need to use various precompressors according to game, not a single universal one.

[KaktoR]

You forgot to mention that GFS is buggy. Use only max 4 cores and 16mb chache, and not all opotions on same scan, use only 1 at same time.

[joekkerr]

Originally Posted by shazzla

LOLZ,MSC : google for it (krinkels.org)

Even after using google translate, its hard to read. Not an native english speaker

[joekkerr]

Holy shit, thank you so so so so much man! you were so informative.

Now tell me, do you know how to configure freearc to use external compressors

I cant understand the meaning of this.
Is it the case now that freearc can now run as an 'batch file' and process all the commands for external compressors in itself ?

Most of the games these days use to compress their games before publishing. Generally we know that a game contains audio, texture, video and etc. But we don't see well known files of these types in every games. That is because they make a compressed archive with all the files required for the game. A file that is compressed can't be compressed again. But if we unpack game files and recompress them again, in most of the cases we lost CRC and in some cases the game refuses to run. So to keep the files CRC-Perfect and achieve better compression, we need to precompress them. Precompressors decompresses contents of the game and restores them with perfect condition and 100% CRC match.

Holy shit i remember back when crysis was released it had packed files which can be easily recreated with winzip or rar, and i made each one separate during unpacking. basically ship it packed, then while unpacking used rar comand line to pack it again, each one of it LOL.

Now games uses other methods too, they are not using zlib only, so you can't only be happy with pzlib/reflate/precomp. Like Far Cry series uses lzo and lz4, Assassin's Creed series uses lzo and oodle, Need for Speed uses lz4 and oodle, mass effect Andromeda used zstd, unreal engine games use lzo and unity uses lz4 etc. To precompress them, we need to use various precompressors according to game, not a single universal one.

You mentioned these methods, by how do you identify which game has which.

[Spexxos]

thats a whole lot of new information. thank you guys for that.

So the general approach for data of unkown compression would be to run a scanner like GFS over it. (Are there any other scanners avaiable besides GFS?) Then figure out what you have to do afterwards but here in lies my problem.

@78372 About xtool vs precomp. Precompv046 gave me better results than xtool on my testfiles (Witcher1Savegames) I probably should scan the files so see what they are made of but for now I am wondering if my xtool settings are wrong. To be honest I have no clue how to properly configure xtool/arc.ini