Current compressing techniques guidance

[doofoo24]

Originally Posted by joekkerr

Holy shit, thank you so so so so much man! you were so informative.

how do you identify which game has which.

Originally Posted by 78372

use Razor12911's GFS for testing what to use...

[78372]

Originally Posted by Spexxos

@78372 About xtool vs precomp. Precompv046 gave me better results than xtool on my testfiles (Witcher1Savegames) I probably should scan the files so see what they are made of but for now I am wondering if my xtool settings are wrong. To be honest I have no clue how to properly configure xtool/arc.ini

From precomp 046, it's a complete compressor, not only a precompressor, so it will give better result on small files and when you have compression on. To use xtool, you must use xtool+srep+lzma(or xtool+srep+lolz) to get better ratio, else it will just inflate files, not compress, plus xtool isn't complete yet, it's a beta tool, it has bugs.

[Spexxos]

Originally Posted by 78372

From precomp 046, it's a complete compressor, not only a precompressor, so it will give better result on small files and when you have compression on. To use xtool, you must use xtool+srep+lzma(or xtool+srep+lolz) to get better ratio, else it will just inflate files, not compress, plus xtool isn't complete yet, it's a beta tool, it has bugs.

My bad I should have been more detailed. I used precompv46 only as a precompressor, so with these parameters ("precomp -cn -intense").
The file that precomp delivered was about %10 bigger than the one from xtoo. Which means precomp found more to decompress?

Aferwards I put both the testfiles from precomp and from xtool into srep393a (-mx), then I proceeded to compress them with 7z(7za.exe a -t7z -m0=LZMA2 -mmt=1 -mx9 -md=1536m -mfb=273 -ms=on -mqs=on)

Result is the precomp testfile is half the size of the xtool one. That makes me think I have to configure xtool differently.

[78372]

May be xtool didn't work good on that, it's a beta tool after all.
Try pZlib v3 instead

[Spexxos]

Originally Posted by 78372

May be xtool didn't work good on that, it's a beta tool after all.
Try pZlib v3 instead

what settings do you suggest for pzlib?


EDIT: pzlibv3, christmas edition, gives the same result as precomp v046 (pZLib3a.exe e -c128m -t75p). It is also noticeably faster. Is there any sense in checking out older versions of pzlib? I have seen older versions have more switches/options. Is that because the needed more manual input from the user to give the best result, and pzlib3 does that automatically or is there an other reason for that?


EDIT2: After Precomp/pzlib what defines the order in which you apply further tools like DELTA and Srep, msc etc? Which one comes first, and why? Or does the order not matter at all?

[78372]

MSC should be used *ONLY* for multimedia data and delta should be used *ONLY* for exe and wav files, applying delta for normal game files will cause worse ratio, and without multimedia data, msc can't do anything.
MSC should come first, then pzlib, then srep and at last lzma/lolz

[Spexxos]

Originally Posted by 78372

MSC should be used *ONLY* for multimedia data and delta should be used *ONLY* for exe and wav files, applying delta for normal game files will cause worse ratio, and without multimedia data, msc can't do anything.
MSC should come first, then pzlib, then srep and at last lzma/lolz

Thank you. The thing about DELTA is that, that on my witchersave testfile I got better compression with precompv46+srep393a+DELTA+lzma2 compared to precompv46+srep393a+lzma2. Any Idea why? Or is that the exception that confirms the rules?

[78372]

That's a save file, not a game file and probably delta works better for it. Not everywhere delta works and using them on executables is purely safe and that's why I mentioned about that. But in general, delta doesn't work better on game files(such as textures).

[Spexxos]

Originally Posted by 78372

That's a save file, not a game file and probably delta works better for it. Not everywhere delta works and using them on executables is purely safe and that's why I mentioned about that. But in general, delta doesn't work better on game files(such as textures).

I see. My main interest for now is compressing my savegames. 500gb unpacked/ 190gb packed. They got quite large recently. I think its time to change that. Anyway. Thank you very much for your help.

Its a shame that there is no "Best Compression Methods for 'Specific' SaveGames INDEX" Maybe its time to do something about that


EDIT:

ok, two more questions popped up: Where can I find lolz, the only thread I found was Lolzx which is apparently a private compressor?
And second whats the matter with pzlib3 according to razor there is this command "-x" which is supposed to give the best output. Cant find it anywhere else mentioned though, not even in the instructions for pzlib. What does this command do? Also what is the use case for pzlibs "-r" deflate. Its different decompression mode I assume?

[Andu21]

Originally Posted by Spexxos

After Precomp/pzlib what defines the order in which you apply further tools like DELTA and Srep, msc etc? Which one comes first, and why? Or does the order not matter at all?

Hard to tell, sometimes delta before srep improves compression, same with msc.

Originally Posted by Spexxos

Where can I find lolz, the only thread I found was Lolzx which is apparently a private compressor?

http://fileforums.com/showthread.php?t=99953

[Spexxos]

According to this page [SPOILER]https://translate.googleusercontent....ygj-rpHOhRlC3g[/SPOILER]

these setting should provide good compression without sacrificing it for a faster compression. Am I right? Results are ~7% better than with lzma2 in my testing. Is there room for any improvement? lolz has a lot of options I havent even touched yet.

"lolz -mmt0 -mt1 -mtb64 -d2032 input output"


How do you acutally unpack lolz archives?

[Spexxos]

So, I have been testing alot the last two days. Compressors like lolz, lz4,5, mcm, rzm nz etc. I found that Nanozip gives me the best compression ratio. Do you guys think that Nanozip is reliable enough for a longterm use?

[elit]

Here is how it work today:

You install the game to get its individual files in folder. First, nowadays we have many specialized compressors, we have bpk for compressing bik/bk2 (bink) videos, we have mpz for mp3 files, we have oggre for ogg files, we have packjpg for jpg files and some others but these count most.

So you use their respective compressors for those. Next we replaced internal lzma of FA for 64bit variant(to allow bigger dictionaries etc). XZ/7zip is one choice, I made xnlz by modifying plzip to get as close to FA:xlzma experience as possible in terms of speed and features, however you can also use FA's default its not that big difference.

Bigger difference is from lolz which can pull ~10% better to lzma. Lolz is a general purpose compressor but specialized on textures using lzma with additional cm modeling for textures/images, whose can detect within files, so its good for games. Its slower so you may combine it with lzma for different files. Its not really better outside of textures than lzma thats why.

As you may noticed, I spoke lot about specialization and individual files, thats the game now. If game files are in packs and compressed, then(and only then) you need tools like ztool, xtool,punity or precomp. In fact for precomp I dont see much purpose anymore, it doesnt support stdio and is generally slower, really tools from Razor is all about. Also, often you may need to go deeper and extract/repack game files using quickbms scripts and other tools, for example you can only compress XCOM2 WotCh that way as it use oodle compression.

I forgot, there are also specialized precomps for certain engines, uelr for Unreal's lzo, afr for anvil(Ubisoft's .forge) lzo, xtool for CRI, punity for Unity's lz4. Ztool is for older lzo/zstd/lz4 but work always for pzlib/deflate.

To scan files you can use GFS scanner or my lz_scan, GFS have nice UI but is buggy and unstable in anything other than zlib/deflate. lz_scan also support afr/uelr scan.

Srep you use latest 393a 64bit, FA 0.67 as it fixed *lot* of bugs and added features, dont even think about anything lower.

Finally, you generally use default internal delta of FA on all but images/textures. As for xdelta, this has to do with recomposing CRC matched files back. For example, before xtool you would have to use quickbms with cpk.bms to extract .cpk files. When you repack them back they were not same because of header data change so you would use xdelta for that final touch. Same goes for recompressing .wem files. Other than that I dont see much point for xdelta compression itself, its all about canceling duplicates and srep is already doing that AFAIK(FitGirl you may want to step here to explain better if I am wrong).


1. What is DELTA and when to use it?
A: delta type compression for canceling duplicates and get diffs, use always except on textures, or never, usually no big deal
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
A: first forget precomp(MT),pzlib,msc and all that crap, just use:
ztool for deflate/pzlib,lzo,lz4,zstd(last 3 will likely not work on newest games)
xtool for deltate/pzlib,CRI
punity for *some* unity games
uelr for most unreal games(.tfc, .upk...)
afr for anvil games
srep always, its for data deduplication not a recompressor
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
A: Because FA is so much more that that. I dont think you can use bpk+oggre+mpz+packjpg+uelr+srep+lzma in one go, on files sorted with one of many advanced ways best fitting its game files structure, in anything else other than FA or based on FA, can you? You can use 7zip as external LZMA within FA if you like.
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
A: Same as 7zip or xnlz, 64bit support
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
A: Its still early alpha/beta and lack feature I believe. Its also not (supposed) to be free once finished. My personal opinion is that Bulat pretty much killed his own legacy, almost nobody will use it and he is wasting his time.
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?
A: Start reading threads on all mentioned tools upside down both here and on encode.ru. In fact, read this whole site(and encode). Then learn Russian and read whole Krinkels.org website. Then (and only then), you will become Jedi.


So thats about it. As you see its not anymore old times when you could just precomp+(srep)+FA and thought you done. Today its much more complicated and often exotic practices are required. Good luck.

[PsYcHo_RaGE]

Originally Posted by elit

Here is how it work today:

You install the game to get its individual files in folder. First, nowadays we have many specialized compressors, we have bpk for compressing bik/bk2 (bink) videos, we have mpz for mp3 files, we have oggre for ogg files, we have packjpg for jpg files and some others but these count most.

So you use their respective compressors for those. Next we replaced internal lzma of FA for 64bit variant(to allow bigger dictionaries etc). XZ/7zip is one choice, I made xnlz by modifying plzip to get as close to FA:xlzma experience as possible in terms of speed and features, however you can also use FA's default its not that big difference.

Bigger difference is from lolz which can pull ~10% better to lzma. Lolz is a general purpose compressor but specialized on textures using lzma with additional cm modeling for textures/images, whose can detect within files, so its good for games. Its slower so you may combine it with lzma for different files. Its not really better outside of textures than lzma thats why.

As you may noticed, I spoke lot about specialization and individual files, thats the game now. If game files are in packs and compressed, then(and only then) you need tools like ztool, xtool,punity or precomp. In fact for precomp I dont see much purpose anymore, it doesnt support stdio and is generally slower, really tools from Razor is all about. Also, often you may need to go deeper and extract/repack game files using quickbms scripts and other tools, for example you can only compress XCOM2 WotCh that way as it use oodle compression.

I forgot, there are also specialized precomps for certain engines, uelr for Unreal's lzo, afr for anvil(Ubisoft's .forge) lzo, xtool for CRI, punity for Unity's lz4. Ztool is for older lzo/zstd/lz4 but work always for pzlib/deflate.

To scan files you can use GFS scanner or my lz_scan, GFS have nice UI but is buggy and unstable in anything other than zlib/deflate. lz_scan also support afr/uelr scan.

Srep you use latest 393a 64bit, FA 0.67 as it fixed *lot* of bugs and added features, dont even think about anything lower.

Finally, you generally use default internal delta of FA on all but images/textures. As for xdelta, this has to do with recomposing CRC matched files back. For example, before xtool you would have to use quickbms with cpk.bms to extract .cpk files. When you repack them back they were not same because of header data change so you would use xdelta for that final touch. Same goes for recompressing .wem files. Other than that I dont see much point for xdelta compression itself, its all about canceling duplicates and srep is already doing that AFAIK(FitGirl you may want to step here to explain better if I am wrong).


1. What is DELTA and when to use it?
A: delta type compression for canceling duplicates and get diffs, use always except on textures, or never, usually no big deal
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
A: first forget precomp(MT),pzlib,msc and all that crap, just use:
ztool for deflate/pzlib,lzo,lz4,zstd(last 3 will likely not work on newest games)
xtool for deltate/pzlib,CRI
punity for *some* unity games
uelr for most unreal games(.tfc, .upk...)
afr for anvil games
srep always, its for data deduplication not a recompressor
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
A: Because FA is so much more that that. I dont think you can use bpk+oggre+mpz+packjpg+uelr+srep+lzma in one go, on files sorted with one of many advanced ways best fitting its game files structure, in anything else other than FA or based on FA, can you? You can use 7zip as external LZMA within FA if you like.
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
A: Same as 7zip or xnlz, 64bit support
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
A: Its still early alpha/beta and lack feature I believe. Its also not (supposed) to be free once finished. My personal opinion is that Bulat pretty much killed his own legacy, almost nobody will use it and he is wasting his time.
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?
A: Start reading threads on all mentioned tools upside down both here and on encode.ru. In fact, read this whole site(and encode). Then learn Russian and read whole Krinkels.org website. Then (and only then), you will become Jedi.


So thats about it. As you see its not anymore old times when you could just precomp+(srep)+FA and thought you done. Today its much more complicated and often exotic practices are required. Good luck.

Can i have your lz_scan??

[PsYcHo_RaGE]

Unfortunately i am too here a newbie i mean newbie to all these things that but i have been reading every post here since 3 months, dude only 3 months

i am new to all these things only 3 months, i was just wandering around looking for games on repack sites and i did't know what was repacking and what r repacks my only target was to find the game download it and play, this was my life after that i was curious whats it different sites different game sizes i feel very curious about what it was and then after searching a lot of stuff on internet i found these forums joined em and start reading posts, using discord chat with new friends and now its still going, now i can repack most of games on my PC, still i am noob i want to be a jedi as @elit mentioned