Newbie Question Thread

[felixible11]

Friends. I have a great interest in repacking and am always fascinated by how blatantly you can compress some games. I can program, but here in the forum you are really overwhelmed with tools and programs.

Theoretically it works like this:

1) Precomp the data with xtool (there are already so many plugins here...)
a. Here it already fails. If I want to read in an entire folder, I always get 0 / 0 streams and the precomp file is exactly the same size as the source (that shouldn't be the case?)

%INPUT% = Sourcedir
xtool.exe precomp -mzlib+reflate -d3 -c64mb -t16 "%INPUT%"

b. In addition, there are countless possibilities and every game reacts differently to the combination of precomp algorithms. Can't I read in a game folder and it automatically returns the best combination? Somehow I'm at a loss here.

c. The precomp files must then be compressed (ideally with FreeArc?).
- If I try

arc.exe a -ep1 -dses --dirs -s; -lc- -di -i2 -r -w.\temp -mxtool:c32mb:mzlib data.arc “pack\*”

the data.arc is 1.5x too large as the source file?

I would love your help - somehow I'm at a loss.

Many thanks!

I will take care of the unpacking/installing afterwards. That should be easier...

[KaktoR]

You have to know what compression a game is using.

Your archive is larger because you did not deduplicated the data (srep or xtool:dd1..5) and you did not compress the data at the end. However if you precompress the data with xtool zlib and the archive is larger than the original input, you have possibly found out that the input is zlib compressed.

[felixible11]

Originally Posted by KaktoR

You have to know what compression a game is using.

And how do you do that? Or is that exactly what the precompress is for?

Originally Posted by KaktoR

Your archive is larger because you did not deduplicated the data (srep or xtool:dd1..5) and you did not compress the data at the end. However if you precompress the data with xtool zlib and the archive is larger than the original input, you have possibly found out that the input is zlib compressed.

To understand: Do I need to know what compression the games already have so that
a. to remove/optimize the compression during precompress and (better) compress again later or
b. so that I can pack it even better (over it)?

I'm (still) at a loss here.

[KaktoR]

Experience, knowledge, repetitions.

I would say you can go with game engine repetitions most of the time.

I can tell you that nearly 99.9% of all unreal engine games using either zlib or oodle compression. Older engine versions like UE3 used lzo algo mainly back then.

Most of the unity engine games (if they have bundle files) using lz4hc compression with level 9.

RE_Engine (Resident Evil and all these games) were compressed with either zlib or zstd, but to be honest I didn't worked with this engine since years so things might have changed.

Precompression just means that already compressed streams will be decompressed, that's why the output will be larger than the input, so you can compress the decompressed streams with a better compression than the original used compression. Like if you compress some random folder on your pc with 7z. The compressed files (7z archive) will be smaller then the original input. If you extract the compressed 7z archive, the output will be larger again. That's how precompression works to keep it simple.

I think a and b are the same, or I am really confused.

To keep it simple, these steps in this order:
Precompress -> remove duplicated data -> apply stronger compression

[felixible11]

Thanks for the explanation!

Let's take this as an example:

Originally Posted by lomesmo

__________________
Ripped-encoded Files : Nothing (Multi-11)

Original Size : 7.5G

Compressed Size : 2.7G

xZLib+srep+exe+delta+lzma2
__________________

With this command

arc.exe a archiv.arc "C:\Bionic Commando" -m xZlib+srep+exe+delta+lzma2

I arrive at 6.21 GB. I don't understand that.
Since it is not compromised according to the quote, does precompress not apply?

Apparently I still don't quite get it.

[KaktoR]

xZLib belongs to an old XTool version. If you use current XTool version (0.8.9 I guess) use xtool:zlib instead (maybe reflate is a possible option aswell, I don't have the game).

srep is there for deduplication. Actually you just need srep.exe (64bit version) and then use xtool option :dd1..5 (where 1 is fastest and 5 is slowest but might get a few more duplicates. Default is 3 and in my humble opinion this is peak most of the time).
exe and delta are freearc built-in compressors (honestly I don't really know what they do actually).
lzma2 is from 7z archiver so you need 7z.exe and 7z.dll files aswell.

arc.exe a archiv.arc "C:\Bionic Commando" -mxtool:zlib:dd3+lzma2

You can also go with freearc built-in lzma in addition with 4x4 instead of lzma2 to use up to 16 cpu threads.

arc.exe a archiv.arc "C:\Bionic Commando" -mxtool:zlib:dd3+4x4:lzma

[KaktoR]

I downloaded that game and investigated it real quick. Here are my results


The *.bundle files contain WAVE audio. You can use xtool flac plugin or msc on them.


The movie files are Bink1 videos. You can use xtool bink plugin on them.

[felixible11]

Thank you!

Where i can find xtool flac and bink plugin?

[L33THAK0R]

Originally Posted by felixible11

Thank you!

Where i can find xtool flac and bink plugin?

FLAC is built-in, bink and the majority if not all plugins can be found here

[wrathma]

i compressed a game to multiple files using diskspan custom disk size value. how do i make it work with Simple Installer Script ?

[zioboh]

Hello everyone, yesterday i tried to repack one of my favourite games, i tried looking for some guides but i wasn't lucky at all. Here are the things i did:
1. Got my game folder and packed it into a .arc file (without compression)
2. Ran the file in precomp and got a .pcf file

After that, i wanted to make an installer, but i really can't find anyway to do it.
Could someone help me? Thanks

[kj911]

Has anyone managed to write or create a working lossless h264 recompressor in recent years from existing losslessh264 and or avrecode sources?? (Not in all games, but the presence of x264 videos, next to/instead of Bink2, is becoming more common.) Compression ratio avg ~15% better than 7z/any ~1-3% size reduction. Leaked executables used from DropBox??

Shelwien preserved few years ago halfly working H264 recompressor from encode.su site. The file, not available.

[TellASpray]

Hi, after reading this thread I think I have some knowledge but I wanna be sure of what I know :

1. Deduplication is the process of deleting identical files ?
2. Precompression is for decompressing already compressed files to compress them better afterwards ?

Now I have some questions about things I don't understand :

3. What are streams ?
4. What do changing chunks, dictionary size and depth do ?
5. If DiskSpanGUI is a compressor why can't I directly extract files with WinRar for example ?

[wareck]

dedup it's not only files, but stucture of file
for example, a .zip file full of GIFs.
Each GIF file will start the same (header), and dedup will also take this into account to reduce the final size.

Precompression => yes decompress file and some kind of file (gif, png ,mp3) to let them compressed later with a better compression system

Chunks/dictionary and depth :
They will impact performances (better speed, better compression, better detection)
these values are a compromise between performance and speed.
values that are too high can also lead to CRC errors.
like dictionary, it's a way to reduce a file size if inside they is several time the same words, same structure, or same hex data
depth is to find stream deeper inside file
like if you have a zip file with inside a movie with a wav file. depth 0 => zip file , depth1 =>zip file with something inside=>movie , depth3 => zip file with something inside=>movie => wav
but with a high depth, soft may need to read file several times => more time to compress/uncompress
also going deep may add some errors and you will have CRC error during decompression
again => these values are a compromise between performance and speed.

streams...
They are in files
For example, to put it simply, a large game file can contain images, sound, texture, text...
during the analysis process, software can indentify a sound stream and decode/decompress/recompress

diskspangui is a compressor, but to be fair, is a collection of diffèrent programs
in the same file you can have several type of data

like if you mix oil and vinegar to make a salad sauce
then you want to put back oil and vinegar in their bottle
you wll need a tool to separate them and work on them each on their own.

diskspangui it's a GUI for several freearc function.
you can open files with freearc (not to unpack, just read them, if you want to unpack, you will need to add soft/config in freearc software)

[TellASpray]

Thank you very much for your answer, do you mind sharing config files (arc.ini I guess ?) or softwares (if that's what you mean by "soft") or even threads about those :]