Newbie Question Thread

[adrianskiloses]

Best compression method for text files?

[shazzla]

Razor,ppmd.

[kj911]

Special question: This minimal software requirement, prerequisities have installed on freshly installed Win7 x64 machine???

Tested few compressors in pure Win7 x64 installations and get very slow compression speed's. Not installed VC++, NETFX and more...

LOLZ in get around 100-150kB/s speed and not utilize more core/threads..
Chechked in LZTurbo v1.2 its slow.
Checked Razor 1.03.7 its slooow.
Checked Nanozip 0.09 x64 its finely goodly packing via optimum1 mode utilize 8 threads and get avg. 13MB/s compression speed! optimum2 or cc mode its slow. (Tested in highly compressibble data sample, compression ratio higher than 20:1)
Tested WinRAR 5.60 x64 and packing in very goodly avg. 25MB/s speed use, Best/Solid mode and 1GB dictionary. CPU usage 30-60% percentage, probably bottlenecked via 17 years old HDD.

Installed VC++ 2012 packages taken from Catalyst 15.7.1 package and LOLZ compress its faster! Any idea?

Any members I have Xeon CPU??

My PC in has Xeon E5-2665 CPU + 16GB RAM. Will hope compression speed via 1 threads around probably really getting avg. 400-800kB/s or higher. (LDMF mode) Or higher than ~2MB/s via MTT mode and 8+ threads.

[FitGirl]

lolz don't have dependencies.
It's compression speed for 1 thread is slow, you can use faster params or turn on one of two MT modes.

On your Xeon you can utilize all 16 threads, it's also the maximum lolz can use anyway.

Also, ldmf mode is very slow compared to non-ldmf, while giving not much better results than srep+lolz

[sailwhist]

Hello. I reviewed literally everything on this forum which is considered as a "guide", and I still can't understand what I should do and how should I apply all that compression methods and scripts on different kind of files (audio, movies etc.) and moreover how should I determine what is what and which files should I compress with which method?
I reviewed best compression methods index for the game I wanted to compress (gta4) and I have no clue what should I do with that information:
gta iv complete edition 12.9GB
audio file srep+lolz...
you can use msc but i didn't see improve ratio...
movie bkp...
rest of the file zlib:max:c32m+srep:m3f:a2+lolz...
I downloaded DiskSpanGUI and all I understood for now is how to select the input game folder. How should I learn about game compression if there is no comprehensible clear guide for begginers, only some general info like "use XTool and compress your games", but not a single example about how to apply all that scary configs and scripts for a certain files.
Please, if I missed something on this forum what potentially could've helped me - point me on that.

[sailwhist]

There is literally not a single comprehensive step-by-step instruction for beginners on this forum, all I see as a beginner is methods and troubleshooting discussions. Sorry for that, but as a beginner who spent 2 days on this forum I understood nothing, even though I really wanted. Thank you.

[KaktoR]

@sailwhist

I will take your example for GTA4.

Sometimes you already see what a game file could include just by file name or folder name.

For example the following file

GTAIV\pc\audio\sfx\ambient_streams.rpf

"sfx" just means "sound effects" in the audio terminology. This means, you already know what is inside this file or what the file will contain mostly - namely sound/audio files of some sorts.

I don't have the game so I have to check steamdb depot for this game and sort the file list by type. What I can see is that literally all *.rpf files contain sound/audio files, just by looking at the file name and/or folder where the files are located.

Another example for this type is

GTAIV\pc\audio\sfx\radio_*.rpf

Just looking at all the file names beginning with radio_ I am pretty sure all these files contain all the music for the radio stations in game, just because I know the word "radio" and the name of the radio station in the file name.

But because I don't have the game nor any of the files, I can just guess that the files contain riff-wave files, mp3 files or raw files or whatever else, because I know msc can handle them mostly. And the post creator you have quoted said "you can use msc but i didn't see improve ratio...", so I think my guess is correct here lol.

Next one: movie files
For bink movies (BIK or BK2) you can mostly use BinkPack (bpk), which will compress this files lossless. Just note that sometimes it will not work, either because the bink files using some sort of alpha channel, using a codec which bpk cannot handle correctly, or - which applies to BK2 files - have a newer codec which BinkPack cannot handle at all. If you have Bink video files, you can do the following:

1. Use HxD and see how the beginning of the file looks like. If the file has a "KB2n" (image) or anything greater then the "n" letter, BinkPack will fail on these files (this is mostly the case in new games, especially games which are build with unreal engine 4/5).
2. Copy all the Bink video files to a test folder and use bpk on them, testing compression and decompression. Or use bpk file-by-file instead of all at once. This way you can sort out faulty ones of the list and note them somewhere, so you know what files making trouble to exclude them from bpk compression.

Next one: zlib etc.
Again back to steamdb depot for the game GTA4, I also see "*img" files, which are 6.91 GiB in size. I would guess that these files contain all sorts of other files and textures. So now, the post you quoted sais "rest of the file zlib:max:c32m+srep:m3f:a2+lolz... ", which is including this *.img files. In other words, it seems that these *.img files are compressed with either zlib or deflate I would say. Devs simply do this to save space. To be sure, you can follow this little guide (to be fair this is for oodle compression, but just ignoring oodle option and using zlib option isn't that complicated).

We always want to extract all the data/streams in the game files if possible, in other words we want to decompress them (xtool/other tools) to have fully possibility of deduplication (srep) and compression (lolz/lzma/whatever else) in order to compress them even better than the original.

[sailwhist]

Originally Posted by KaktoR

@sailwhist

I will take your example for GTA4.

Sometimes you already see what a game file could include just by file name or folder name.

For example the following file

GTAIV\pc\audio\sfx\ambient_streams.rpf

"sfx" just means "sound effects" in the audio terminology. This means, you already know what is inside this file or what the file will contain mostly - namely sound/audio files of some sorts.

I don't have the game so I have to check steamdb depot for this game and sort the file list by type. What I can see is that literally all *.rpf files contain sound/audio files, just by looking at the file name and/or folder where the files are located.

Another example for this type is

GTAIV\pc\audio\sfx\radio_*.rpf

Just looking at all the file names beginning with radio_ I am pretty sure all these files contain all the music for the radio stations in game, just because I know the word "radio" and the name of the radio station in the file name.

But because I don't have the game nor any of the files, I can just guess that the files contain riff-wave files, mp3 files or raw files or whatever else, because I know msc can handle them mostly. And the post creator you have quoted said "you can use msc but i didn't see improve ratio...", so I think my guess is correct here lol.

Next one: movie files
For bink movies (BIK or BK2) you can mostly use BinkPack (bpk), which will compress this files lossless. Just note that sometimes it will not work, either because the bink files using some sort of alpha channel, using a codec which bpk cannot handle correctly, or - which applies to BK2 files - have a newer codec which BinkPack cannot handle at all. If you have Bink video files, you can do the following:

1. Use HxD and see how the beginning of the file looks like. If the file has a "KB2n" (image) or anything greater then the "n" letter, BinkPack will fail on these files (this is mostly the case in new games, especially games which are build with unreal engine 4/5).
2. Copy all the Bink video files to a test folder and use bpk on them, testing compression and decompression. Or use bpk file-by-file instead of all at once. This way you can sort out faulty ones of the list and note them somewhere, so you know what files making trouble to exclude them from bpk compression.

Next one: zlib etc.
Again back to steamdb depot for the game GTA4, I also see "*img" files, which are 6.91 GiB in size. I would guess that these files contain all sorts of other files and textures. So now, the post you quoted sais "rest of the file zlib:max:c32m+srep:m3f:a2+lolz... ", which is including this *.img files. In other words, it seems that these *.img files are compressed with either zlib or deflate I would say. Devs simply do this to save space. To be sure, you can follow this little guide (to be fair this is for oodle compression, but just ignoring oodle option and using zlib option isn't that complicated).

We always want to extract all the data/streams in the game files if possible, in other words we want to decompress them (xtool/other tools) to have fully possibility of deduplication (srep) and compression (lolz/lzma/whatever else) in order to compress them even better than the original.

Thank you for your assistance, but still I have no clue how to operate with all of these in DiskSpanGUI and how to apply all these methods, select files by files and make everything combined work altogether in a final compressed game.
Talking about audio/movies - I'm beginning to understand where they stored, but I'm not sure how to find every single audio/movie file in game folder to make sure I compressed all of these files with the right method.
Also, I don't understand how to choose exact settings for that method for gta4 (zlib and etc.). In diskspangui it only shows me presets for that but without numbers like in those guy method. There are a lot of settings in that software which scares me.
I literally don't know everything and I would like to see a detailed guide with an example on some game about how to apply all of these so me and others could use that as a complete FAQ on compressing games.
Thank you.

[KaktoR]

Ah, I forget to explain a small but important detail about all the numbers in the method "those guys" sometimes share.

First, you have to keep in mind that tools are changing over time. Old tools like "zlib:max:c32m" (which should be this tool or maybe this) are not used anymore nowadays, because some tools are "evolving" over time, new tools are comming and old ones are obsolete.

So, the guy you quoted posted this method on 14-03-2018, which was over 5 years ago. On this time there was no "xtool". Means, methods and tools can (and most likely will) change over time. The method "zlib:max:c32m" belongs to an old tool. No one is using this tool anymore. "zlib" is the codec, ":max" is some other option I don't remember (I guess it has something to do with stream detection) and ":c32m" is the chunk size used for this tool.

Basically you have to "transfer old methods into new methods". So the old method "zlib:max:c32m" is something like "xtool:c32mb:mzlib" in diskspangui.

Edit:
Again I forgot to tell you "how to use this method on GTA4 for your needs" lol, sorry.

Personally I do all of this configuration stuff through the INI file "Resources\DSG_Settings.ini", just because I find it more easier.

So for your example of GTA4 it could look like this:

[Game1Settings]
Name=Grand Theft Auto 4
Exec=
EGPath=C:\Program Files (x86)\Rockstar Games\Grand Theft Auto 4
MethodState=0
Data1.Name=Data<game><data>-01.bin
Data1.Method1=msc|0
Data1.InclText=*.rpf

Data2.Name=Data<game><data>-01.bin
Data2.Method1=xtool:mzlib+srep+lolz|0
Data2.InclText=*.img

Data3.Name=Data<game><data>-01.bin
Data3.Method1=bpk|0
Data3.InclText=*.bik

Data4.Name=Data<game><data>-01.bin
Data4.Method1=srep+lolz|0
Data4.ExclText=*.img|*.rpf|*.bik

Data1 only includes *.rpf (audio) files, so I would suggest msc over srep+lolz, just because for the fact that lolz is not meant to used with audio files at all. lolz is meant to be used especially for texture files. So I would use msc, which is faster and should have near the same ratio. Sometimes it is a good idea to use even msc+srep, but I cannot say for sure here in this case because I don't have this game to test it.
Data2 includes only *.img files, which probably using some compression like zlib or deflate, so you have to use xtool here. If xtool:mzlib doesn't work, try with another codec like xtool:mpreflate or xtool:mreflate.
Data3 includes only *.bik files. I guess it ok to use bpk method only.
Data4 includes all other files, excluding *.img, *.rpf and *.bik, so you can just use srep+lolz if you want.

|0 indicates that this method belongs to freearc, while |1 is 7-zip (.7z) and |2 is RAR (.rar) (freearc methods cannot be used with |1 or |2). I guess using |1 or |2 are special and absolutely rare cases of some sorts.
InclText and ExclText telling diskspangui/freearc, that only given file types/files with paths should be included in given archive, or exclude them of the given archive. In the example above we have 3 archives which includes specific file types. So we need a last archive which includes all other files except the 3 previous archives, otherwise we would missing all other files on installation which are not exclusively included in other archives. Makes sense, does it? PS: In InclText and ExclText you can (or have to) separate file types/files with paths with a | character.
<game><data> are just placeholders or variables. <game> is actually [Game#Settings] (it takes the number # from [Game1Settings] section, so 1), <data> is actually Data#, which takes the number # from Data#. However you can name the archives to whatever you want. Just make sure to not use any super special characters.

[kj911]

FitGirl: Thanks from reply!

Two MT mode? Or MTT only?? LDMF its supporting multithreaded compression, with lower ratio! Special commands trick to usage.

https://i.kek.sh/bNhofYWbNzw.png
https://i.kek.sh/2udWoqik8Ut.png

Xeon 16 threads with 16GB RAM from LOLZ --> Use 128MB dictionary size. Memory its fully loaded.

srep:m3f:a0:l512+LOLZ:mtt1:mt2/4+ worser ratio than srep:m3f:a0:l512+LOLZ:ldmf1 mode.

[sailwhist]

Originally Posted by KaktoR

Ah, I forget to explain a small but important detail about all the numbers in the method "those guys" sometimes share.

First, you have to keep in mind that tools are changing over time. Old tools like "zlib:max:c32m" (which should be this tool or maybe this) are not used anymore nowadays, because some tools are "evolving" over time, new tools are comming and old ones are obsolete.

So, the guy you quoted posted this method on 14-03-2018, which was over 5 years ago. On this time there was no "xtool". Means, methods and tools can (and most likely will) change over time. The method "zlib:max:c32m" belongs to an old tool. No one is using this tool anymore. "zlib" is the codec, ":max" is some other option I don't remember (I guess it has something to do with stream detection) and ":c32m" is the chunk size used for this tool.

Basically you have to "transfer old methods into new methods". So the old method "zlib:max:c32m" is something like "xtool:c32mb:mzlib" in diskspangui.

Edit:
Again I forgot to tell you "how to use this method on GTA4 for your needs" lol, sorry.

Personally I do all of this configuration stuff through the INI file "Resources\DSG_Settings.ini", just because I find it more easier.

So for your example of GTA4 it could look like this:

[Game1Settings]
Name=Grand Theft Auto 4
Exec=
EGPath=C:\Program Files (x86)\Rockstar Games\Grand Theft Auto 4
MethodState=0
Data1.Name=Data<game><data>-01.bin
Data1.Method1=msc|0
Data1.InclText=*.rpf

Data2.Name=Data<game><data>-01.bin
Data2.Method1=xtool:mzlib+srep+lolz|0
Data2.InclText=*.img

Data3.Name=Data<game><data>-01.bin
Data3.Method1=bpk|0
Data3.InclText=*.bik

Data4.Name=Data<game><data>-01.bin
Data4.Method1=srep+lolz|0
Data4.ExclText=*.img|*.rpf|*.bik

Data1 only includes *.rpf (audio) files, so I would suggest msc over srep+lolz, just because for the fact that lolz is not meant to used with audio files at all. lolz is meant to be used especially for texture files. So I would use msc, which is faster and should have near the same ratio. Sometimes it is a good idea to use even msc+srep, but I cannot say for sure here in this case because I don't have this game to test it.
Data2 includes only *.img files, which probably using some compression like zlib or deflate, so you have to use xtool here. If xtool:mzlib doesn't work, try with another codec like xtool:mpreflate or xtool:mreflate.
Data3 includes only *.bik files. I guess it ok to use bpk method only.
Data4 includes all other files, excluding *.img, *.rpf and *.bik, so you can just use srep+lolz if you want.

|0 indicates that this method belongs to freearc, while |1 is 7-zip (.7z) and |2 is RAR (.rar) (freearc methods cannot be used with |1 or |2). I guess using |1 or |2 are special and absolutely rare cases of some sorts.
InclText and ExclText telling diskspangui/freearc, that only given file types/files with paths should be included in given archive, or exclude them of the given archive. In the example above we have 3 archives which includes specific file types. So we need a last archive which includes all other files except the 3 previous archives, otherwise we would missing all other files on installation which are not exclusively included in other archives. Makes sense, does it? PS: In InclText and ExclText you can (or have to) separate file types/files with paths with a | character.
<game><data> are just placeholders or variables. <game> is actually [Game#Settings] (it takes the number # from [Game1Settings] section, so 1), <data> is actually Data#, which takes the number # from Data#. However you can name the archives to whatever you want. Just make sure to not use any super special characters.

Hello again. Thank you, now there is some more insight into this for me. I copied preset you provided to .ini file and currently it is in a process of conversion. But is it okay that it going that slow, I mean it shows me 6 hours and it's gradually ascending. https://pasteboard.co/U1kdlXT9Qr0b.png

Also I don't understand how to pack it all in installer after compression, I wouldn't mind a detailed explanation of how to create an installer after compressing all the files Thank you

[KaktoR]

lolz compressor is slow, yes. However you can speed up the process by using some options for it, for example utilize all your cpu cores.

Example

lolz:mtt1:mt12

12 = number of cpu threads. If you have more or less cpu threads, change the number.

As for the installer, you have several choices here. There are some installers which are compatible with diskspangui. You can find supported installers in the diskspangui topic, I think it is the second or third post in the topic. However you can also use diskspangui itself to unpack the archives. Just copy diskspan_gui.exe in the same folder where all the created archives are and run it.

[sailwhist]

Originally Posted by KaktoR

lolz compressor is slow, yes. However you can speed up the process by using some options for it, for example utilize all your cpu cores.

Example

lolz:mtt1:mt12

12 = number of cpu threads. If you have more or less cpu threads, change the number.

As for the installer, you have several choices here. There are some installers which are compatible with diskspangui. You can find supported installers in the diskspangui topic, I think it is the second or third post in the topic. However you can also use diskspangui itself to unpack the archives. Just copy diskspan_gui.exe in the same folder where all the created archives are and run it.

But, where do I set options for threads? I had to abort the process since it shown me 10 hours estimated time and it was ascending more and more. I just copypasted these and started compression: https://pasteboard.co/4orsQiqonK2O.png
I don't think 24 GB game should be compressing to a 13 GB for 10+ hours

[KaktoR]

It will look like this in the settings:

Data2.Method1=xtool:mzlib+srep+lolz:mtt1:mt12|0

However you can set it directly even in the GUI aswell by type it manually or select one of the presets for lolz:
https://i.imgur.com/xMp4E29.png


* All possible options for the lolz compressor can be found here: "Resources\FINAL\LOLZ\Options.txt"

Time for compression mostly depends on your hardware, especially how fast your cpu and ram is, and of course of the input data and used settings. So there is no way to telling you how long compression will take if no one write the time and his pc specs.

[sailwhist]

Originally Posted by KaktoR

It will look like this in the settings:

Data2.Method1=xtool:mzlib+srep+lolz:mtt1:mt12|0

However you can set it directly even in the GUI aswell by type it manually or select one of the presets for lolz:
https://i.imgur.com/xMp4E29.png


* All possible options for the lolz compressor can be found here: "Resources\FINAL\LOLZ\Options.txt"

Time for compression mostly depends on your hardware, especially how fast your cpu and ram is, and of course of the input data and used settings. So there is no way to telling you how long compression will take if no one write the time and his pc specs.

After adding these settings to .ini file, now it is compressing as it should and percentage of progress goes up, last time it was just stuck at 10% and time showed 10 hours, now its 1:30 hours estimate and it looks like it is working. https://pasteboard.co/QPptA0Q6bKIw.png
Thank you, I hope it'll work.
BTW, what's wrong with Imgur? It's not opening any links, just shows it like that: https://pasteboard.co/rE3HCJIzwHaM.png