#31
|
|||
|
|||
Best compression method for text files?
|
Sponsored Links |
#32
|
|||
|
|||
Razor,ppmd.
|
The Following User Says Thank You to shazzla For This Useful Post: | ||
adrianskiloses (12-05-2023) |
#33
|
|||
|
|||
LOLZ speed vs. Windows frameworks...
Special question: This minimal software requirement, prerequisities have installed on freshly installed Win7 x64 machine???
Tested few compressors in pure Win7 x64 installations and get very slow compression speed's. Not installed VC++, NETFX and more... LOLZ in get around 100-150kB/s speed and not utilize more core/threads.. Chechked in LZTurbo v1.2 its slow. Checked Razor 1.03.7 its slooow. Checked Nanozip 0.09 x64 its finely goodly packing via optimum1 mode utilize 8 threads and get avg. 13MB/s compression speed! optimum2 or cc mode its slow. (Tested in highly compressibble data sample, compression ratio higher than 20:1) Tested WinRAR 5.60 x64 and packing in very goodly avg. 25MB/s speed use, Best/Solid mode and 1GB dictionary. CPU usage 30-60% percentage, probably bottlenecked via 17 years old HDD. Installed VC++ 2012 packages taken from Catalyst 15.7.1 package and LOLZ compress its faster! Any idea? Any members I have Xeon CPU?? My PC in has Xeon E5-2665 CPU + 16GB RAM. Will hope compression speed via 1 threads around probably really getting avg. 400-800kB/s or higher. (LDMF mode) Or higher than ~2MB/s via MTT mode and 8+ threads. Last edited by kj911; 19-05-2023 at 12:43. Reason: typo/info fix |
#34
|
|||
|
|||
lolz don't have dependencies.
It's compression speed for 1 thread is slow, you can use faster params or turn on one of two MT modes. On your Xeon you can utilize all 16 threads, it's also the maximum lolz can use anyway. Also, ldmf mode is very slow compared to non-ldmf, while giving not much better results than srep+lolz Last edited by FitGirl; 20-05-2023 at 06:10. |
#35
|
|||
|
|||
Hello. I reviewed literally everything on this forum which is considered as a "guide", and I still can't understand what I should do and how should I apply all that compression methods and scripts on different kind of files (audio, movies etc.) and moreover how should I determine what is what and which files should I compress with which method?
I reviewed best compression methods index for the game I wanted to compress (gta4) and I have no clue what should I do with that information: gta iv complete edition 12.9GB audio file srep+lolz... you can use msc but i didn't see improve ratio... movie bkp... rest of the file zlib:max:c32m+srep:m3f:a2+lolz... I downloaded DiskSpanGUI and all I understood for now is how to select the input game folder. How should I learn about game compression if there is no comprehensible clear guide for begginers, only some general info like "use XTool and compress your games", but not a single example about how to apply all that scary configs and scripts for a certain files. Please, if I missed something on this forum what potentially could've helped me - point me on that. |
#36
|
|||
|
|||
There is literally not a single comprehensive step-by-step instruction for beginners on this forum, all I see as a beginner is methods and troubleshooting discussions. Sorry for that, but as a beginner who spent 2 days on this forum I understood nothing, even though I really wanted. Thank you.
|
#37
|
||||
|
||||
@sailwhist
I will take your example for GTA4. Sometimes you already see what a game file could include just by file name or folder name. For example the following file Code:
GTAIV\pc\audio\sfx\ambient_streams.rpf I don't have the game so I have to check steamdb depot for this game and sort the file list by type. What I can see is that literally all *.rpf files contain sound/audio files, just by looking at the file name and/or folder where the files are located. Another example for this type is Code:
GTAIV\pc\audio\sfx\radio_*.rpf But because I don't have the game nor any of the files, I can just guess that the files contain riff-wave files, mp3 files or raw files or whatever else, because I know msc can handle them mostly. And the post creator you have quoted said "you can use msc but i didn't see improve ratio...", so I think my guess is correct here lol. Next one: movie files For bink movies (BIK or BK2) you can mostly use BinkPack (bpk), which will compress this files lossless. Just note that sometimes it will not work, either because the bink files using some sort of alpha channel, using a codec which bpk cannot handle correctly, or - which applies to BK2 files - have a newer codec which BinkPack cannot handle at all. If you have Bink video files, you can do the following:
Next one: zlib etc. Again back to steamdb depot for the game GTA4, I also see "*img" files, which are 6.91 GiB in size. I would guess that these files contain all sorts of other files and textures. So now, the post you quoted sais "rest of the file zlib:max:c32m+srep:m3f:a2+lolz... ", which is including this *.img files. In other words, it seems that these *.img files are compressed with either zlib or deflate I would say. Devs simply do this to save space. To be sure, you can follow this little guide (to be fair this is for oodle compression, but just ignoring oodle option and using zlib option isn't that complicated). We always want to extract all the data/streams in the game files if possible, in other words we want to decompress them (xtool/other tools) to have fully possibility of deduplication (srep) and compression (lolz/lzma/whatever else) in order to compress them even better than the original.
__________________
Haters gonna hate
|
The Following User Says Thank You to KaktoR For This Useful Post: | ||
sailwhist (20-05-2023) |
#38
|
|||
|
|||
Quote:
Talking about audio/movies - I'm beginning to understand where they stored, but I'm not sure how to find every single audio/movie file in game folder to make sure I compressed all of these files with the right method. Also, I don't understand how to choose exact settings for that method for gta4 (zlib and etc.). In diskspangui it only shows me presets for that but without numbers like in those guy method. There are a lot of settings in that software which scares me. I literally don't know everything and I would like to see a detailed guide with an example on some game about how to apply all of these so me and others could use that as a complete FAQ on compressing games. Thank you. |
#39
|
||||
|
||||
Ah, I forget to explain a small but important detail about all the numbers in the method "those guys" sometimes share.
First, you have to keep in mind that tools are changing over time. Old tools like "zlib:max:c32m" (which should be this tool or maybe this) are not used anymore nowadays, because some tools are "evolving" over time, new tools are comming and old ones are obsolete. So, the guy you quoted posted this method on 14-03-2018, which was over 5 years ago. On this time there was no "xtool". Means, methods and tools can (and most likely will) change over time. The method "zlib:max:c32m" belongs to an old tool. No one is using this tool anymore. "zlib" is the codec, ":max" is some other option I don't remember (I guess it has something to do with stream detection) and ":c32m" is the chunk size used for this tool. Basically you have to "transfer old methods into new methods". So the old method "zlib:max:c32m" is something like "xtool:c32mb:mzlib" in diskspangui. Edit: Again I forgot to tell you "how to use this method on GTA4 for your needs" lol, sorry. Personally I do all of this configuration stuff through the INI file "Resources\DSG_Settings.ini", just because I find it more easier. So for your example of GTA4 it could look like this: Code:
[Game1Settings] Name=Grand Theft Auto 4 Exec= EGPath=C:\Program Files (x86)\Rockstar Games\Grand Theft Auto 4 MethodState=0 Data1.Name=Data<game><data>-01.bin Data1.Method1=msc|0 Data1.InclText=*.rpf Data2.Name=Data<game><data>-01.bin Data2.Method1=xtool:mzlib+srep+lolz|0 Data2.InclText=*.img Data3.Name=Data<game><data>-01.bin Data3.Method1=bpk|0 Data3.InclText=*.bik Data4.Name=Data<game><data>-01.bin Data4.Method1=srep+lolz|0 Data4.ExclText=*.img|*.rpf|*.bik Data2 includes only *.img files, which probably using some compression like zlib or deflate, so you have to use xtool here. If xtool:mzlib doesn't work, try with another codec like xtool:mpreflate or xtool:mreflate. Data3 includes only *.bik files. I guess it ok to use bpk method only. Data4 includes all other files, excluding *.img, *.rpf and *.bik, so you can just use srep+lolz if you want. |0 indicates that this method belongs to freearc, while |1 is 7-zip (.7z) and |2 is RAR (.rar) (freearc methods cannot be used with |1 or |2). I guess using |1 or |2 are special and absolutely rare cases of some sorts. InclText and ExclText telling diskspangui/freearc, that only given file types/files with paths should be included in given archive, or exclude them of the given archive. In the example above we have 3 archives which includes specific file types. So we need a last archive which includes all other files except the 3 previous archives, otherwise we would missing all other files on installation which are not exclusively included in other archives. Makes sense, does it? PS: In InclText and ExclText you can (or have to) separate file types/files with paths with a | character. <game><data> are just placeholders or variables. <game> is actually [Game#Settings] (it takes the number # from [Game1Settings] section, so 1), <data> is actually Data#, which takes the number # from Data#. However you can name the archives to whatever you want. Just make sure to not use any super special characters.
__________________
Haters gonna hate
Last edited by KaktoR; 20-05-2023 at 10:17. |
The Following 2 Users Say Thank You to KaktoR For This Useful Post: | ||
Enluaphelis (05-07-2023), sailwhist (20-05-2023) |
#40
|
|||
|
|||
FitGirl: Thanks from reply!
Two MT mode? Or MTT only?? LDMF its supporting multithreaded compression, with lower ratio! Special commands trick to usage. https://i.kek.sh/bNhofYWbNzw.png https://i.kek.sh/2udWoqik8Ut.png Xeon 16 threads with 16GB RAM from LOLZ --> Use 128MB dictionary size. Memory its fully loaded. srep:m3f:a0:l512+LOLZ:mtt1:mt2/4+ worser ratio than srep:m3f:a0:l512+LOLZ:ldmf1 mode. |
#41
|
|||
|
|||
Quote:
Also I don't understand how to pack it all in installer after compression, I wouldn't mind a detailed explanation of how to create an installer after compressing all the files Thank you |
#42
|
||||
|
||||
lolz compressor is slow, yes. However you can speed up the process by using some options for it, for example utilize all your cpu cores.
Example Code:
lolz:mtt1:mt12 As for the installer, you have several choices here. There are some installers which are compatible with diskspangui. You can find supported installers in the diskspangui topic, I think it is the second or third post in the topic. However you can also use diskspangui itself to unpack the archives. Just copy diskspan_gui.exe in the same folder where all the created archives are and run it.
__________________
Haters gonna hate
Last edited by KaktoR; 21-05-2023 at 00:19. |
The Following User Says Thank You to KaktoR For This Useful Post: | ||
sailwhist (21-05-2023) |
#43
|
|||
|
|||
Quote:
I don't think 24 GB game should be compressing to a 13 GB for 10+ hours |
#44
|
||||
|
||||
It will look like this in the settings:
Code:
Data2.Method1=xtool:mzlib+srep+lolz:mtt1:mt12|0 https://i.imgur.com/xMp4E29.png * All possible options for the lolz compressor can be found here: "Resources\FINAL\LOLZ\Options.txt" Time for compression mostly depends on your hardware, especially how fast your cpu and ram is, and of course of the input data and used settings. So there is no way to telling you how long compression will take if no one write the time and his pc specs.
__________________
Haters gonna hate
|
The Following User Says Thank You to KaktoR For This Useful Post: | ||
sailwhist (21-05-2023) |
#45
|
|||
|
|||
Quote:
Thank you, I hope it'll work. BTW, what's wrong with Imgur? It's not opening any links, just shows it like that: https://pasteboard.co/rE3HCJIzwHaM.png |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
newbie question | bigorangekitty | PS2 Games | 5 | 15-05-2002 17:51 |
Total newbie...dumb question...patience please! | xxxMikexxx | PC Games | 1 | 03-04-2002 00:19 |
newbie question | troi22 | PS2 Games | 1 | 28-02-2002 04:18 |
newbie PAL patch question for Pro Evolution | Purple Cowboy | PS2 Games | 2 | 19-02-2002 13:39 |
Newbie question...how do I know if the game I've downloaded is Autoboot or not? | konichiwa | DC Games | 1 | 23-11-2000 21:35 |