FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

L33THAK0R 23-09-2022 02:34

Does anyone know if theres a way to bulk apply/insert a hexadecimal pattern into a given chunk of data if given a start and end offset? I've got some projects I'm working on that require a few hundred blanking/0-filling operations and its awfully slow to do so with HxD's GUI. I've gone through the SFK documentation a half dozen times and alas to no avail. My current solution is just to have a massive blank file and drawn on the given offsets from it but this is obviously not very practical and is a pain having to increase its size should I be attempting to patch out a chunk from a larger archive in the future.

KaktoR 23-09-2022 02:43

Try with xtool's erase/replace command

dixen 23-09-2022 05:34

Pathfinder: Kingmaker (GOG Version) [New repack]

Bundle Folder (excluded manifest files)
XTool v0.65 (unity+lz4 plugin v1.75)+srep+lolz
21 gb > 50 gb > 25 gb > 7.2 gb

Sound files (*.wem & *.bnk)
srep+lzma2
3.2 gb > 2.6 gb

*.assets *.resS & extensionless files
srep+lolz
13.4 gb > 2.8 gb

Rest files
400 mb > 65 mb

Repack's size - 12.7 gb
Install time - 29 min. (R5 3600)

shazzla 24-09-2022 10:44

Quote:

Originally Posted by L33THAK0R (Post 498404)
Ah didnt realise I had to specify lz4f, many thanks, also what tool are you using to get that console output if you dont mind me asking?

Hi Kaktor and everybody !

What is this tool in the screenshot ? And what parameter was used?
Im out nearly 2 years from repacking!
Btw anyone can share me an lz4 library collection for xtool?
Many thanks! :)

KaktoR 24-09-2022 10:51

1 Attachment(s)
Quote:

Originally Posted by shazzla (Post 498466)
Hi Kaktor and everybody !

What is this tool in the screenshot ? And what parameter was used?
Im out nearly 2 years from repacking!
Btw anyone can share me an lz4 library collection for xtool?
Many thanks! :)

xtool --verbose mode

Attached file contains some lz4 libraries

hwang4 24-09-2022 12:35

Ebola 3
 
Which is best compression method ?

Masquerade 24-09-2022 13:55

Quote:

Originally Posted by hwang4 (Post 498474)
Which is best compression method ?

Probably zlib used on game PAK file so use xtool+srep+lolz, srep on mp4 videos. Haven't downloaded this game so cannot confirm if zlib is used or not. Check with FModel

hwang4 24-09-2022 14:07

tried this method without success ! it doesn't contains zlibs!

Masquerade 25-09-2022 00:47

Quote:

Originally Posted by hwang4 (Post 498476)
tried this method without success ! it doesn't contains zlibs!

... so check with FModel like I said, it will tell you what algo is used.

L33THAK0R 26-09-2022 01:36

Are there any other LZ4 libraries outside of the ones in the GitHub releases that can yield any half decent results/are there any reliable means to pick up dropped LZ4 streams? In a previous post here I was inquiring on some samples including an LZ4 one, and whilst the level 2 detection processes a number of streams, the total is still fairly low (~65/700) with the final result only being a ~110% inflation. I know not all streams are created equal and some are larger/smaller than others but I'm curious if processing the other dropped streams could result in a more significant result.

Dragonis40 26-09-2022 02:59

Good morning! I'd like to replace "Ultraarc v2.9.0.0 R9" with a simple .bat file, but the issue (for me) is I don't know how to merge the various .tmp file generated from compression into a final .bin file.
The reason why I'm asking this is because, sometimes, final .bin files decompression (whose setup.exe is generated using Inno Setup) generates an error (unarc.dll -1 error). If I decompress final .bin file with a simple .bat file, I have no unarc.dll error.
Thank you for your reply!

Masquerade 26-09-2022 04:56

Quote:

Originally Posted by Dragonis40 (Post 498496)
Good morning! I'd like to replace "Ultraarc v2.9.0.0 R9" with a simple .bat file, but the issue (for me) is I don't know how to merge the various .tmp file generated from compression into a final .bin file.
The reason why I'm asking this is because, sometimes, final .bin files decompression (whose setup.exe is generated using Inno Setup) generates an error (unarc.dll -1 error). If I decompress final .bin file with a simple .bat file, I have no unarc.dll error.
Thank you for your reply!

The tmp files are only used when Freearc compression is running. FA should automatically take tmp file from the finished operation and put it into the archive

Dragonis40 26-09-2022 06:17

Quote:

Originally Posted by Masquerade (Post 498497)
The tmp files are only used when Freearc compression is running. FA should automatically take tmp file from the finished operation and put it into the archive

Thank you for your reply, but how can I use masks using batch file? Let me better explain: when I compress a batch of files using a specific mask (e.g. .bik or .bk2 files compressed with MASK=BPK), FA creates a .tmp file. What I'd like to do is add more .tmp files (created by the compression of other files using specific masks) to the one previously created and, when all files have been compacted, "close" the compression creating a final .bin file.

KaktoR 26-09-2022 08:53

-m<default method>/$<mask1>=<method>/$<mask2>=<method> ...

arc.groups is required

Tbh I don't understand what your question is about at all. It is like Masquerade already said, FA will create tmp files with every compression step and collect them together on final stage to create a readable archive then.

dixen 27-09-2022 03:31

Project CARS 2 (7.1)

*.bff - XTool v053 (BMS DB)+srep+lolz
45.6 gb > 102 gb > 46 gb > 17.2 gb

*.bik - bpk
2.6 gb > 1.9 gb

*.bank - srep+lzma2
1 gb > 500 mb

Rest files - srep+lolz
1 gb > 129 mb

RePack's size - 20 gb
Install time - 30 min (R5 3600)

BMS Database - here

Note.
Also need oo2core_4_Win64.dll from Game folder.

hwang4 27-09-2022 04:20

Ebola 3
 
Masquerade , checked with fmod , this game uses "oodles"
but need some help , i need to know which version of oo2core.dll , i must use?
I tried (4,5,6,7,8,9) without good ratio!
thx!:)

Masquerade 27-09-2022 05:07

Quote:

Originally Posted by hwang4 (Post 498530)
Masquerade , checked with fmod , this game uses "oodles"
but need some help , i need to know which version of oo2core.dll , i must use?
I tried (4,5,6,7,8,9) without good ratio!
thx!:)

Please may you cut the first 100mb of the file and upload it? thanks.

Dragonis40 29-09-2022 06:42

Quote:

Originally Posted by KaktoR (Post 498502)
-m<default method>/$<mask1>=<method>/$<mask2>=<method> ...

arc.groups is required

Tbh I don't understand what your question is about at all. It is like Masquerade already said, FA will create tmp files with every compression step and collect them together on final stage to create a readable archive then.

This is what I was looking for: how to concatenate several masks in a batch file! Thank you so much!
One more question: is there any way to speed up decompression? I've tried XLolz, which is very faster in compression rather than lolz, but decompression is slower and compression ratio is not as good as lolz.
Any suggestions? Thank you.

Masquerade 29-09-2022 08:43

Quote:

Originally Posted by Dragonis40 (Post 498554)
This is what I was looking for: how to concatenate several masks in a batch file! Thank you so much!
One more question: is there any way to speed up decompression? I've tried XLolz, which is very faster in compression rather than lolz, but decompression is slower and compression ratio is not as good as lolz.
Any suggestions? Thank you.

Don't use XLolz - there's absolutely no need for it to exist. Lolz decompression speed through cls-lolz plugin is imensely quick and already much faster to the point where your limiting factor is the speed of your storage medium, not the algo.

Your decompression speed will be limited by any precompressor you have used and what algo it's trying to recompress. I think you were talking about CyberPunk 2077, right? That game uses Oodle Kraken compression and XTool is working it's ass off to compress that for you as quick as possible.

KaktoR 29-09-2022 10:58

Kena Bridge of Spirits
pak zlib

bk2 bpk doesn't work (kb2n)

Dragonis40 29-09-2022 14:19

Quote:

Originally Posted by Masquerade (Post 498555)
Don't use XLolz - there's absolutely no need for it to exist. Lolz decompression speed through cls-lolz plugin is imensely quick and already much faster to the point where your limiting factor is the speed of your storage medium, not the algo.

Your decompression speed will be limited by any precompressor you have used and what algo it's trying to recompress. I think you were talking about CyberPunk 2077, right? That game uses Oodle Kraken compression and XTool is working it's ass off to compress that for you as quick as possible.

I'm talking about Cyberpunk 2077. I've used Oodle Kraken (the one you can find into bin\x64 folder) to compress most of .archive files. I've also noticed two of them can be decompressed and files inside can be compressed with other methods.
Of course I've used Oodle Kraken file for decompression as well.
For decompression, I've used cls-srep, cls-lolz, xtool_2020, precomp and something else to compress decompressed .archive files.
I'm still unexperienced in compressing and decompressing world, but how can Fitgirl version be 200 mb smaller then my final compressed .bin file and taking almost 20 minutes less (installation time) then mine?

Masquerade 29-09-2022 14:32

OK now you've completely lost me. The archives are already compressed with kraken. You use the precompressor (XTool) on them. There's absolutely no need to use precomp here. It's not needed.

Dragonis40 29-09-2022 15:07

Quote:

Originally Posted by Masquerade (Post 498561)
OK now you've completely lost me. The archives are already compressed with kraken. You use the precompressor (XTool) on them. There's absolutely no need to use precomp here. It's not needed.

Let me better explain: I've used precomp only for "Bonus content" files, not for .archive ones. Two .archive files, once decompressed, give better compression result rather than original .archive files. That's the reason why I've decompressed two .archive files.

Masquerade 29-09-2022 15:12

Brewmaster: Beer Brewing Simulator

Code:

xzlib+srep+lolz/$jpg=brunsli
  • 3.2GB ---> 1.5GB

L33THAK0R 29-09-2022 19:19

Whats the best method for processing unencrypted UE4 ".pak" files? I've tried just applying xtool's zlib+reflate / oodle codecs but whilst the streams are picked up I always have issues decompressing, with the archives being reported as corrupted. I used to just process the pak's by unpacking and compressing the loose files but with larger titles this isn't very efficient for the user's disk I/O and with some titles I'm investigating currently (such as "Phantom Doctrine"), seemingly random files fail to decompress, pushing me to investigate again into packing the compressed ".pak" files as they're originally presented instead.

L33THAK0R 29-09-2022 19:28

Also quick second question, has anyone else had issues with using AFR on AnvilEngine 1.0 titles (or whatever the engine was named prior to AnvilNext)? I'm getting some ".forge" titles being processed with a warning at the end of processing them with AFR: "WARNING! In file found corrupted LZO-streams. Check your data!" and I have to manually test each files or a batch of files to determine which produce this error. packing the files that produce this error separately to the ones that don't results in a consistent successful decompression however the size is inflated due to them not being de-duplicated together. Currently affected titles that I've noticed are "Assassin's Creed" "III", "Black Flag", "Freedom Cry" & "Rogue". I'll note that I've been using AFR v0.0.1.9, using the "-vr" parameter as well as Srep v0.392, since other versions have produced issues with the "*.forge" files from my experience.

KaktoR 30-09-2022 03:40

Quote:

Originally Posted by L33THAK0R (Post 498567)
Also quick second question, has anyone else had issues with using AFR on AnvilEngine 1.0 titles (or whatever the engine was named prior to AnvilNext)? I'm getting some ".forge" titles being processed with a warning at the end of processing them with AFR: "WARNING! In file found corrupted LZO-streams. Check your data!" and I have to manually test each files or a batch of files to determine which produce this error. packing the files that produce this error separately to the ones that don't results in a consistent successful decompression however the size is inflated due to them not being de-duplicated together. Currently affected titles that I've noticed are "Assassin's Creed" "III", "Black Flag", "Freedom Cry" & "Rogue". I'll note that I've been using AFR v0.0.1.9, using the "-vr" parameter as well as Srep v0.392, since other versions have produced issues with the "*.forge" files from my experience.

I don't know where your "problem" is exactly now. You can ignore the AFR message about corrputed LZO-Streams. This just means that some streams aren't compressed with lzo, use another codec or lzo version or whatever else. Just ignore it.

KaktoR 30-09-2022 03:42

Quote:

Originally Posted by L33THAK0R (Post 498566)
Whats the best method for processing unencrypted UE4 ".pak" files? I've tried just applying xtool's zlib+reflate / oodle codecs but whilst the streams are picked up I always have issues decompressing, with the archives being reported as corrupted. I used to just process the pak's by unpacking and compressing the loose files but with larger titles this isn't very efficient for the user's disk I/O and with some titles I'm investigating currently (such as "Phantom Doctrine"), seemingly random files fail to decompress, pushing me to investigate again into packing the compressed ".pak" files as they're originally presented instead.

I took your example "Phantom Doctrine" and it works fine with xtool+srep+lzma, so... ?

L33THAK0R 30-09-2022 08:02

Quote:

Originally Posted by KaktoR (Post 498569)
I took your example "Phantom Doctrine" and it works fine with xtool+srep+lzma, so... ?

Ah cheers, maybe it's lolz then thats acting up, I'll definitely have to do a bit of testing I reckon, thanks for testing it on your end!

Quote:

Originally Posted by KaktoR (Post 498569)
I don't know where your "problem" is exactly now. You can ignore the AFR message about corrputed LZO-Streams. This just means that some streams aren't compressed with lzo, use another codec or lzo version or whatever else. Just ignore it.

Ah alright, I'll definitely give recompressing with another codec a shot, since that might be whats messing things up on my end, honestly cant think of what else it could be considering I'm getting decompression issues when just packing with afr+srep (to test).

KaktoR 30-09-2022 08:05

There is anvil plugin for xtool, you should give it a try.

hwang4 30-09-2022 13:44

Ebola 3
 
1 Attachment(s)
Masquerade , thank you for your help!:)

Finally , I found the right oo2core.dll!

oo2reck (7) + srep+ lolz

23gb ------> 16.6gb

KaktoR 01-10-2022 04:59

Pathfinder Wrath of the Righteous
Enhanced Edition

Code:

10:58:21 - Selected ARC/DS method for Data1a-01.bin was: xtool:munity,lz4hc,l12:lz4_193+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 10:58:21 - Selected ARC/DS method for Data1b-01.bin was: srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 10:58:22 - Compression initialized in 01.10.2022 at 10:58:22.
 10:58:23 - Starting Data1a-01.bin archive creation.
 12:46:03 - Moving Data1a-01.bin file to media folder.
 12:46:04 - Starting Data1b-01.bin archive creation.
 12:59:09 - Moving Data1b-01.bin file to media folder.
 12:59:10 - Compression finished in 01.10.2022 at 12:59:10.
-------------------------------------------------------------------------------------------------------------------------------------------
 12:59:10 - Getting size of the game: Pathfinder - Wrath of the Righteous
 12:59:10 - Installation folder size: 37.88 GB
 12:59:10 - Data1a-01.bin file size: 14.17 GB
 12:59:10 - Data1b-01.bin file size: 3.85 GB
 12:59:10 - Compressed game size: 18.02 GB (Ratio 47.58%)
-------------------------------------------------------------------------------------------------------------------------------------------
 12:59:10 - Overall input size: 37.88 GB
 12:59:10 - Overall output size: 18.02 GB (Ratio 47.58%)
 12:59:10 - Overall conversion time: 02:00:38


KaktoR 01-10-2022 08:02

Command & Conquer Remastered Collection

I made this as a test and showcase with use of quickbms + bink patching. In this case use of quickbms is possible because game files are not compressed (or at least not seem).

Steps:
  1. Make Backup of original archives (Data\movies_*.meg)
  2. Unpack original archives (Data\movies_*.meg) with quickbms
    Code:

    for %%f in ("MOVIES_*.MEG") do quickbms_4gb_files -d -o -Y petroglyph.bms "%%f"
  3. Compress binks and make patches (bink_pack.exe, bink_unpack.exe, hdiffz.exe, hpatchz.exe)
    There are some bk2 files which even bink_unpack.exe has problems with, so patch size is same as original size (416mb diff size). Take care of them.
  4. Null unpacked bk2 files (fill contents with zeros)
  5. Reimport nulled bk2 files with quickbms into original archives (petroglyph.bms)
    Code:

    quickbms_4gb_files -w -r petroglyph.bms "input .meg file" "input folder containing nulled bk2 files"
    Before reimporting you have to separate all the files in the unpacked folders, you only need bk2 files for reimporting
    Tbh I don't know if quickbms has a option to just import defined file types lol
    Remove Data\movies_*.meg.bak files if there are any (quickbms creates these files I guess)
  6. Delete all (nulled) bk2 files, so you have just compressed binks and patch files + nulled .meg archives left
    These nulled .meg archives have a compression ratio by nearly 0%, so don't mind the size of these (14.8 GB -> some KB)
  7. Compress the game
  8. Include required files in Installer or game folder (before compress the game)
  9. Let your Installer run a batch file to do it's job after extracting all files (example batch)

Installation/batch stuff will take some time, .meg archives are crc perfect at the end.

Results:
Code:

15:11:02 - Selected ARC/DS method for Data1a-01.bin was: srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 15:35:12 - Overall input size: 36.90 GB
 15:35:12 - Overall output size: 15.67 GB (Ratio 42.47%)
 15:35:12 - Overall conversion time: 00:24:08

Results without quickbms + bink patching stuff:
https://fileforums.com/showpost.php?...postcount=1772

Wanterlude 02-10-2022 16:42

Terra Invicta (v.0.3.18)

Code:

XTool (Unity plugin + liblz4 1.9.3)+srep+lolz

Compressed 979 files, 20,418,359,013 => 7,908,853,159 bytes. Ratio 38.73%
Compression time: cpu 149.16 sec/real 9259.21 sec = 2%. Speed 2.21 mB/s
Overall conversion time: 02:34:21

19.02 GB => 7.37 GB


angeldarkRS 02-10-2022 20:33

Quote:

Originally Posted by Wanterlude (Post 498587)
Terra Invicta (v.0.3.18)

Code:

XTool (Unity plugin + liblz4 1.9.3)+srep+lolz

Compressed 979 files, 20,418,359,013 => 7,908,853,159 bytes. Ratio 38.73%
Compression time: cpu 149.16 sec/real 9259.21 sec = 2%. Speed 2.21 mB/s
Overall conversion time: 02:34:21

19.02 GB => 7.37 GB


What specs your PC/Laptop?

Wanterlude 02-10-2022 20:36

Quote:

Originally Posted by angeldarkRS (Post 498588)
What specs your PC/Laptop?

11th Gen Intel(R) Core(TM) i5-11400F
16,0 GB RAM
M.2 SSD

L33THAK0R 03-10-2022 19:11

Quote:

Originally Posted by L33THAK0R (Post 498567)
Also quick second question, has anyone else had issues with using AFR on AnvilEngine 1.0 titles (or whatever the engine was named prior to AnvilNext)? I'm getting some ".forge" titles being processed with a warning at the end of processing them with AFR: "WARNING! In file found corrupted LZO-streams. Check your data!" and I have to manually test each files or a batch of files to determine which produce this error. packing the files that produce this error separately to the ones that don't results in a consistent successful decompression however the size is inflated due to them not being de-duplicated together. Currently affected titles that I've noticed are "Assassin's Creed" "III", "Black Flag", "Freedom Cry" & "Rogue". I'll note that I've been using AFR v0.0.1.9, using the "-vr" parameter as well as Srep v0.392, since other versions have produced issues with the "*.forge" files from my experience.

For anyone interested or who's encountered similar issues. I found out the cause of my troubles with both this issue and the immediate post prior to this one. Recently I streamlined and revamped my whole pipeline for packing titles and in the process accidentally included an outdated version of Srep somehow, using an updated version has fixed all my issues thankfully.

L33THAK0R 04-10-2022 22:00

Sleeping Dogs Update
 
Quote:

Originally Posted by L33THAK0R (Post 496898)
Hi all,

Been looking into Sleeping Dogs, and after a bit of discussion with some more experienced/technical tool engineers, I've got to say I'm quite intrigued. It uses a very aggressive "LZ" algorithm, with a given ".BIG" (archive format for the title in question) decompressing to over twice its original size! It also seems the WWise ".pck" files used for audio within the title have the potential for additional gains using "OGGRE"/"MSC". Unfortunately for the former I don't possess the skill-set (yet) to adapt the provided compression/decompression algorithm to be able to parse inputs for use with a chain of compressors, but none the less I think its still interesting enough for others to see.

Attached is a tool suite for processing the aforementioned ".BIG" archives, as well as a sample archive.

Came back to this one after a bit of a break and after learning about xtool's "verbose" mode, turns out LZO is used on the archives, however it seems that no matter what level depth setting or library I use no streams are able to be processed, does anyone know the reason behind this? Attached is an image supporting my findings.https://i.imgur.com/8XrV9Jy.png

kuyhaa 05-10-2022 01:19

hi, how to TEXTURE Rebuild and compression .dds file use "crunch" ?

KaktoR 05-10-2022 05:21

Dakar Desert Rally

bk2 (kb2n) = no bpk
pak = kraken (oo2core 7)

Code:

Streams: 17099/17099
Time: 00:12:38 (00:12:36)
Memory: 245 MB (245 MB)

Compressed 1 file, 2,125,021,534 => 4,260,018,174 bytes. Ratio 200.47%
Compression time: cpu 1.53 sec/real 125.97 sec = 1%. Speed 16.87 mB/s
All OK



All times are GMT -7. The time now is 06:31.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com