FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

Masquerade 18-05-2021 13:38

Peter72
Chain commands together using the -m switch in freearc:

-msrep+lolz

example

Peter72 18-05-2021 15:27

Quote:

Originally Posted by Masquerade (Post 492074)
Peter72
Chain commands together using the -m switch in freearc:

-msrep+lolz

example

Thanks for your reply. I added the "-m" before the compression line, but I think I don't get it correctly. So here is my code:

arc.exe a -ep1 -r -ed -s; -w.\temp\setup-3.bin srep:m3f+lzma:176m:normal:bt4:273:mc1000000000:lc8 "D:\Games\Immortals Fenyx Rising\PCK\*"
pause

I have divided the game files, and I am attempting to compress just the .pck files. If it write -msrep+lzma.... , it doesn't compile; it says (Unsupported compression method or error in parameters: srep:m3f). But when I remove it to just "srep+.." it starts compressing. But the thing now that it stops after compressing, leaves me with the .tmp file. It looks like this:

{
D:\Setup Files\oo2recm>arc.exe a -ep1 -r -ed -s; -w.\temp\setup-3.bin srep:m3f+lzma:176m:normal:bt4:273:mc1000000000:lc8 "D:\Games\Immortals Fenyx Rising\PCK\*"
FreeArc 0.67 (March 15 2014) creating archive: srep:m3f+lzma:176m:normal:bt4:273:mc1000000000:lc8 .arc
Compressed 21 files, 1,796,076,026 => 1,685,152,333 bytes. Ratio 93.82%
Compression time: cpu 124.30 sec/real 26.31 sec = 472%. Speed 68.27 mB/s
Copying temporary archive freearc1.tmp to srep:m3f+lzma:176m:normal:bt4:273:mc1000000000:lc8 .arc
Keeping temporary archive freearc1.tmparc.exe: srep:m3f+lzma:176m:normal:bt4:273:mc1000000000:lc8 .arc: open: invalid argument (Invalid argument)

D:\Setup Files\oo2recm>pause
Press any key to continue . . .
}

Keeping in mind that I don't know the use of those commands "a -ep1 -r -ed -s;". So what is the problem here? What am I not understanding or doing wrong?

:( Sad8669 18-05-2021 19:22

Peter72

you need :

Code:

arc.ini - Set methods here
arc.exe - Execute compression
pack folder - where you need to keep your files that will be compressed
temp folder - where temporary data will be kept by freearc during compression
and the compressors - the compressor itself along with required libraries.

all in the same folder.

https://i.imgur.com/Kf5YQ80.png

Peter72 18-05-2021 19:57

Okay... I downloaded XTool, copied all the binaries and libraries into the folder, and added the above code in the pack.bat file. Everything is still the same. I am still stuck with a 1.6 GB .tmp file as in the image. Correct me if I am wrong, there are some coding that should be written in the arc.ini file. If so, I have no idea about it? What to do now?https://i.imgur.com/9l0knL3.png

:( Sad8669 18-05-2021 20:28

Don't start repacking IFR like games right off the bat.

lzma uses FAZip32.exe which is not present in your folder.
lzma needs to be set in arc.ini before it can be used.

open your arc.ini:

Code:

[External Compressor:X]
header = 0
packcmd = X.exe e {options} <stdin> <stdout>

a basic structure is like this. You can't learn it all in one day, just keeping compressing games until you evolve and don't use commands as a beginner, use something like UltraArc or DiskSpan as a beginner.

Masquerade 18-05-2021 22:49

Peter72
You set your temp folder as setup3.bin instead of the output archive.

:( Sad8669 19-05-2021 07:55

Star Wars : Squadrons
 
Code:

Method : xtool_zstd+srep:m3f:l512+lolz:dtb1:d128:mtt0:mt4:mc1023
Compression Time : 20 Hours
Decompression Time : 1 Hour 39 Minutes
Initial Size : 31.5 GB
Final Size : 20.5 GB

Possibilities : You can rip Multiplayer files if you don't need them.

Peter72 19-05-2021 10:33

Okay... I will slow it down. But the problem is that there is no beginner guide in the whole process. So when I stumble upon something for a beginner, I try it out. But I have no clue what are the things to learn sequentially. If you can direct me to the things to learn, I will be so grateful. So for example,

"You need to learn
1) X, then (Links to 1)
2) Y, then alternate your methods to (Links to 2)
3) Z (coding for instance) (Links to 3)"

Maybe one day, I will make a beginner book.

:( Sad8669 20-05-2021 00:18

Mini Car Racing - Tiny Split Screen Tournament
 
Code:

Method : ZTool+SREP+LOLZ
Compression Time : 49 Minutes
Decompression Time : 34 Seconds
Initial Size : 2.06 GB
Final Size : 284 MB

Side Notes : I don't know if the game contains any lz4 streams and by the time i was compressing this game, i didn't realized that there was a Unity precompressor plugin for XTool.

L33THAK0R 20-05-2021 02:07

Quote:

Originally Posted by Peter72 (Post 492106)
Okay... I will slow it down. But the problem is that there is no beginner guide in the whole process. So when I stumble upon something for a beginner, I try it out. But I have no clue what are the things to learn sequentially. If you can direct me to the things to learn, I will be so grateful. So for example,

"You need to learn
1) X, then (Links to 1)
2) Y, then alternate your methods to (Links to 2)
3) Z (coding for instance) (Links to 3)"

Maybe one day, I will make a beginner book.

I'd recommend reading the pinned posts on this board, plus giving as much information as possible, when posting about any issues or queries. These 2 things have helped me immensely with learning about this hobby/field.

:( Sad8669 20-05-2021 02:25

I agree, just start reading this thread from the first page, the problems solved in the past might help you start.

Masquerade 20-05-2021 11:02

Peter72
A nice place to start is, if you like the steps order:

1. take a look at freearc's command line and see what all your various switches do. Using a custom temp folder is hugely useful if your OS is on an SSD (-w switch). -m switch is for the compression method.

2. Look into using arc.ini to add external compressors. Start with something like XTool or srep. There's plenty of premade compressors in this forum which already have packcmds in the arc.ini, and you can use these to learn the correct structure.

3. Learn about using multiple compressors chained together using the + sign in -m switch, e.g. -msrep+lolz

4. Learn to use multiple methods on 1 archive by using arc.groups to make a mask ( I won't type all this in 1 message, but I will happily explain masks you are interested - just this way I save you some time by looking at building blocks first)

:( Sad8669 20-05-2021 11:25

1 Attachment(s)
When you have arc.exe in your folder, you can run a .bat file to learn about their parameters.

Copy below and save it as .bat and run it in the same folder as arc.exe.

Help.bat
Code:

arc.exe
pause

From here you can learn what each command does.

Peter72 20-05-2021 12:52

Thank you very much for these info. It very much useful. And to Masquerade, I will be asking you about the masks when I get there :)

Masquerade 20-05-2021 14:24

Quote:

Originally Posted by :( Sad8669 (Post 492245)
When you have arc.exe in your folder, you can run a .bat file to learn about their parameters.

Shift+right click to open command window from context menu? :confused:

HunterEmrah 20-05-2021 15:32

Can you suggest something good for Days Gone?

KaktoR 20-05-2021 20:36

Quote:

Originally Posted by HunterEmrah (Post 492248)
Can you suggest something good for Days Gone?

https://fileforums.com/showpost.php?...&postcount=101

Code:

21:43:42 - Selected ARC method for Data1.bin was: oo2reck7+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

 03:01:52 - Overall input size: 44.40 GB
 03:01:52 - Overall output size: 23.05 GB
 03:01:52 - Overall conversion time: 05:18:04


:( Sad8669 20-05-2021 21:42

Quote:

Originally Posted by Masquerade (Post 492247)
Shift+right click to open command window from context menu? :confused:

He is eventually going to use .bat files in future, taking long methods may help build more experience.

KaktoR 22-05-2021 12:47

Mass Effect Legendary Edition

Code:

12:08:32 - Selected ARC method for Data1a.bin was: bpk
 12:08:32 - Selected ARC method for Data1b.bin was: xtool_ue3_mele+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
------------------------------------------------------------------------------------
 18:18:58 - Overall input size: 103.74 GB
 18:18:58 - Overall output size: 60.84 GB
 18:18:58 - Overall conversion time: 06:10:21

Edit: Do not use bpk. You will get crc errors on unpacking.

L33THAK0R 22-05-2021 19:03

Quote:

Originally Posted by KaktoR (Post 492296)
Mass Effect Legendary Edition

Code:

Selected ARC method for Data1b.bin was: xtool_ue3_mele+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

Sorry if this is obvious, but what exactly is
Code:

xtool_ue3_mele
I'm guessing it's an xtool plugin specific to the title "Mass Effect Legendary Edition", but was just wondering what exactly it does.

Cesar82 22-05-2021 19:08

Quote:

Originally Posted by L33THAK0R (Post 492300)
Sorry if this is obvious, but what exactly is
Code:

xtool_ue3_mele
I'm guessing it's an xtool plugin specific to the title "Mass Effect Legendary Edition", but was just wondering what exactly it does.

Yes. It is the plugin shared by Razor12911 in this #post.
The information shared by KaktoR is information used in DiskSpan_GUI (so it used "xtool_ue3_mele" and "srep_new").

L33THAK0R 22-05-2021 23:01

Quote:

Originally Posted by Cesar82 (Post 492301)
Yes. It is the plugin shared by Razor12911 in this #post.
The information shared by KaktoR is information used in DiskSpan_GUI (so it used "xtool_ue3_mele" and "srep_new").

Thank you for this! I wonder why the developers of the title chose to use a different compression technique, (I'm guessing lzo & oodle) when the assets don't seem to have changed all that drastically (although I've only seen trailers for the title so I'm probably wrong)! Seems a bit strange.

Masquerade 22-05-2021 23:27

Quote:

Originally Posted by L33THAK0R (Post 492307)
Thank you for this! I wonder why the developers of the title chose to use a different compression technique, (I'm guessing lzo & oodle) when the assets don't seem to have changed all that drastically (although I've only seen trailers for the title so I'm probably wrong)! Seems a bit strange.

I think it's leviathan.

dixen 23-05-2021 01:12

Quote:

Originally Posted by Masquerade (Post 492308)
I think it's leviathan.

Yep.

Quote:

Universal Oodle stream scanner

Created by Razor12911

[0] = Unknown/Invalid
[1] = Kraken/Hydra
[2] = Mermaid/Selkie/Hydra
[3] = Leviathan/Hydra

0001[3]| Pos: 00000935, Size: 31674
0002[3]| Pos: 000084EF, Size: 37461
0003[3]| Pos: 00011744, Size: 126043
0004[3]| Pos: 0003039F, Size: 26440
0005[3]| Pos: 00036B0F, Size: 129688
0006[3]| Pos: 000565A7, Size: 83893
0007[3]| Pos: 0006AD5C, Size: 95555
0008[3]| Pos: 000822C7, Size: 106465

Rekulou 24-05-2021 13:12

Hello, Do you guys know what is the best way to compress The Binding of Issac Afterbirth? I've been having quite some trouble recently with it. Thanks

Masquerade 24-05-2021 14:41

Quote:

Originally Posted by Rekulou (Post 492356)
Hello, Do you guys know what is the best way to compress The Binding of Issac Afterbirth? I've been having quite some trouble recently with it. Thanks

Optimal method is to unpack the game archives, but currently the unpack tool has outdated file list leading to incorrect data unpacking and missing files meaning you;ll have a broken game.

Rekulou 25-05-2021 06:34

Quote:

Originally Posted by Masquerade (Post 492360)
Optimal method is to unpack the game archives, but currently the unpack tool has outdated file list leading to incorrect data unpacking and missing files meaning you;ll have a broken game.

Aw man, that sucks. What is the next best way to compress the game then?

Masquerade 26-05-2021 01:38

Does anyone know the size of It Takes Two under LZMA?

:( Sad8669 26-05-2021 07:47

Masquerade

I can post the results tomorrow if you would like, its already night here so i will let it run overnight.

Masquerade 26-05-2021 08:24

Quote:

Originally Posted by :( Sad8669 (Post 492418)
Masquerade

I can post the results tomorrow if you would like, its already night here so i will let it run overnight.

Awesome, thanks.

:( Sad8669 27-05-2021 02:59

Masquerade

Code:

Input size: 43.63 GB
Output size: 30.95 GB
Method : xtool_zlib+srep_new+lzma:a1:mfbt4:256m:fb273:mc1000000000:lc8

Compressing 5,611 files, 46,846,025,126 bytes. Processed  11.1%
Compressing 86,819,705,627 bytes with "OTHERS\SREP\Win64\srep.exe" -m3f -a0/0 InFile OutFile
 11.1%SREP 3.93a beta (October 11, 2014): input size 82797 mb, memory used 4922 mb, -m3f -l512 -c512 -a0/0 -hash=vmac -b8mb
100%: 86,819,705,627 -> 68,740,152,915: 79.18%.  Cpu 18 mb/s (4527.188 sec), real 17 mb/s (4785.019 sec) = 95%.  Remains 00 11.1%    %
Decompression memory is 1748 mb.  11,685,296 matches = 186,964,736 bytes = 0.27% o 11.1%
 11.1%
Errorlevel=0
Compressed 5,611 files, 46,846,025,126 => 33,235,367,739 bytes. Ratio 70.95%
Compression time: cpu 240.02 sec/real 47292.86 sec = 1%. Speed 0.99 mB/s
All OK

Side Notes : Please don't post this in the INDEX, i will shortly update it in another post.

:( Sad8669 27-05-2021 09:27

It Takes Two
 
Code:

Method : xtool_zlib+srep_new+lzma:a1:mfbt4:256m:fb273:mc1000000000:lc8
Compression Time : 13 hours 8 minutes 19 seconds.
Initial Size : 43.63 GB
Final Size : 30.95 GB


L33THAK0R 27-05-2021 22:13

Hi all,

I recently finally figured out how to properly encode and package .usm files, and I was wondering if anyone who has some experience here might be able to help me out with a little issue pertaining to implementing the new, encoded files. In the title "Batman: Arkham City" every encoded .usm, bar the initial file loaded "0_CW_Intro.usm", soft-locks the title at the conclusion of the video file, the only solution to this is manual user input to skip the cut-scene, which the title enables users to do for every pre-rendered cut-scene (all pre-rendered cut-scenes are contained within .usm files). This isn't a massive issue, as it can be solved by adding additional footage informing the user when to skip, however, I want to minimise this additional user-inputs if it is at all possible. I have multiple other titles I want to test out encoding .usm files on ("The Witcher 3: Wild Hunt", "Batman: Arkham Origins" & "Batman: Arkham Knight", to name a few), and as this is the first title I have attempted to encode .usm files on, I cannot say whether this is a title-specific issue or not.

My current plan is to stitch on a 10 second clip at the end of each effected video notifying the user their input is required, as I don't think there is any other solution.

:( Sad8669 28-05-2021 22:53

Days Gone - Precompression

XTool - Kraken is 50% faster than OodleRec - Kraken, while the difference is between their ratio is 5% - 6%

OodleRec - Kraken
Code:

Compressed 34 files, 35,821,723 => 75,484,606 bytes. Ratio 210.72%
Compression time: cpu 0.06 sec/real 32.63 sec = 0%. Speed 1.10 mB/s
All OK


XTool - Kraken
Code:

Compressed 34 files, 35,821,723 => 73,598,522 bytes. Ratio 205.46%
Compression time: cpu 0.06 sec/real 17.51 sec = 0%. Speed 2.05 mB/s
All OK

Side Notes : It's basically waiting for twice the XTool - Kraken and getting only 10% ~ 15 % ratio.

Final Compressors :

XTool:Kraken + SREP + LOLZ :
Code:

Compressed 34 files, 35,821,723 => 16,565,774 bytes. Ratio 46.25%
Compression time: cpu 0.06 sec/real 86.91 sec = 0%. Speed 0.41 mB/s
All OK

OodleRec:Kraken + SREP + LOLZ :
Code:

Compressed 34 files, 35,821,723 => 16,007,052 bytes. Ratio 44.69%
Compression time: cpu 0.17 sec/real 98.59 sec = 0%. Speed 0.36 mB/s
All OK

Side Notes : Squadrons Decompression time updated.

L33THAK0R 31-05-2021 19:20

Hi all,

Does anyone know if there are any public methods/tools to rip and restore, (however just knowing if it's possible to rip files would be appreciated) files from "id Tech Engine" data files? I'm particularly interested in the 5th and 6th iterations of the "id Tech" engine.

:( Sad8669 05-06-2021 21:35

Days Gone - Full Game Precompression

XTool : v0.3.9 Used.
Codec : Kraken.
Libraries : Provided by Razor12911 for Days Gone.

Results :
Code:

Compressed 3,092 files, 64,172,576,777 => 103,896,774,882 bytes. Ratio 161.90%
Compression time: cpu 108.59 sec/real 12011.73 sec = 1%. Speed 5.34 mB/s
All OK

Initial Size : 59.7 GB
Final Size : 96.7 GB

:( Sad8669 07-06-2021 00:37

Days Gone [All Languages][v1.0]
 
Code:

Method :xtool_kraken+srep:m3f:l512+lzma:a1:mfbt4:256m:fb273:mc1000000000:lc8
Compression Time : 10 Hours.
Decompression Time : 1 Hour 16 Minutes. (4T)
Initial Size : 59.7 GB.
Final Size : 36.9 GB.
Libraries : Both Oodle7 and Oodle8 Provided by Razor12911.
Possibilities : You can rip foreign languages.


doofoo24 08-06-2021 14:07

unravel 1+2
 
unravel 1 final size 2.04gb
3.17gb to 4.68gb using munravel1 -c64mb without --dedup
then srep+lolz from 4.68gb to 2.04gb nothing rip include trial exe
srep:m3f:d1g:a2+lolz:mt4:mtt1:d128:fba4096:mc1023: pc4:gm20:dtb1:x2

unravel 2 final size 3.5gb
6.76gb to 10.1gb using munravel2 -c32mb without --dedup
then srep+lolz from 10.1gb to 3.5gb...
srep:m3f:d1g:a2+lolz:mt4:mtt1:d128:fba4096:mc1023: pc4:gm20:dtb1:x2

using razor unravel_R1.7z with xtool 0.3.18

*note you can improve ratio by using mt0 and increase d but super slow...

doofoo24 09-06-2021 14:37

Mass Effect Legendary Edition 43.5gb
 
Mass Effect 1 8.7gb
Mass Effect 2 15.3gb
Mass Effect 3 19.7gb
only english
using razor unreal_R5...

method
mue3:m2+srep:m3f:d1g:a2:lolz:mt4:mtt1:d128:fba4096 :mc1023:pc4:dtb1

edit : compression with cls-bpk pass but give CRC error during decompression,
maybe because of multichannel audio ?
if you edit bik file it may work with cls-bik i remember that i tested it in the past when profargo released on number of games and some give the same error duing compression , need to tested...
ME MISTAKE FOR NOT TESTING CLS-BPK UNPACK, SORRY...

Masquerade 10-06-2021 08:29

Guys, you can't use bpk for the videos in Mass Effect - crash on decompression...


All times are GMT -7. The time now is 04:59.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2023, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com