Best Compression Methods for 'Specific' Games. Q&A

[rishikrap]

Originally Posted by Masquerade

uhhhh ok

1. There's no need for bpk since there's no bink videos in this game

2. Why are you using the cyberpunk 2077 precompressor on not cyberpunk 2077??????

3. xzlib - we're getting somewhere, but this game is made in unreal engine 3 and is compressed with lzo and not zlib (zlib is ue4) - which is why you need to use UELR [Unreal Engine LZO Recompressor]

4. srep, great!

5. lolz_mtx - this has been proven to be slower IIRC and not worth whatever result you get. If you want a fast compression, use lzma2.

I don't know.
I'm totally new to this like literally 100%.
Found a config, used CUI V3 GUI for diskspan to do this and it took 21 hrs to do this.
Now I don't know how to extract this things again so finding the forum.

Originally Posted by KaktoR

Your chain does not make any sense, just saying.

I know, just googled and found the best options.

[Masquerade]

Sorry if I sounded harsh, not the intention, please feel free to ask questions

[rishikrap]

Originally Posted by Masquerade

Sorry if I sounded harsh, not the intention, please feel free to ask questions

1. Could you help me out with what can I do to create an installer for this thing.
2. It'll install correct anyways regardless of all the unwanted methods right?
Update: I managed to find a Corepack UI installer which just needed me to change the name of the file in compier.ini so I did that and it proceeded, but it gives me a the error https://prnt.sc/w85cc2
Update 2: Managed to also make installer with the default lego template provided in CIU V3 and it did start and extracted some files and hung at 0.9% without utilizing any resources. I guess my compression technique sucked more than I expected.
So, I won't be able to install it in any way or is there some way I can save time and use what I compressed initally?
2 (A). If not, could you give me a single config that I can copy paste in Diskspan GUI by CIU V3.
3. Does SSD help with speeds at a good margin. I'm using HDD (not a good one either).
4. Also if this is too much hassle, can you give me a way to make an SFX using Ultraarc.
And by the way I used Cyberpunk 2077 compresser because it said in bracket Xtools 2020 and I thought this'd be the latest Xtools. And fitgirl says it uses Xtools library by Razor 1911 on the Rocket League Page, so I figured I should too.
I made one and it was easy install too, no issues at all, I didn't know what Xtools was that time, so I used default Ztools as precompressor, Lolz, Lzma2, Srep but it only compressed to 10.1gb, I don't know if X tools would improve the compression.
Only problem I have with Ultraarc sfx is it splits files too much even in a,b,c. Any way to use diskspan to make an sfx? And also achieve this conversion.
Hope you could solve all the doubts.

Update: I'm now using xlzo+uelr+srep_new+lzma2 to recompress, I hope I got it right this time.
I'll just need some proper tutorial on making an installer, that's the only thing that has eaten my brain. I don't care about the graphics or anything, I just want it to work ;__;.
My aim with this is simply to compress the Epic Games version maximum possible such that I have to download less. I'm doing this on an RDP (Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz 4 cores 8gb ram) which doesn't have bandwidth limitations.

[KaktoR]

Originally Posted by rishikrap

Update: I'm now using xlzo+uelr+srep_new+lzma2 to recompress, I hope I got it right this time.

No.

uelr+srep_new+lzma2

I think you have to work with file list to avoid errors with uelr (only include tfc, upk, ... files for uelr) if I'm not mistaken. Long time not worked with uelr to be fair. To achive that you have to open up config with _DiskSpan_GUI_CONFIG.bat file and go to the file lists. Make 2 file list files somewhere (1.txt, 2.txt). In 1.txt write all files which have lzo streams inside (mainly tfc, upk, I think). You can use wildcards (for example: *.tfc, each file type in a new line). In 2.txt write just a "*" (without ").




Save, close, start the GUI, select game folder, start.

You can also use a freearc archive with masking data compression, but I have no compatible files on hand atm. Maybe someone else can hep you otherwise.

------------------------------------------------

There are currently only 2 installer scripts comptaible, namely "Lite Installer" and "CIU beta 7". The diskspan GUI will create a "Setup.dll" file which contains all used settings to install the files.

Compression speed depends on some more things then drive speed. First of all CPU speed and input.

UltraArc make a SFX only if you check the box "Self-Extractor". Keep in mind that this way you have to done things a bit different and you have to use/edit a mask method inside Config0.ini and maybe change arc.groups file. PS: In next update UltraArc will be removed because you can do all of this things directly inside diskspan GUI (testing atm). I'm sure my friend Masquerade will tell you more on how to config mask groups and methods properly (otherwise this here will be a whole book soon and I'm not in the mood to write one).

Using xtool or any other precompressor depends on input you have. Unreal engine games which use lzo codec aren't compatible with xtool (or vise-versa) because they use some special codec (if I remember right).

If you want to avoid the split file in UltraArc, open UltraArc.ini and change

FirstVolumeSize=9999999999
CustomVolumeSize=9999999999

or something even bigger, don't matter.

If you want max compression use lolz instead of lzma2.

[rishikrap]

Originally Posted by KaktoR

No.

Yep, learned i the hard way. I slept after putting this and it got error after 2hr (Part of conversion log attached).
Only if I had stayed up for 6 more minutes.
Then I also tried

Originally Posted by KaktoR

 uelr+srep_new+lzma2

Replaced lzma2 with lolz_mtx. I hadn't seen the screenshot so again got the same LZO error.
Now, I have followed all that you have mentioned and started again.
Used

 uelr+srep_new+lolz_mtx for Data1

And

 srep_new+lolz_mtx

for Data2 with 2.txt as included and 1.txt as excluded.
Also, does it matter if I use lolz or lolz_mtx, I just used mtx so it utilized full cpu, don't know if it would make difference in conversion time since I want the best conversion. And there are other lolz conversions n, f etc do they matter? I'm using default lolz_mtx
Thanks for the detailed screenshots and all the other informatin else I'd have done some mistake for sure.
https://drive.google.com/drive/folde...TJA5Qhd34FZAlb
This was my inital output, is there any way, like any way in which I can just extract these without any error ;__;

 bpk+cp_2077(Xtools 2020)+xzlib+srep_new+lolz_mtx

I get the error unsupported config in every installer I tried, even Light as you mentioned and not CIU either.
Are those files now useles since I can't extract them, should I delete them and accept that I wasted 21hrs and RDP money ;___;

Update: I used the config as mentioned above and got the same error again
https://prnt.sc/w8brei
I'm now using xtools and not uel(lzo) since you said it could be vice versa too. Plus fitgirl page also mentiones xtools library Also, I have reset the config to default without and file lists or so.
Currently retying with

 xzlib+srep_new+lolz_mtx

If I get an error I'll replace xzlib with xlzo, since I don't know what'll work, and I had gotten success with xzlib earlier, hnce I'm trying xzlib first.

[Masquerade]

rishikrap
Using xzlib gives you no benefit, so there's no point to use it (UE3 does not use zlib). And then xlzo will not help either, since XTool cannot process these files IIRC (despite a .tfc file is given in example).

I'm downloading rocket league now from my Steam library, will give it a whirl then come back to you with some explanations on what to do.

[Masquerade]

OK so I'd like to start by apologising to you for denying the use of xtool zlib, this is the first UE3 game I've seen use zlib on tfc/upk archives:

SAMPLE - xzlib

Compressed 24 files, 1,926,995,396 => 7,144,830,307 bytes. Ratio 370.78%
Compression time: cpu 1.89 sec/real 94.48 sec = 2%. Speed 20.40 mB/s
All OK

With that established, I have tagged below a sample compressor that will allow you to make an archive compatible with the ASIS script (I recommend using compact mode)

How to use:

1. Download the below 7z, extract it
2. Put your rocket league game folder in to In folder
3. Run the bat
4. Inside the bat, type:
   

   xprecomp+srep+lolz/$wem=srep+pmt_razor

5. Wait until done!

Compression method explanation:

xprecomp = xtool, configured for zlib.

srep+lolz - srep to remove duplicates and lolz for main compressor. If you want to use lzma2, swap lolz for "7z".

What does $wem mean?

This is a mask, and masks are a way of using multiple compression methods in one archive.

Each mask (and the file types included by it) is included in arc.groups.

This allows the main data to be compressed with xtool,srep,lolz then the audio gets compressed with srep,razor(pmt).

PMT is a chunk based accelerator for slow compressors like mtx, but ASIS doesn't have MTX yet, so PMT is what to use.

Because PMT is chunk based, we run srep beforehand to catch duplicates.

You can define a mask using the arc.groups file, and the grouping must begin with $groupname:

Example:

$help
*.bin
*.blob

This means you can separate off bin and blob files using the /$help identifier in your method.

If that's a bit confusing, I'm sorry!

[KaktoR]

Anno 1800
v9.2, nothing ripped

 01:27:17 - Selected ARC method for Data1.bin was: srep_new+razor_mtx
 01:27:17 - Selected ARC method for Data2.bin was: xtool_zlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 01:27:19 - Compression initialized in 23.12.2020 at 01:27:19.
 01:27:19 - Starting Data1.bin archive creation.
 01:50:04 - Moving Data1.bin.001 file to media folder.
 01:50:05 - Starting Data2.bin archive creation.
 03:36:09 - Moving Data2.bin.001 file to media folder.
 03:36:11 - Compression finished in 23.12.2020 at 03:36:11.
--------------------------------------------------------------------------------------------------------------------------------------
 03:36:12 - Getting size of the game: Anno 1800
 03:36:12 - >> Installation folder size: 58.05 GB
 03:36:12 - >> Data1.bin file size: 2.67 GB
 03:36:12 - >> Data2.bin file size: 22.15 GB
 03:36:12 - >> Compressed game size: 24.82 GB
--------------------------------------------------------------------------------------------------------------------------------------
 03:36:12 - Overall input size: 58.05 GB
 03:36:12 - Overall output size: 24.82 GB
 03:36:12 - Overall conversion time: 02:08:52

[KaktoR]

@Masquerade: Latest ASIS beta has mtx

[rishikrap]

Originally Posted by Masquerade

OK so I'd like to start by apologising to you for denying the use of xtool zlib, this is the first UE3 game I've seen use zlib on tfc/upk archives:

SAMPLE - xzlib

Compressed 24 files, 1,926,995,396 => 7,144,830,307 bytes. Ratio 370.78%
Compression time: cpu 1.89 sec/real 94.48 sec = 2%. Speed 20.40 mB/s
All OK

So, that means I was right after all, regardless of just using hit and trial and seeing fitgirl page to do it lol.

With that established, I have tagged below a sample compressor that will allow you to make an archive compatible with the ASIS script (I recommend using compact mode)

How to use:

1. Download the below 7z, extract it
2. Put your rocket league game folder in to In folder
3. Run the bat
4. Inside the bat, type:
   

   xprecomp+srep+lolz/$wem=srep+pmt_razor

5. Wait until done!

So, I have current compression running on CUI V3 b7.u4 with from around more than an hour and it hasn't yielded any errors either with these settings.

 xzlib+srep_new+lolz_mtx

https://prnt.sc/w8eac5
So, do I keep it running or do I cancel it and do with what you specified or, will what I'm doing give me proper results that I can extract too by making setup using the ciu script. I haven't used the $web or masking since there was no option of doing anything as such.

Compression method explanation:

xprecomp = xtool, configured for zlib.

srep+lolz - srep to remove duplicates and lolz for main compressor. If you want to use lzma2, swap lolz for "7z".

What does $wem mean?

This is a mask, and masks are a way of using multiple compression methods in one archive.

Each mask (and the file types included by it) is included in arc.groups.

This allows the main data to be compressed with xtool,srep,lolz then the audio gets compressed with srep,razor(pmt).

PMT is a chunk based accelerator for slow compressors like mtx, but ASIS doesn't have MTX yet, so PMT is what to use.

Because PMT is chunk based, we run srep beforehand to catch duplicates.

You can define a mask using the arc.groups file, and the grouping must begin with $groupname:

Example:

$help
*.bin
*.blob

This means you can separate off bin and blob files using the /$help identifier in your method.

If that's a bit confusing, I'm sorry!

It was a little confusing not because of your explaination but because I don't know anything, but on reading multiple times I understood almost all of it
The only thing I'm confusd about is do I have to make/edit the arc.groups file manually or was the example just an example for explaination.
Thanks for all this, this explaination did make me understand the basic stuff that's happening which I didn't have any clue about.

[Masquerade]

The only thing I'm confusd about is do I have to make/edit the arc.groups file manually or was the example just an example for explaination.

I added arc.groups for you myself, but if you want to make future groupings, you must make them. If you don't have any issues, stick with what you have.

[KaktoR]

@Masquerade: Mass Effect 3 use zlib too (Unreal Engine).

[rishikrap]

Originally Posted by Masquerade

I added arc.groups for you myself, but if you want to make future groupings, you must make them. If you don't have any issues, stick with what you have.

Thanks for doing that.
Now, just tell me the last thing, the settings I used have yielded me to 22% without errors, so should I use what you sent me or let CIU run its course and then create the setup using asis?
Screenshot 2020-12-23 113804.png
I'm thinking of making another virtual machine and running what you sent me there, my current machine runs Intel Xeon Platinum 8168 4cores @2.70 Ghz with 8gb ram.
Can I cheap out on cores and use a dual core AMD Epyc 7452 dual core @2.40Ghz for that with 8gb as well. I know the task is cpu intensive hence virtual cpu's are already slowing me down significantly.
I don't expect 2 cores do the task at same time but just an hour or so more would not hurt as much and would be half the cost of 4 cores on Azure and if the time difference would be more than that then 4cores it is.

[Masquerade]

I'd just let it run the course now.

[oltjon]

Originally Posted by KaktoR

Anno 1800
v9.2, nothing ripped

 01:27:17 - Selected ARC method for Data1.bin was: srep_new+razor_mtx
 01:27:17 - Selected ARC method for Data2.bin was: xtool_zlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 01:27:19 - Compression initialized in 23.12.2020 at 01:27:19.
 01:27:19 - Starting Data1.bin archive creation.
 01:50:04 - Moving Data1.bin.001 file to media folder.
 01:50:05 - Starting Data2.bin archive creation.
 03:36:09 - Moving Data2.bin.001 file to media folder.
 03:36:11 - Compression finished in 23.12.2020 at 03:36:11.
--------------------------------------------------------------------------------------------------------------------------------------
 03:36:12 - Getting size of the game: Anno 1800
 03:36:12 - >> Installation folder size: 58.05 GB
 03:36:12 - >> Data1.bin file size: 2.67 GB
 03:36:12 - >> Data2.bin file size: 22.15 GB
 03:36:12 - >> Compressed game size: 24.82 GB
--------------------------------------------------------------------------------------------------------------------------------------
 03:36:12 - Overall input size: 58.05 GB
 03:36:12 - Overall output size: 24.82 GB
 03:36:12 - Overall conversion time: 02:08:52

srep_new