Best Compression Methods for 'Specific' Games. Q&A

[L33THAK0R]

I'm very new to lzo1x pre-compression, could anyone explain why XTool in this example is able to detect some streams but not able to process any? I've tried levels 1 - 10 and the same amount of streams are detected but none are able to be processed

Creating Archive Please Wait...

FreeArc 0.67 (March 15 2014) updating archive: Data.arc
Compressing 1 file, 38,255,712 bytes. Processed   0%
Compressing 38,255,712 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mlzo1x:l6 -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 0/2854
Time: 00:00:01 (00:00:03)
Memory: 128 MB (128 MB)

100%
Errorlevel=0
Compressed 1 file, 38,255,712 => 38,255,982 bytes. Ratio 100.00%
Compression time: cpu 0.05 sec/real 1.63 sec = 3%. Speed 23.44 mB/s
All OK

All done.

The algorithm used is definitely lzo1x, and creating a database with bms2xtl detects an additional ~5000 streams but alas none are detected. I've also not been able to find many other lzo libraries to test with so if its a library issue, if anyone could point me in the direction to look for others it'd be greatly appreciated.

[Lord.Freddy]

Originally Posted by L33THAK0R

I'm very new to lzo1x pre-compression, could anyone explain why XTool in this example is able to detect some streams but not able to process any? I've tried levels 1 - 10 and the same amount of streams are detected but none are able to be processed

Creating Archive Please Wait...

FreeArc 0.67 (March 15 2014) updating archive: Data.arc
Compressing 1 file, 38,255,712 bytes. Processed   0%
Compressing 38,255,712 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mlzo1x:l6 -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 0/2854
Time: 00:00:01 (00:00:03)
Memory: 128 MB (128 MB)

100%
Errorlevel=0
Compressed 1 file, 38,255,712 => 38,255,982 bytes. Ratio 100.00%
Compression time: cpu 0.05 sec/real 1.63 sec = 3%. Speed 23.44 mB/s
All OK

All done.

The algorithm used is definitely lzo1x, and creating a database with bms2xtl detects an additional ~5000 streams but alas none are detected. I've also not been able to find many other lzo libraries to test with so if its a library issue, if anyone could point me in the direction to look for others it'd be greatly appreciated.

maybe you need to remove the level option, sometimes it works.

[L33THAK0R]

Originally Posted by Lord.Freddy

maybe you need to remove the level option, sometimes it works.

Unfortunately seems like some data actually got stripped away doing that, resulted in a 99.51% ratio

[Edison007]

Originally Posted by L33THAK0R

I'm very new to lzo1x pre-compression, could anyone explain why XTool in this example is able to detect some streams but not able to process any? I've tried levels 1 - 10 and the same amount of streams are detected but none are able to be processed

Creating Archive Please Wait...

FreeArc 0.67 (March 15 2014) updating archive: Data.arc
Compressing 1 file, 38,255,712 bytes. Processed   0%
Compressing 38,255,712 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mlzo1x:l6 -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 0/2854
Time: 00:00:01 (00:00:03)
Memory: 128 MB (128 MB)

100%
Errorlevel=0
Compressed 1 file, 38,255,712 => 38,255,982 bytes. Ratio 100.00%
Compression time: cpu 0.05 sec/real 1.63 sec = 3%. Speed 23.44 mB/s
All OK

All done.

The algorithm used is definitely lzo1x, and creating a database with bms2xtl detects an additional ~5000 streams but alas none are detected. I've also not been able to find many other lzo libraries to test with so if its a library issue, if anyone could point me in the direction to look for others it'd be greatly appreciated.

Maybe used pro-functions. Upload example file for testing

[L33THAK0R]

Originally Posted by Edison007

Maybe used pro-functions. Upload example file for testing

Thanks for offering to have a looksie, really appreciate any insight you can offer, also in case its of any interest I've also got an LZ4 sample I've been having similar issues with, tried about a dozen libraries but no dice so far, xtool (0.6.5) is also detecting streams but unable to pick any up unfortunately.

[KaktoR]

For your lz4 sample you have to use lz4f with level 2 and pick the right lz4 library

Compressed 1 file, 32,882,867 => 158,560,939 bytes. Ratio 482.20%
Compression time: cpu 0.08 sec/real 1.18 sec = 7%. Speed 27.85 mB/s
All OK

You have to look which level produces the closest recompressed size to the original size



However in a quick test there are still some streams left behind.

[L33THAK0R]

Originally Posted by KaktoR

For your lz4 sample you have to use lz4f with level 2 and pick the right lz4 library

Compressed 1 file, 32,882,867 => 158,560,939 bytes. Ratio 482.20%
Compression time: cpu 0.08 sec/real 1.18 sec = 7%. Speed 27.85 mB/s
All OK

You have to look which level produces the closest recompressed size to the original size



However in a quick test there are still some streams left behind.

Ah didnt realise I had to specify lz4f, many thanks, also what tool are you using to get that console output if you dont mind me asking?

[KaktoR]

--verbose mode from xtool

[Edison007]

lzo_sample.rar

The old function LZO1X-1 was used. You can use these libraries. But not all streams are successfully recompressed, because lzo-detect in xtool is't ideal.

[L33THAK0R]

Originally Posted by Edison007

The old function LZO1X-1 was used. You can use these libraries. But not all streams are successfully recompressed, because lzo-detect in xtool is't ideal.

Just ran a quick test, this works wonderfully, thanks so much! I've had this sitting on the back-burner for months, it's very satisfying to have it finally sorted!

[L33THAK0R]

Does anyone know if theres a way to bulk apply/insert a hexadecimal pattern into a given chunk of data if given a start and end offset? I've got some projects I'm working on that require a few hundred blanking/0-filling operations and its awfully slow to do so with HxD's GUI. I've gone through the SFK documentation a half dozen times and alas to no avail. My current solution is just to have a massive blank file and drawn on the given offsets from it but this is obviously not very practical and is a pain having to increase its size should I be attempting to patch out a chunk from a larger archive in the future.

[KaktoR]

Try with xtool's erase/replace command

[dixen]

Pathfinder: Kingmaker (GOG Version) [New repack]

Bundle Folder (excluded manifest files)
XTool v0.65 (unity+lz4 plugin v1.75)+srep+lolz
21 gb > 50 gb > 25 gb > 7.2 gb

Sound files (*.wem & *.bnk)
srep+lzma2
3.2 gb > 2.6 gb

*.assets *.resS & extensionless files
srep+lolz
13.4 gb > 2.8 gb

Rest files
400 mb > 65 mb

Repack's size - 12.7 gb
Install time - 29 min. (R5 3600)

[shazzla]

Originally Posted by L33THAK0R

Ah didnt realise I had to specify lz4f, many thanks, also what tool are you using to get that console output if you dont mind me asking?

Hi Kaktor and everybody !

What is this tool in the screenshot ? And what parameter was used?
Im out nearly 2 years from repacking!
Btw anyone can share me an lz4 library collection for xtool?
Many thanks!

[KaktoR]

Originally Posted by shazzla

Hi Kaktor and everybody !

What is this tool in the screenshot ? And what parameter was used?
Im out nearly 2 years from repacking!
Btw anyone can share me an lz4 library collection for xtool?
Many thanks!

xtool --verbose mode

Attached file contains some lz4 libraries