Tools for Repacking

[newfolder]

Unity Precomp, the open source tool to precompress unity3d/bundle files, includes the binary and the source code in c

[newfolder]

Unity Decomp is a program that decompresses Unity bundle files, essentially expanding them so the expanded file is compatible with the Unity engine. This is ideal for people who want to optimize compression, since removing the lz4/lz4hc data allows you to use better compressors like 7-Zip, LOLZ, or FreeARC.

Includes the binary and the source code in c

[wrathma]

this is a standalone bz3 decompressor. based on latest iczelia/bzip3
commit 97a6da2. use bzip3 v1.5.3 to compress. decompression
speed same as official bzip3 release.

tried to build with winxp support but failed. i dont think bzip3 source
codes support winxp at all. eventually got the standalone compiled
x86 exe down to 35kb (no ucrt/msvcrt dll dependency) but it wouldnt
work on win7 without ucrt. worked fine on 10+ tho. so had to add
ucrt. works on all win7+ systems even without any msvcrt installation.

attachment includes source code. to compile make a decompress/ dir
inside bzip3/ git repo and copy the attached CMakeLists.txt and
main_decompress.c and then build the project using cmake.

> bunzip3_x86.exe

 bz3 decompressor by SYMM

 usage: bunzip3 input output
 replace input and/or output with - to use stdin/stdout

> bunzip3_x86.exe input.bz3 output.exe
output.exe : 824130 bytes

> bunzip3_x86.exe - - < input.bz3 > output.exe
<stdin>: 824130 bytes

[kj911]

BUNZIP3: A question arose, how can it not get an error if the "libsais" library is not included?

Please note that if you are unpacking these *.bz3 archives on a 32-bit machine, do not use a large block size and/or multiple threads, otherwise it will crash and/or exit silently, resulting in a 0-byte file.

Packed in "-b 511" switches:
x64 exe only one capable it! x86 version (include my XP-versions) still crashing!

Packed in 320MB's block size:
x64 exe: works.
x86: silently exit and will get 0 byte file.
x86_laa: only one capable ~2 million KB memory allocation from x64 Windows host, perfectly decompress.

Packed in 255MB's block size:
all x86 exe hopefully works, included all x32 systems.

Recommendedly use max. 255MB's block size or failsafely smaller values from targeting real 32bit systems. (ex: 64/128/192/224MB.)
These "-b 255" packed archives, I need ~1 600 000KB free memory allocation.
Tested from "enwik9" file.

[wrathma]

Originally Posted by kj911

BUNZIP3: A question arose, how can it not get an error if the "libsais" library is not included?

if you see at the top of CMakeLists.txt youll find that it includes src/libbz3.c in the build command which imports libsais.h
btw i tried your winxp builds and it works fine on my vm, can you share how you did that ? did you use very old visual studio ?

[kj911]

The source code wouldn't compile by hand... (Neither with MinGW nor with w64devkit.)

The main solution was (if all files are in the correct right place), that I had to generate a MinGW-compatible "Makefile" from the "CMakeLists.txt" file using "cmake" in "w64devkit". (last v2.6.0 or 2.7.0 with GCC15.2.0. I'm looking now, v2.8.0 has already been released.)

In principle it was this: cmake -G "MinGW Makefiles" "CMakeLists.txt"

After that, I just had to run "mingw32-make.exe" and the binary file was ready. (After that, I cut out the excess with "strip.exe".)
"*_laa.exe" was created by patching the original file with a small program called "4gb_patch".

w64devkit can also build XP-compatible EXEs. (With some luck.)

These lines changed to:
/SUBSYSTEM:CONSOLE,6.01 --> 5.01
not resolve and generate XP-compatible binaries?

Visual Studio?? As far as I know, it's very difficult to get this nowadays (let alone install it offline), not to mention the many GB of storage space required...

By any chance, you don't have the xp v141 package on your machine? There might be a couple of CAB compression sources. What does it actually do, compared to makecab? Rust, you don't have it installed by any chance?

[wrathma]

Originally Posted by kj911

These lines changed to:
/SUBSYSTEM:CONSOLE,6.01 --> 5.01
not resolve and generate XP-compatible binaries?

By any chance, you don't have the xp v141 package on your machine? There might be a couple of CAB compression sources. What does it actually do, compared to makecab? Rust, you don't have it installed by any chance?

i have v141_xp msvc toolset. i tried atleast 20 times with different compiler/linker
args to try to add xp support. yes i also tried /SUBSYSTEM:CONSOLE,5.01 it doesnt
help much. as i have seen using it and not using it is the same. if it were to run on
winxp it would run without this. btw even if you have v141_xp toolset, cmake will
try to target latest toolset. you have to specify it before building like this.

cmake -G "Visual Studio 18 2026" -A Win32 -T v141_xp -B build

yeah it is heavy. visual studio is taking up almost 50gb on my system drive.
but thanks for the tip, i never knew using non msvc compiler could fix it.

[kj911]

CMake can, in principle, be configured to always use the given "toolkit" to compile the given source code. (ex: Ninja, MinGW, VisualStudio, etc...)

To be honest, I don't have much experience with these programs,.. If, the package is put together well, the desired/tested program can usually be produced, if there is source code for it.

Manual patching of EXEs does not always solve the problem of whether they can be run on XP? (If, Vista+ compatible EXEs are released after compilation.) If there are no significant number of Vista/7 kernel-specific calls in the binary file, then they usually work on XP as well.

[wrathma]

so this patched dll uses true stdio (no fileio) compression. directly
from ram to ram. it doesnt write anything to disk by itself.
as it is ram to ram, it will use more memory compared to fileio.

maximum memory usage during compression - input filesize +
output file size + compression memory. maximum memory
usage during decompression - decompression memory + ~4mb.

notes:

1. as i hooked multiple fileio functions of win32api, normal fileio
   compression/decompression through razor is not possible (with
   my patch). using fileio with my patch may lead to bsod. so i have
   added some checks to prevent accidental fileio operations. to do
   fileio operations use the original rz.exe.
2. you might see my patch reading/writing V_IN.SYMM or
   V_OUT.SYMM files, those are just my handles for virtual memory
   address. not actual files on disk.
3. its NOT interchangeable with Razor12911's patch or my old
   "fake" stdio patch.

compiled with mingw-clang. will add the source code after some
cleaning. works fine with arc. added sample arc.ini. for usage
instructions see test.bat/arc.ini.

edit:
fixed a memory leak issue (idk how it slipped past me). removed
debug-specific checks that are not required now. some cosmetic
changes and removed unused debug code. also fixed a bug where
it would fail with freearc decompression.
during testing i found out that razor decompressor is sequential so
made this patch do sequential streamed decompression. so memory
usage is same as (+ ~4 mb) non stdio operations. optimized dll
size (115kb to 14 kb).

[Dunnowho69]

Originally Posted by wrathma

so this patched dll uses true stdio (no fileio) compression. directly
from ram to ram. it doesnt write anything to disk by itself.
as it is ram to ram, it will use more memory compared to fileio.

maximum memory usage during compression - input filesize +
output file size + compression memory. maximum memory
usage during decompression - input filesize + 8mb output chunks
+ decompression memory.

i had to hook multiple win32api system calls to fool razor archiver
to use a memory address instead of a file.

compiled with msvc. will add the source code after some cleaning.
works fine with arc. added sample arc.ini. for usage instructions
see test.bat/arc.ini.

If you don't mind, what's the difference between this and https://www.fileforums.com/showpost....81&postcount=8 ? I use razor all the time that's why just asking if i should use this from now on

[KaktoR]

The difference is no temp file copy which simply means it is faster. Not compression itself but it simply doesn't make any temp files. If you use this to compress files on HDD it should make a clear difference in overall compression times. With the old stdio patch rz.exe will copy temp from HDD to HDD (same as lolz does). This step is obsolete. There is nothing faster than RAM copy.

[kj911]

The other day I noticed that the original Razor compressor (rz.exe) created a 32MB temporary file in the %TEMP% folder while testing a 7+ GB archive. Maybe because it would have run faster, testing/unpacking the compressed data, than the HDD speed.

Could someone port Razor to WinXP?? There is an early, v0-1 "draft" source code, heaven knows where it came from. The point is that the program can be compiled for XP, but it doesn't matter what kind of compiler. The MinGW version compresses EXEs worse than the "draft" EXE. This one runs on XP with a little patching. It only handles one file, it's not "fuzzsafe" or whatever they call it. The compression difference is 3-4% worse in favor of the v1.0x series, with a dictionary size of 64MB. There is also a decoder/decompressor compatible with the original RZ, but whether this can be ported to WinXP is a good question.

[Masquerade]

^^
Maybe you'd get a better answer asking on encode's forum.

[wrathma]

Originally Posted by wrathma

fixed a memory leak issue (idk how it slipped past me). removed
debug-specific checks that are not required now. some cosmetic
changes and removed unused debug code. also fixed a bug where
it would fail with freearc decompression.
during testing i found out that razor decompressor is sequential so
made this patch do sequential streamed decompression. so memory
usage is same as (+ ~4 mb) non stdio operations. optimized dll
size (115kb to 14 kb).

if anyone is using my patch i will highly suggest to update it
as some bugs and dead code unfortunately slipped past me
on my last build. a bsod-level issue was fixed. and overall
dll size dropped by almost 90%. will try to be more careful
from now.

Originally Posted by Dunnowho69

If you don't mind, what's the difference between this and https://www.fileforums.com/showpost....81&postcount=8 ? I use razor all the time that's why just asking if i should use this from now on

as KaktoR said, all it does is compress from ram to ram
eliminating the need to copy stdin to a temp file. if you
try using it on a hdd you will understand the difference.
decompression is same as Razor12911's build.
also a smaller dll wouldnt hurt i think (1.6mb vs 14kb)

Originally Posted by kj911

Could someone port Razor to WinXP??

out of my skillset