Tools for Repacking

[Masquerade]

Very nice tool wrathma and pleased someone has nulled the need for my bat2exe nightmare creation.

[kj911]

wrathma: Many questions!

wem-packer: Why are both "oggre_enc/oggre_dec" binaries needed? Also, how are they 9KB larger than the original files?
In the original "ww2ogg" package, there is another "packed_codebooks.bin" file. Could the lack of this cause problems in some cases?
There is a 64-bit version of this program on GitHub, I won't link it directly, find it and take a look, you'll get the hang of it!

There is also a "revorb" project, isn't that better than "ww2ogg"? Instead of the HDiff package, would xdelta3 be a worse solution? (it runs natively on XP too.) HDiffPatch XP issues from GitHub. (Two x86_64/i686 XP binaries found the posts.)

Is there a way to configure the program, especially "hdiffz", to use very little memory when creating a diff file? (It should comfortably fit within the 512MB upto 1-2GB memory limit.) It is true that when applying a diff file as a patch, in principle a lot of memory is required.

You're obviously asking, why XP?? If everything is true, around 2005 (or earlier?) games that have *.wem files were released and regarding these files, it would be possible to make the repacks publishable/archivable back to XP.

There's a bug in the 32bit version! The "hdiffz" binary file in it is 64bit!

Yesterday, I managed to make all the binaries run on XP. Although, a full native XP-compatible HDiff/patchz binaries would be better case/idea.

In the case of v4.8.0, the "hpathcz.exe", with the simple NT6.0 -> NT5.1 trick, runs smoothly, while the larger "hdiffz.exe" file, I had to tinker a bit more with CFF Explorer + 3 extra DLL files required, to run it.

In the case of v4.11+ versions, the latter situation is present for both files.

The only task that remains to be solved is to create an EXE file that can be run natively under XP by the GO-compiler. (Why does this require the "bcryptprimitives.dll" file?)

[KaktoR]

Added new oodle library
Some new libraries added. I don't know from where I have them since the last update. I am just collecting from games. I don't remember because I have not made notes about it.

[artag]

i don't know if this is the right place to ask, sorry if my english is bad:

wich is the safest and stable cls for precomp right now? (with stdio support) and which
precomp version should i use?

which is the safest cls for srep? (or maybe should i use srep without it). what version is the most stable until today ? 2.2, 3.93, 4.90?


can i remake my old repacks with xtool and forget about pzlib (for zlib streams of course) or pzlib should be used for old games or something like that?

i used lolz 21a7, is there a new updated version?

inno setup 6xxx is stable with isdone dll?

many thanks!

[Masquerade]

^^
For stdio precomp, you can use this version: https://fileforums.com/showthread.php?t=105410

CLS-SREP by ProFrager is still the best cls-srep for decompression. Use srep.exe for compression. 3.93b is alright for most data.

pzlib is obsolete, use xtool.

lolz v22c4b is the latest lolz version.

Inno Setup 6 does work with isdone, yes.

[kj911]

wrathma: There are problems with this GOLang miracle! Does it need an AVX/AVX2-compatible processor? Or do the compiled binaries only work flawlessly on the given machine? What if I remove the GO compiler from the machine, will the binaries compiled until then break?

The GO compiler* used. (I also compiled THIS with it, on XP it prints out the Help nicely, that's all, when compiling (go.exe build source.go) it also prints out similar errors as what can be read below)

*Used last v1.26.2-1 i386 packages and separately compiled from sources. (This v1.24.4 from XP) from make EXEcutables.

The x32 binary you posted, writes these (under Win7 SP1 x64):

C:\wemtest>wem-packer.exe
Exception 0xc0000005 0x8 0x0 0x0
PC=0x0

internal/runtime/syscall/windows.asmstdcall(0x20)
        internal/runtime/syscall/windows/asm_windows_386.s:37 +0x2a fp=0x3afa08
sp=0x3afa04 pc=0xd1029a
eax     0x8
ebx     0x117c630
ecx     0x0
edx     0x3afa38
edi     0x3afa0c
esi     0x3afa70
ebp     0x3afa0c
esp     0x3afa00
eip     0x0
eflags  0x10202
cs      0x23
fs      0x53
gs      0x2b

The x64 version also produces similar and longer errors. (Checked on two different Win7 x64 PCs.)
If you chain the compiled program to the given machine/Windows files, it is very bad and makes the given program and the compiler unreliable.
If the source code were rewritten to c/c++, how much better would it be? (With AI, I had a CPP produced, but the program is faulty, it could only serve as some guidance for porting to CPP.)

In its current state, the CPP file only produces "apparently" working programs after compilation!

So far, I have produced 3 different packages from it, but under XP it throws errors similar to the above, the program prints the prompt in the console window regardless. Under Win7 there are no more strange errors, but as it turned out, if you have to wait for IO errors, HDD or the CPU is too busy, then it will make errors, there is a possibility of making errors. On a laptop with a 2-core CPU, not foolproof, on an 8C/16T Xeon PC, it seems to be better. That is, it doesn't unpack the files, it doesn't run the "hpatchz.exe" file at first, or some other anomaly.

First version maked from Win7 via GO v1.26.2-1: Link
Second version maked from Win7 via GO v1.24.4, hopefully targeted WinXP: Link

Masquerade: Do you still have the BAT2EXE program that you used to create the "WemTool.exe" file? Would you create it again? I'm attaching the patched/replaced/updated files. That way it will be 100% compatible with XP!

[panker1992]

i decided to pop some stuff after a long time, not only i updated my archiver to be a beast!

But i run a showcase to demonstrate the vast capabilities it can do!

i decided to run resident evil requiem precompression only

[artag]

Originally Posted by Masquerade

^^
For stdio precomp, you can use this version: https://fileforums.com/showthread.php?t=105410

CLS-SREP by ProFrager is still the best cls-srep for decompression. Use srep.exe for compression. 3.93b is alright for most data.

pzlib is obsolete, use xtool.

lolz v22c4b is the latest lolz version.

Inno Setup 6 does work with isdone, yes.

sorry, for the srep cls, are you talking about this one?:

https://krinkels.org/resources/cls-srep.261/

or are you talking about srep inside?

i cant donwload from krinkels, can you or anybody provide the md5 of the cls
so i can find it on my folders? (I surely have it, but I must be sure that it is the right one.)

i have lolz v22c4b, is everybody using it? is it stable?

thanks again

[wrathma]

Originally Posted by KaktoR

Some small bug

Found 247 .wem files to pack (recursive)
Using 12 threads

[███████████████████████████████████████░] 246/247 (99.6%) ETA: 0s
All files packed successfully!

Also it would be good to give some information on how long the process took.

Suggestions:
Add -c option for hdiffz compression
Change skip -s option to skip files even if they are same size as input -> equal or greater then

thanks for your detailed review KaktoR. the progress bar was shamelessly copied from stackoverflow and unfortunately it
doesnt work great with multithreaded operations. so i removed it and added a simple progress bar. should be fixed by now. also
added -c option for hdiffz. everything you pass after this will be passed directly to hdiffz -c-{here}. example -

wem-packer.exe -czstd-22 wemfolder -> hdiffz -c-zstd-22 ...

Originally Posted by kj911

wem-packer: Why are both "oggre_enc/oggre_dec" binaries needed? Also, how are they 9KB larger than the original files?

There is also a "revorb" project, isn't that better than "ww2ogg"? Instead of the HDiff package, would xdelta3 be a worse solution? (it runs natively on XP too.) HDiffPatch XP issues from GitHub. (Two x86_64/i686 XP binaries found the posts.)

Is there a way to configure the program, especially "hdiffz", to use very little memory when creating a diff file? (It should comfortably fit within the 512MB upto 1-2GB memory limit.) It is true that when applying a diff file as a patch, in principle a lot of memory is required.

The only task that remains to be solved is to create an EXE file that can be run natively under XP by the GO-compiler. (Why does this require the "bcryptprimitives.dll" file?)

both oggre encoder and decoder is required because sometimes doing oggre_enc and oggre_dec on a file gives different
files (crc error). but doing oggre_enc on the output of oggre_dec will always give same files. try this out yourself
on diffrent ogg files you will get it.
i have never used revord, i guess ww2ogg works so ill use it for now (rule of engineering, if it works dont touch it).

xdelta3 and hdiffz could be interchanged as they are basically doing the same thing. i have added a xdelta build for
you. keep in mind that xdelta produces a little bigger diff files compared to hdiffz. tho the difference is minor.
during my tests, the entire tool running on 20 threads didnt cross the 500 mb ram usage mark. stayed mostly on
380-400 mb of ram usage (wempak+hdiffz+oggre+ww2ogg). applying patch is easy while making patch is harder and takes
lot of ram. but when the files are small ram usage also goes down.

golang turned down support for windows xp long ago i forgot to mention that so this tool shouldnt really work on
windows xp. i dont even ship wemunpak on my repacks and neither should you. the binary is bloated with golang
bloat code. this is why i added a feature to make a batch file to run the entire decompression chain.
check the -n option. you can then ship the batch file with the tools (hpatchz+oggre) and a
parallel processor (prl or mparallel)

Originally Posted by kj911

wrathma: There are problems with this GOLang miracle! Does it need an AVX/AVX2-compatible processor? Or do the compiled binaries only work flawlessly on the given machine? What if I remove the GO compiler from the machine, will the binaries compiled until then break?

The x64 version also produces similar and longer errors. (Checked on two different Win7 x64 PCs.)
If you chain the compiled program to the given machine/Windows files, it is very bad and makes the given program and the compiler unreliable.
If the source code were rewritten to c/c++, how much better would it be? (With AI, I had a CPP produced, but the program is faulty, it could only serve as some guidance for porting to CPP.)

In its current state, the CPP file only produces "apparently" working programs after compilation!

it doesnt need a avx/avx2 compatible processor to run. atleast the logic should
work on all processors. golang programs are compiled and they are standalone.
unlike python/nodejs you dont need a interpreter to be installed on the user end to run this.

honestly i dont see any reason to rewrite this tool into cpp/c. performance
difference would be minor or nonexistent, but this type of tool is not meant
to be coded in these low level languages. i have seen your cpp code, you mostly
copied my structure a to z but the encoding/decoding logic is not there. if you
complete this on c the binary sizes might be cut to half or even more. and it
would probably work on older systems. if you follow what i do, i pack the files
in a modern device and then ship the tools required with a batch file. well if
you want feel free to rewrite this tool in c/cpp.

[KaktoR]

Thanks for the update.

Found 1 .wem files to pack (recursive)
Using 12 threads

[0%] 0/1 Files processed, ETA: 0s
All files packed successfully in 0s!

It's just a visual bug, so nothing to deal with on tryhard level.

[kj911]

wrathma: Thanks for the detailed description!

Speaking of switches:

wem-packer: Here, using "-s" or "-s -b" does not cause any particular confusion, during compress, unlike the unpacker.

wem-unpacker: Do not use the "-b" switch here, otherwise it may happen that the first time it is run it will fail (referring to "hdiff/xdelta"), while the second time it will run without errors. It is better to use it without switches, except for the "-tN" switch.

This is related to the fact that the "original" *.wem file is also in the folder, which conflicts with one of the members of the hdiffz+oggre chain and should appear at the end of the process, as the new *.wem file. That is, wem-unpacker deletes the original *.wem file and exits here, without continuing to run the hdiffz+oggre chain to then finish the unpacking properly, as it should.

C:\>wem-packer.exe -s -b music_frontend
Found 4 .wem files to pack (recursive)
Using 2 threads with size check (keeping backups)


268707965.wem: output is bigger than input, skipped

681733649.wem: output is bigger than input, skipped

775589494.wem: output is bigger than input, skipped

All files packed successfully in 7s!
C:\>wem-unpacker.exe music_frontend
Found 1 .ww files to unpack (recursive)
Using 2 threads

[0%] 0/1 Files processed, ETA: 0s
Errors encountered:
  108187332.ww: xdelta failed
Completed with errors in %s
0s
C:\>wem-unpacker.exe music_frontend
Found 1 .ww files to unpack (recursive)
Using 2 threads

[0%] 0/1 Files processed, ETA: 0s
All files unpacked successfully 4s!

wempack-xdelta package: This doesn't run under Win7 x64 either. (Has anyone tested it under Win7, are you using the uploaded files or only Win10/11?)

Running the GO compiler with these switches, I managed to produce somewhat smaller EXEs. It should be 100% compatible with Win7 (also with 32-bit editions, theoretically going back to Vista, up to the early 32-bit version of Win10.) although, all "embedded binaries" are XP-compatible.

go build -a -gcflags=all="-l -B" -ldflags="-w -s" -o myapp main.go

The uploaded CPP file would require a more experienced programmer than me, this was also transpiled with AI, from GO to C++.

UPD: Embedding binary files via alternative methods: https://github.com/samuelngs/binary-go workable from these "wem-(un)packer's" project?

[wrathma]

Originally Posted by kj911

wempack-xdelta package: This doesn't run under Win7 x64 either. (Has anyone tested it under Win7, are you using the uploaded files or only Win10/11?)

that was my mistake. i compiled it with latest golang (v1.26.2) that doesnt
support windows 7. i tried to build it today with golang v1.20 on win10 and
the compiled exe works fine on windows 7. also one thing i noticed, that
xdelta or wemtool also doesnt work on windows xp (i might be missing out
something, check screenshots). i tried to compress the wem files in win 7 x64
vm it worked. then i generated a batch script and sent that to my win xp vm,
and then it gave me some errors that say xdelta wont work. unfortunately
thats something i cant fix.

Originally Posted by kj911

Running the GO compiler with these switches, I managed to produce somewhat smaller EXEs. It should be 100% compatible with Win7 (also with 32-bit editions, theoretically going back to Vista, up to the early 32-bit version of Win10.) although, all "embedded binaries" are XP-compatible.

go build -a -gcflags=all="-l -B" -ldflags="-w -s" -o myapp main.go

UPD: Embedding binary files via alternative methods: https://github.com/samuelngs/binary-go workable from these "wem-(un)packer's" project?

i also used a similar build flag -

go build -ldflags="-s -w" -tags windows -trimpath -gcflags=all="-l -B" ...

i did not use -a because all it does is rebuild external packages (wempack doesnt
use any external packages). and for the alternative binary embedding method, i
dont see any reason touse it. i also used to use this binary-go library long ago but
recent golang updates made this one obsolete. now the functionality is built into
golang. this is what i use (you can see the first few lines of my code) -

//go:embed tools/ww2ogg.exe
var ww2ogg []byte

//go:embed tools/oggre_enc.exe
var oggre_enc []byte

it simply copies the raw data as a a binary stream while compiling the exe. and
when running the tool it simply copies the binary data from the exe to a temp
folder. see extractTools() from line 50-72 of wem-packer.go.

[kj911]

Use the binary files I uploaded! I've sorted them and patched them so they can run under XP! (For use the 32bit wem-(un)packer releases.)

List: WemTool_XP_compat_files.7z (Although this pack was uploaded directly for Masquerade.) You can download the XP compatible "xdelta.exe" from the original site (marked "i686"). Or you can use the files below. (After running, copy them from the TEMP directory.) The my uploaded "mp.exe" XP-compatible.

wempack_x86_hdiff.7z
wempack_x86_xdelta.7z

If all *.exe files, fixed for XP, run without errors, then the batch script should run without errors when compressing and decompressing *.wem files. (I haven't even experimented with this yet.)

Win7 compatible GOLang (I already linked it earlier.): https://github.com/thongtech/go-legacy-win7

[wrathma]

Originally Posted by kj911

If all *.exe files, fixed for XP, run without errors, then the batch script should run without errors when compressing and decompressing *.wem files. (I haven't even experimented with this yet.)

your patched exes works under xp environment. i think you can use
the batch script function of wem-unpacker with your fixed winxp
executables to decompress wem files under winxp environment.

currently im working on editing my prl tool (a mparallel alternative)
to move to win32 system api calls instead of relying on crt libraries.
that way it can run natively on any post winxp system without needing
to install any msvcrt libraries. plus the file size will also shrink (it is still small).

[wrathma]

based on latest github/zstd-1.5.7 sources. needs thorough testing. use 1.5.7 to compress. examples -

> zstd_decompress.exe

 Experimental ZSTD Decompressor by SYMM

 usage: zstd_decompressor input output
 replace input and/or output with - to use stdin/stdout

> zstd_decompress.exe input.zst output.exe
output.exe : 126607283 bytes

zstd_decompress.exe - - < input.zst > output.exe
<stdin>: 126607283 bytes written

on benchmarks its almost 2x slower than the official zstd binary release (1.5mb). still it is fast with almost 350MBps decompression speed for 4GB uncompressed data on my test system (4c8t i3) on nvme.