My Precomp fork with stdin/stdout support

[ncomerci]

Hi, I've been working on a precomp fork for a few weeks now.
There is still a lot of stuff I want to do with it, but I think right now its on a place where it might be cool to let you guys try it.

Interesting features so far:

• No changes to PCF header format, meaning at least for now, there should still be full compatibility with PCF files created from main precomp project (v0.4.8) lastest executable and this fork
• Temp files have a random tag prepended on their filename, meaning you can run multiple instances working on files on the same folder and they should not interfere with each other
• stdout support for precompression (stdin impractical, at least without heavy reengineering as we do seek on the input file many times during recompression)
• stdin/stdout support for recompression

Usage:
Mostly the same as regular precomp, the only differences are for using stdin/stdout

• (example of decompression using stdout) precomp -ostdout -cn -intense -brute file.dat > file.dat.pcf
• (example of using stdin and stdout for recompression) srep -v0 -d file.tar.pcf.srep - | precomp -ostdout -r stdin | 7za x -si -ttar

so essentially, just use stdout as "output filename" or stdin as "input filename"

This fork should make precomp more practical to use in contexts where only xtool was practical before.

If anybody wants to see the code its at https://github.com/nicolas-comerci/precomp-cpp
Code is still on the compatible_refactor branch

You can find attached 32/64 bits zipped executables.
Please feel free to test and use and report any errors.
For now I am mostly interested on making sure I haven't introduced any new bugs, so I probably won't be fixing anything that also fails on mainline Precomp 0.4.8

EDIT 08/21/22: Added v1.0 of my fork, has been tested a little more and I fixed some OTF compression/decompression problems

[Lord.Freddy]

There seem to be some problems

[Masquerade]

Originally Posted by Lord.Freddy

There seem to be some problems

Output is being redirected to STDERR (console window)

[Gehrman]

Thanks ncomerci

Good Work

[External compressor:precomp048f]
header = 0
packcmd   = precomp64 -cn -intense0 -ostdout <stdout> $$arcdatafile$$.tmp 
unpackcmd = precomp64 -ostdout -r stdin <stdin> <stdout>

[Gehrman]

Compressed 2 files, 314,572,800 => 511,394,374 bytes. Ratio 162.57%
Compression time: cpu 0.31 sec/real 75.05 sec = 0%. Speed 4.19 mB/s
All OK

Tested 2 files, 511,394,374 => 314,572,800 bytes. Ratio 162.57%
Testing time: cpu 0.94 sec/real 30.75 sec = 3%. Speed 10.23 mB/s
All OK

[Lord.Freddy]

I got this error in test archive

[ncomerci]

Originally Posted by Lord.Freddy

I got this error in test archive

it says liblzma error, you are using precomp's included compression?
it wouldn't surprise me if I broke it because I didn't test much with it, will look into it.

however for now you can use -cn option to disable the included compression (this I did test thoroughly) and just use some other compression method with freearc, or pipe it through to xz/7za...

In fact, I might even completely delete the included compression capabilities in a future version on this fork... seems to me like stdin/stdout support makes that obsolete and just unnecessarily complicates the code

[ncomerci]

Originally Posted by Lord.Freddy

I got this error in test archive

Added new version on the original post, on my tests OTF compression/decompression now works correctly

[Gehrman]

Compatible with Windows XP

[kj911]

Gehrman: Thanks from Win XP compatible executables!

One problem from precompf: Totally ignore the external INI file and settings, located in main EXE folder. See the picture.



Second smaller problems: More huge external dll's. (45 file!)

Solutions? 1. Recompile again or make one executable file via embedding more resources. Created my Enigma Vbox-ed versions the package. (Used in last v9.9.0) Attached in and testing now.

UPD: Added size optimized versions the package from smaller EXE's size fans.

Precomp(f).exe files and few dll's packed in UPX and packed all files to one Enigma VBoxed EXE file. More AV in falsely alarms!! Use at your own risk! This one positive the package: 5.19/5.20MB vs. 1.94MB's EXE file size in HDD. Compressibility via to 7Z in get ~0.1MB's smaller size usage from installers. (One EXE file packed to 7z archive.) See TXT file from more infos. Safer ways the many AV's conflicts to use bigger EXE files.

Compresing via stdin/out mode testing now in next days.

UPD2: The quick test results added comprare the Precomp0.4.8 stdio vs. xtool 0.12 x86 versions. This make few MB's archive and decompressing, its precomp now send ~1kB's binary input data from console window without stdout. Decompression its finished without any extra issues. 2nd decompression, not have stderr issues. More bigger files test its OK. No problems or any moe. CRC's its OK tested in 2nd decompression stage will finished and extracted files overwriting. See TXT file from test results from comparing.

[Gehrman]

@kj911
The following tools are used.

Winpp
HXD
kernelxp.dll
QTfiles32 DLL Files

The following example was made using the above tools

[kj911]

Gehrman: Two quick questions.

1. What would that be "QTfiles32 DLL Files" ?? No found reasonable results.
2. WinPP: Would that be it? -> https://www.winholdem.net/winpp/

HXD use patching EXE's running to older OS? (Win7 to WinXP) See more post in the topic --> https://encode.su/threads/1838-Comma...Profiling-Tool

This precomp EXE files, compiled in GCC or use any compiler?

[Gehrman]

Winpp
QTfiles
We use HXD to change KERNEL32.dll to KERNELXP.dll.


Winpp can also add CLS-LOLZ_x64.exe and CLS-LOLZ_x86.exe compatibility on Windows XP.

[kj911]

Gehrman: Thanks from replies!

In last cls-lolz_x86.exe file natively works and running from Windows XP. (To my knowledge, this original EXE CRC: 97A4C611) Older v21a5/6/7 versions required from exe patching. (And OGGRE and DLZ.) In better ways posted from external site use "MakeLX and MakeXP" mini tools, using patched exe from XP system, without using external dll's. See test it.... and attached.

MakeXP-x86.exe -> XP compatibility patcher.
MakeLAA-x86.exe -> Add large-address-aware functions. (Or just use 4GB memory patcher tools from available via net.)

Usage from via cmd: MakeXP/LAA-x86.exe anyfile.exe or drop any exe/dll file to EXE file icon.

Installed 64bit Win XP not available in my own circle of acquaintance. Not have available any x64 XP install disk and hardware from testing purpose.

UPD: Comparing the MakeLX and WinPP exe patching. No size difference from two patched executables. WinPP patched to OS and Subsystem version to 4.0 its original higher than 6.0 or any values. MakeLX patched to OS/Subsystems version to 5.0... No se any differences. (Except patcher exe size. 520kB vs. 2kB )

UPD2: Quickly compare MakeLX and 4GB patch EXE tools differences... MakeLX not change 3bytes in exe file, located in hexa: 00120 lines. Hexa values: "00 00 00 00 03 00 00 81" 4GB pated version hex: "AF 23 03 00 03 00 00 81" OS/Subsystem line located in: 00108 and 00110 and first hexa byte changed 06 to 05 or 04 values.

[panker1992]

Hello, Mr Nomerci. I have been testing your stuff secretly

I have found some bugs regarding your version of precomp
Game that i Found a bug: Resident Evil HD Remaster
Folder and files: nativePC\arc\id\e_rom\text\fre\mes\*.*
(They are proccessed and resotred but CRC doesnt remain the same. the test was outside freearc)

Furthermore, I have seen in your github you uploaded a neo Releases series.
and you have removed the ability to keep data decompressed thus making the data lzma-ed by default. srep cant be used and lolz cant be used

any fixes? in the making?