FileForums - MTX - The Universal Accelerator

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - MTX - The Universal Accelerator (https://fileforums.com/showthread.php?t=103853)

MTX - The Universal Accelerator

https://i.ibb.co/k2VpTrJj/hh.png

MTX v0.3
Universal Multi-Threading Accelerator for FreeArc External Compressors
Windows x64 · single exe, no dependencies · by BLACKFIRE69

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━

▌ OVERVIEW
Many of the strongest external compressors (precomp, mpz, rz, nz, srep …) are single-threaded: on a modern multi-core CPU they leave most of the machine idle.

MTX sits between FreeArc and the compressor, splits the archive data stream into fixed-size chunks, runs N parallel instances of the compressor on those chunks, and merges the results back into one sequential stream — in guaranteed original order. One config line turns a single-threaded compressor into an all-cores compressor.

▌ TECHNICAL SPECIFICATIONS

Version — v0.3
Author — BLACKFIRE69
Platform — Windows x64, single exe (no dependencies)
File Size — ~1.5 MB
Compatibility — FreeArc v0.67 (and forks using arc.ini external compressors); also runs standalone from the command line or any tool that can pipe stdin/stdout

Wrapped-tool I/O modes (auto-detected from the method's command lines):

Code:

╭───────────────┬────────────────────────┬───────────────────────────────╮

│ Mode          │ Tool reads from        │ Tool writes to                │

├───────────────┼────────────────────────┼───────────────────────────────┤

│ Full piping   │ <stdin>                │ <stdout>                      │

│ stdin only    │ <stdin>                │ $$arcpackedfile$$ temp file   │

│ stdout only   │ $$arcdatafile$$ file   │ <stdout>                      │

│ Files only    │ $$arcdatafile$$ file   │ $$arcpackedfile$$ temp file   │

╰───────────────┴────────────────────────┴───────────────────────────────╯

▌ PERFORMANCE (v0.3 vs v0.2 vs v0.1)
256 MB of heterogeneous data compressed with zstd -17 as the wrapped single-threaded compressor. v0.3 was run with --progress=off --toolProgress=off; each version round-trips its OWN archive (SHA-256 verified identical).

Code:

╭──────────────────────┬─────────┬─────────┬─────────╮

│ Operation            │ v0.1    │ v0.2    │ v0.3    │

├──────────────────────┼─────────┼─────────┼─────────┤

│ encode -t10p  -c32m  │ 24.3 s  │ 24.1 s  │ 23.6 s  │

│ encode -t50p  -c32m  │ 15.7 s  │ 15.4 s  │ 14.4 s  │

│ encode -t100p -c32m  │ 12.7 s  │ 12.7 s  │ 11.1 s  │

│ encode -t100p -c8m   │ 11.9 s  │ 11.8 s  │ 10.6 s  │

│ encode -t100p -c4m   │ 13.3 s  │ 12.4 s  │ 11.0 s  │

│ decode -t100p -c32m  │ 1.1 s   │ 1.3 s   │ 0.7 s   │

╰──────────────────────┴─────────┴─────────┴─────────╯

v0.3 is faster in every configuration, and its lead grows with more threads and smaller chunks: v0.1's batch loop stalls each wave on its slowest chunk — it even slows down at -c4m — while v0.3's continuous reader/worker/writer pipeline never waits and stays flat at ~11 s. Decode is ~1.6× faster, and the margin widens further on machines with more cores.

▌ KEY FEATURES

✦ True Parallel Compression -t#
Continuous reader/workers/writer pipeline — chunks are read, compressed and written simultaneously by N parallel tool instances (-t#, default -t100p = one per logical core). The slowest chunk never stalls the rest.

✦ Guaranteed Stream Order
Output chunks are always written in the original input order, whatever order the workers finish in. FreeArc sees one opaque sequential stream.

✦ Memory Budgeting --mem=
--mem=8g sets a total RAM budget for chunks in flight (default: half of free RAM). --toolMem=2g declares how much RAM one tool instance needs — MTX then starts only as many instances as the machine can hold. A memory-hungry tool like rz can no longer freeze the whole system at -t100p.

✦ Safe Archive Format (MTX2)
Magic + version header, per-chunk frames carrying both sizes and a CRC32 checksum, explicit end marker with chunk count and total size. Damaged, truncated or foreign data is detected and reported — never silently unpacked into garbage.

✦ Independent Thread Counts
Every chunk frame is self-describing, so compression and decompression thread counts are fully independent. Pack with -t2 on a mid-range PC, unpack with -t16 on a high-end gaming PC.

✦ Live Status Display
A gray "#[#]" line above the MTX status shows the wrapped compressor's OWN live progress for the chunk currently gating the output; file-based tools get a per-chunk pseudo-console so they report progress even when piped. --progress=on|auto|off controls the whole display; --toolProgress=auto|on|off controls just the tool line. Errors always print regardless.

✦ Machine-Global Config (MTX.ini)
An optional MTX.ini next to the exe carries per-machine defaults (threads, chunk, mem, toolMem, okcodes, tempPath, basePath, cfgFile, progress, logs). Anything on the command line overrides it, so shared configs stay minimal and machine tuning lives next to the exe.

✦ Strict Error Contract
Tool failures, I/O errors and archive damage reach FreeArc as a nonzero exit code with a clear message: 0 = OK, 1 = runtime error, 2 = usage/config error, 3 = archive damaged. MTX never exits 0 after a failure, so FreeArc never finalizes a broken archive.

✦ Tolerant Tool Handling
--okcodes=0,2 accepts benign nonzero exit codes (precomp exits 2 when it finds nothing to precompress). Tool stderr is passed through, so the wrapped compressor's own messages stay visible.

✦ Collision-Free Temp Handling
Every parallel instance works in its own private temp directory — file-based tools and scratch-file writers (srep) cannot overwrite each other. --tempPath="X:\MyTemp" redirects all temp I/O to a chosen drive.

✦ Per-Method Defaults
In an MTX-only config file, a method section can carry its own defaults: okcodes, mtxthreads, mtxchunk, mtxmem. Command-line flags always win.

✦ Session Logging
--logs appends a timestamped session log (resolved options, per-chunk timings, child command lines) to MTX.log next to the exe.

▌ COMMAND REFERENCE

Code:

MTX64.exe <Command> <Options> [Settings] Input Output



Command:   a = encode (compress)   x = decode (extract)

Options:   -mXXX  method name from the config file

           -t#    parallel instances (-t100p = all cores, default)

           -c#    chunk size, e.g. -c64m (encode only; default 64 MB)

Settings:  --basePath=  --cfgFile=  --tempPath=  --mem=  --toolMem=

           --okcodes=  --progress=  --toolProgress=  --logs

Input/Output: file names, or "-" for stdin/stdout.

▌ QUICK START EXAMPLES

Example 1 — Wrap a method in FreeArc's arc.ini

Code:

; the plain single-threaded method

[External compressor:rz64]

header    = 0

packcmd   = Rz\rz.exe a -d 128m $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

unpackcmd = Rz\rz.exe x $$arcpackedfile$$.tmp $$arcdatafile$$.tmp



; the MTX-accelerated method

[External compressor:rzmt]

header    = 0

default   = -t100p --progress=on --toolMem=2g --basePath=".\..\..\" --cfgfile="$basePath$\arc.ini"

packcmd   = MTX\Win64\MTX64.exe a -mrz64 -c64m {options} - - <stdin> <stdout>

unpackcmd = MTX\Win64\MTX64.exe x -mrz64 {options} - - <stdin> <stdout>



arc.exe a -m=rzmt archive.arc files\*

Example 2 — Standalone, full piping

Code:

MTX64.exe a -mprecomp -c128m -t100p - - <stdin> <stdout>

MTX64.exe x -mprecomp -t100p - - <stdin> <stdout>

Example 3 — Memory-hungry tool, bounded RAM

Code:

MTX64.exe a -mrz -c64m -t100p --toolMem=2g --mem=8g - - <stdin> <stdout>

(starts only as many rz instances as 8 GB can hold at ~2 GB each)

Example 4 — Benign nonzero tool exit codes (precomp)

Code:

MTX64.exe a -mprecomp -c64m --okcodes=0,2 data.tmp packed.tmp

Example 5 — Per-method defaults in an MTX-only config file (MTX_Config.ini)

Code:

[External compressor:rz64]

header     = 0

packcmd    = Rz\rz.exe a -d 128m $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

unpackcmd  = Rz\rz.exe x $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

okcodes    = 0

mtxthreads = 4

mtxchunk   = 32m

mtxmem     = 2g



MTX64.exe a -mrz64 --cfgFile="X:\cfg\MTX_Config.ini" - - <stdin> <stdout>

Keep these keys OUT of FreeArc's own arc.ini — FreeArc rejects unknown keys in its sections. In arc.ini, pass --okcodes= / --toolMem= / … inside the MTX method's "default" line instead, as in Example 1.

Example 6 — Maximum throughput (MTX status on, tool line off)

Code:

MTX64.exe a -mzstd -c8m -t100p --progress=on --toolProgress=off - - <stdin> <stdout>

Keeps MTX's own live status but drops the wrapped tool's progress line — and with it the per-chunk pseudo-console that file-based tools need. For a fast compressor split into many small chunks that cost is measurable, so --toolProgress=off runs noticeably faster.

▌ DISTRIBUTION FILES

MTX64.exe — the tool (single exe, Win64)
MTX.ini — optional machine-global defaults (place next to the exe)
arc.ini — example FreeArc config with plain + MTX-wrapped methods (precomp, mpz, rz, nz, zstd, srep …)
README.txt — full documentation
WhatsNew.txt — version history

▌ NOTES

v0.3 archives use the MTX2 format; archives created by v0.1 must be repacked with the v0.3 build before migrating.
Compression and decompression machines never have to match in cores or RAM.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━

Feedback, bug reports, and edge cases are all welcome.

.

If anyone wants an example of "Inno setup" for MTX.exe, just let me know. ;)

MTX console version works perfectly. Also I was tested MCM works perfectly.

MTX.exe supports MASKS (Arc.Groups)

Quote:

example:

[Compression Methods]
MASK = srep+razorx/$zlib=precompx+srep+bcmx/$other=srep+razorx/$lyrfile=precompx+srep+bcmx/$bmp=srep+razorx

Quote:

Originally Posted by BLACKFIRE69 (Post 488133)

MTX.exe supports MASKS (Arc.Groups)

Can you explain about masks?

Quote:

Originally Posted by Balaji007 (Post 488138)

Can you explain about masks?

Masks allow you to use more than one method for different files in an archive

Different methods are specified with groups, which you can write yourself into the file "arc.groups" next to your arc.exe for compression.

Write your group as example:

Code:

$nameofgroup

*.filename

*.otherfile

Then we can use this in our compression method:

Code:

srep+lolz/$nameofgroup=rzx

So here, all of the files defined in the $nameofgroup list will be compressed using rzx, where all other files will be compressed with srep+lolz.

Quote:

Originally Posted by Masquerade (Post 488139)

Code:

$nameofgroup

*.filename

*.otherfile

Then we can use this in our compression method:

Code:

srep+lolz/$nameofgroup=rzx

So here, all of the files defined in the $nameofgroup list will be compressed using rzx, where all other files will be compressed with srep+lolz.

Ok thanks for your information friend

MTX.exe Feature update

MTX.exe Feature update v.3.0.0.0

What's New:

Some improvements.
A new key (MainPath) has been added to MTX.ini.
-- this allows you to set the root directory manually.
-- MTX.exe can be executed within a sub-folder.
There is no need to rename "mtx.ini" (as before).
-- before: MTX64.exe ---> MTX64.ini --> [MTX64]
-- now : MTX64.exe ---> MTX.ini --> [MTX]
Added an example for XLolz.
Added an example for MASKS.

Additional Options For both Compression and the Decompression:

Code:

-----------------------------------------------------------------------------

[MTX]

MainPath=..\..\

Config=arc.ini

//TmpPath=C:\Users\Rexton\Desktop\temptest\

-----------------------------------------------------------------------------







// config := default is arc.ini (current directory).



// config = arc.ini                        <--- MainPath\arc.ini

// config = cfg\config0.ini                <--- MainPath\cfg\config0.ini  







// TmpPath= full path.  ( with or without quotes (") )

Oh, thank you!
Can I have an example or small guide to adapt Inno/ASIS to this please? I'm probably doing something wrong with my frankenstein script lol

Edit/question: Is Xlolz working multithreading via MTX? It doesn't seem for me.
Edit2: It works with fixed thread count and not % like default (?)

Quote:

Originally Posted by Nezha (Post 488197)

You can't use "XLolz" via "MTX" (because both are accelerators). but it supports "MTX + Lolz".

ok, I'll give you some examples for both INNO Setup and Lolz with MTX.

Quote:

Originally Posted by BLACKFIRE69 (Post 488199)

You can't use "XLolz" via "MTX" (because both are accelerators). but it supports "MTX + Lolz".

ok, I'll give you some examples for both INNO Setup and Lolz with MTX.

Oh, yea, my bad: I mean the one inside the default "arc.ini" with MTX, for reference this one:

Quote:

[External compressor:XLolz, LolzMTX, xlolz]
header = 0
packcmd = bin\MTX\MTX.exe a:lolz -c:64m -t:100p - - <stdin> <stdout>
unpackcmd = bin\MTX\MTX.exe x:lolz -t:100p - - <stdin> <stdout>

___

So.. Something works but not well

Using unpack.bat

Quote:

Extracted 6,874 files, 10,021,997,180 => 16,726,226,492 bytes. Ratio 59.92%
Extraction time: cpu 30.00 sec/real 469.39 sec = 6%. Speed 35.63 mB/s
All OK

Using my modified InnoSetup scripts with MTX gives "UnArc.dll -11 error: archive damaged" at ~80% but it actually extracts something

Quote:

Originally Posted by Nezha (Post 488202)

Oh, yea, my bad: I mean the one inside the default "arc.ini" with MTX, for reference this one:

___

So.. Something works but not well

Using unpack.bat

Using my modified InnoSetup scripts with MTX gives "UnArc.dll -11 error: archive damaged" at ~80% but it actually extracts something

ok, just wait for the example.

MTX v.5.0.0.0 - October Update

MTX.ini

Code:

Example:=



exefile:  MTX64.exe

ini:      supports both MTX.ini or MTX64.ini    (next to the exe file)

section:  [MTX] or [MTX64]



exefile:  MTX32.exe

ini:      supports both MTX.ini or MTX32.ini    (next to the exe file)

section:  [MTX] or [MTX32]

Percentage

Code:

Example:=



packcmd   = "MTX.exe" a:bcm -c:64m -t:100p - - <stdin> <stdout>



   or



packcmd   = "MTX.exe" a:bcm -c:64m -t:100% - - <stdin> <stdout>

Supports Custom Input/Output

Code:

Example:=



[External compressor:bcm, bcm32]

header    = 0 

default   = -9



packcmd   = "Bcm\{compressor}" {options} MyInput.in MyOutput.out

unpackcmd = "Bcm\{compressor}" -d MyOutput.out MyInput.in

datafile   = MyInput.in

packedfile = MyOutput.out

New Options

Code:



1. --help   (Show long help)

2. --logs   (Generate a log file)

3. --mainpath=< main path >

4. --config=< config file >

5. --tmppath=< temp path >





Example:=  (When not using MTX.ini.)





[External compressor:BscX, BscMTX, bscx]

header = 0

default   = --logs --mainpath=".\..\" --config="%MainPath%\confis\arc.ini" --tmppath="%ExePath%\TEMP"

packcmd   = MTX\MTX.exe a:bsc32 -c:64m -t:100% {options} - - <stdin> <stdout>

unpackcmd = MTX\MTX.exe x:bsc32 -t:100p  {options} - - <stdin> <stdout>

MTX.ini Configuration

Code:

 MTX Variables:

    1. %MainPath%

         -- default is the directory that containing MTX.exe.

            ex: C:\Testing\mtxTest\Res\MTX\MTX.exe  then %MainPath% = C:\Testing\mtxTest\Res\MTX\



         -- Users can change manually.



    2. %ExePath%

         -- the directory that containing MTX.exe.

            ex: C:\Testing\mtxTest\Res\MTX\MTX.exe  then %ExePath% = C:\Testing\mtxTest\Res\MTX\





 Other

    1. MainPath

        -- default is         %ExePath%



        -- Users can change manually.



        -- MainPath = (full path)

            ex:        MainPath=C:\Testing\mtxTest\Res\



        -- MainPath = (short path)

            ex:        MainPath=.\..\

              > MainPath is centered on ExePath. (for short paths)

              > If ExePath = C:\Testing\mtxTest\Res\MTX\  then  MainPath = C:\Testing\mtxTest\Res\  (MainPath=.\..\)



        -- MainPath = (alternatively)

            ex:         MainPath=%ExePath%





    2. Config

        -- default is        %ExePath%\Arc.ini



        -- Users can change manually.



        -- Config = (full path)

            ex:        Config=C:\Testing\mtxTest\Res\Arc.ini



        -- Config = (short path)

            ex:        Config=.\..\Arc.ini

               > Config is centered on MainPath. (for short paths)

               > If MainPath = C:\Testing\mtxTest\Res\MTX\  then  Config = C:\Testing\mtxTest\Res\Arc.ini  (Config=.\..\Arc.ini)



        -- Config = (alternatively)

            ex:        Config=%MainPath%\Arc.ini

            ex:        Config=%ExePath%\Arc.ini





    3. TmpPath

        -- default is        %ExePath%



        -- Users can change manually.



        -- TmpPath = (full path)

            ex:        TmpPath=C:\Testing\mtxTest\Res\TEMP\



        -- TmpPath = (short path)

            ex:        TmpPath=.\..\TEMP\

               > TmpPath is centered on MainPath. (for short paths)

               > If MainPath = C:\Testing\mtxTest\Res\MTX\  then  Config = C:\Testing\mtxTest\Res\TEMP\  (TmpPath=.\..\TEMP\)



        -- TmpPath = (alternatively)

            ex:        TmpPath=%MainPath%\TEMP\

            ex:        TmpPath=%ExePath%\TEMP\

Will show some errors in nanozip decompression with MTX.exe. therefore it is better to use cls-nzx.dll instead of MTX.exe. ;)

guys, I don't have enough space to upload it here. so use this link....

MTX v5.0.0.0 + Full Examples.rar

_

@BLACKFIRE69, very good the new key default options in example 5.
It will be very useful for the need to use MRX.ini.
Thanks for the great job.

If you have both configurations, which one will be used?
I mean if you have MTX.ini configured in one way and use the key default with --mainpath, etc. in another way, which one will be used?

Quote:

Originally Posted by Cesar82 (Post 488276)

first configured file (MTX.ini) then checks the default keys (--mainpath , ...).

means,

If MTX.ini is not configured, it uses the default keys.

Quote:

Originally Posted by BLACKFIRE69 (Post 488277)

first configured file (MTX.ini) then checks the default keys (--mainpath , ...).

means,

If MTX.ini is not configured, it uses the default keys.

Just out of curiosity, if I use it like this:

DiskSpan_x64.ini:

Code:

[External compressor:mpz,MPZ,mpzapi,MPZAPI]

header    = 0

packcmd   = "MEDIA\MPZ\mpzapi.exe" c <stdin> <stdout>

unpackcmd = "MEDIA\MPZ\mpzapi.exe" d <stdin> <stdout>



[External compressor:mpz_mtx,MPZ_MTX,mpzapi_mtx,MPZAPI_MTX]

header    = 0

default   = -c:64m -t:100p --mainpath=".\..\..\..\" --config="%MainPath%\DiskSpan_x64.ini" --tmppath="%MainPath%\..\TEMP"

packcmd   = "OTHERS\MTX\Win64\MTX.exe" a:mpz {options} - - <stdin> <stdout>

unpackcmd = "OTHERS\MTX\Win64\MTX.exe" x:mpz {options} - - <stdin> <stdout>

And add MTX.ini file next to MTX.exe with the config:

MTX.ini

Code:

[MTX]

MainPath=.\..\..\..\

Config=%MainPath%\DiskSpan_x64.ini

TmpPath=%MainPath%\..\TEMP

Question is as follows:
Will the "-c:64m -t:100p" settings be applied to MTX.exe, or will they be ignored?

Thanks!

Quote:

Originally Posted by Cesar82 (Post 488278)

Just out of curiosity, if I use it like this:

DiskSpan_x64.ini:

Code:

[External compressor:mpz,MPZ,mpzapi,MPZAPI]

header    = 0

packcmd   = "MEDIA\MPZ\mpzapi.exe" c <stdin> <stdout>

unpackcmd = "MEDIA\MPZ\mpzapi.exe" d <stdin> <stdout>



[External compressor:mpz_mtx,MPZ_MTX,mpzapi_mtx,MPZAPI_MTX]

header    = 0

default   = -c:64m -t:100p --mainpath=".\..\..\..\" --config="%MainPath%\DiskSpan_x64.ini" --tmppath="%MainPath%\..\TEMP"

packcmd   = "OTHERS\MTX\Win64\MTX.exe" a:mpz {options} - - <stdin> <stdout>

unpackcmd = "OTHERS\MTX\Win64\MTX.exe" x:mpz {options} - - <stdin> <stdout>

And add MTX.ini file next to MTX.exe with the config:

MTX.ini

Code:

[MTX]

MainPath=.\..\..\..\

Config=%MainPath%\DiskSpan_x64.ini

TmpPath=%MainPath%\..\TEMP

Question is as follows:
Will the "-c:64m -t:100p" settings be applied to MTX.exe, or will they be ignored?

Thanks!

those settings are not ignored ( -c:64m -t:100p ).

ok cesar,

we use MTX.ini to define only "MainPath", "ConfigFile" and "TempPath".

let us consider,
if MTX.exe doesn't find any MTX.ini near MTX.exe, it will look for MTX.exe options ( --mainpath, --config and --tmppath). if those options are also not found, it uses its default values.

in your case, there is MTX.ini next to MTX.exe. therefore, the "--mainpath", "--config" and "--tmppath" options will be ignored.

but the only place to set values for "-t:" and "-c:" is the command line. so nothing can ignore those values.

finally, MTX.exe will ignore some options as the code below because you have used MTX.ini. (Marked in red).

Code:

[External compressor:mpz_mtx,MPZ_MTX,mpzapi_mtx,MPZAPI_MTX]

header    = 0

default   = -c:64m -t:100p --mainpath=".\..\..\..\" --config="%MainPath%\DiskSpan_x64.ini" --tmppath="%MainPath%\..\TEMP"

packcmd   = "OTHERS\MTX\Win64\MTX.exe" a:mpz {options} - - <stdin> <stdout>

unpackcmd = "OTHERS\MTX\Win64\MTX.exe" x:mpz {options} - - <stdin> <stdout>

Extra: (not related to the question)

during decompression, MTX.exe doesn't retrieve values for "ChunkSize" from users. therefore it is better to use as follows. :)

Code:

[External compressor:mpz_mtx,MPZ_MTX,mpzapi_mtx,MPZAPI_MTX]

header    = 0

default   = -t:100p

packcmd   = "OTHERS\MTX\Win64\MTX.exe" a:mpz -c:64m {options} - - <stdin> <stdout>

unpackcmd = "OTHERS\MTX\Win64\MTX.exe" x:mpz {options} - - <stdin> <stdout>

Help BlackFire69

first thing thanks to you for this tool and other cls
i need IS example to unpack compressed data by mtx and thanks

Quote:

Originally Posted by Snapppr6 (Post 488332)

first thing thanks to you for this tool and other cls
i need IS example to unpack compressed data by mtx and thanks

here is the basic inno example for MTX v5.0.0.0.

note that the two arc.ini files in the "Compressor" folder and the "IS_Example" folder are different. ;)

MTX v5.0.0.0 Inno Example.rar

MTX v5.1.0.0 New Update Test.

MTX v5.1.0.0 New Update Test

it's time for a new update. :D you guys, can you do some testing for me?

TESTS:
MTX + oo2reck

MTX + oo2recm

MTX + oo2recs

Arc.ini

Code:

; //================== FreeArc =================\\

[External compressor:oo2reck,oo2recm,oo2recs]

header    = 0

packcmd   = {compressor} e <stdin> <stdout>

unpackcmd = {compressor} d <stdin> <stdout>





; //==================== MTX ====================\\

[External compressor:MTX_oo2reck, oo2reckMTX, oo2reckX]

header = 0

default   = -t:100% --logs

packcmd   = MTX.exe a:oo2reck -c:64m {options} - - <stdin> <stdout>

unpackcmd = MTX.exe x:oo2reck {options} - - <stdin> <stdout>



[External compressor:MTX_oo2recm, oo2recmMTX, oo2recmX]

header = 0

default   = -t:100% --logs

packcmd   = MTX.exe a:oo2recm -c:64m {options} - - <stdin> <stdout>

unpackcmd = MTX.exe x:oo2recm {options} - - <stdin> <stdout>



[External compressor:MTX_oo2recs, oo2recsMTX, oo2recsX]

header = 0

default   = -t:100% --logs

packcmd   = MTX.exe a:oo2recs -c:64m {options} - - <stdin> <stdout>

unpackcmd = MTX.exe x:oo2recs {options} - - <stdin> <stdout>

i'll attach the MTX v5.1.0.0 test files.

thanks @dixen for the idea.

_

Code:

MTX

Compressed 1 file, 119,898,862 => 168,186,700 bytes. Ratio 140.27%

Compression time: cpu 0.13 sec/real 8.59 sec = 1%. Speed 13.96 mB/s

All OK



Extracted 1 file, 168,186,700 => 119,898,862 bytes. Ratio 140.27%

Extraction time: cpu 0.19 sec/real 3.90 sec = 5%. Speed 30.74 mB/s

All OK





No MTX

Compressed 1 file, 119,898,862 => 168,186,552 bytes. Ratio 140.27%

Compression time: cpu 0.14 sec/real 10.16 sec = 1%. Speed 11.80 mB/s

All OK



Extracted 1 file, 168,186,552 => 119,898,862 bytes. Ratio 140.27%

Extraction time: cpu 0.17 sec/real 3.84 sec = 4%. Speed 31.26 mB/s

All OK

Thanks to BLACKFIRE69 for update but...tests..Not particularly impressive

patchpack002\dlc.rpf

MTX+OO2RECK (pack)

Quote:

Compressed 1 file, 1,060,477,032 => 1,820,126,829 bytes. Ratio 171.63%
Compression time: cpu 1.28 sec/real 186.69 sec = 1%. Speed 5.68 mB/s
All OK

MTX+OO2RECK (Unpack)

Quote:

Extracted 1 file, 1,820,126,829 => 1,060,477,032 bytes. Ratio 171.63%
Extraction time: cpu 2.11 sec/real 132.35 sec = 2%. Speed 8.01 mB/s
All OK

Razor's oo2reck (Pack)

Quote:

Compressed 1 file, 1,060,477,032 => 1,820,656,947 bytes. Ratio 171.68%
Compression time: cpu 1.19 sec/real 216.05 sec = 1%. Speed 4.91 mB/s
All OK

Razor's oo2reck (unpack)

Quote:

Extracted 1 file, 1,820,656,947 => 1,060,477,032 bytes. Ratio 171.68%
Extraction time: cpu 1.77 sec/real 125.85 sec = 1%. Speed 8.43 mB/s
All OK

I'll test this on DOOM Eternal & Horizon Zero Dawn..

Guys, try with more chunk sizes.

-c:128mb
-c:256mb
-c:512mb

Same file, chunk size 256

Unpack

Quote:

Extracted 1 file, 1,820,527,500 => 1,060,477,032 bytes. Ratio 171.67%
Extraction time: cpu 2.06 sec/real 117.64 sec = 2%. Speed 9.01 mB/s
All OK

chunk size 512

Quote:

Extracted 1 file, 1,820,527,500 => 1,060,477,032 bytes. Ratio 171.67%
Extraction time: cpu 2.09 sec/real 123.59 sec = 2%. Speed 8.58 mB/s
All OK

Quote:

Originally Posted by dixen (Post 488630)

Same file, chunk size 256

Unpack

chunk size 512

It seems to be MTX is 0.58 mb/s faster than the original version right ? ( for chunk = 256mb )

Quote:

Originally Posted by BLACKFIRE69 (Post 488631)

It seems to be MTX is 0.58 mb/s faster than the original version right ? ( for chunk = 256mb )

Yeah it looks like the idea failed

MTX + Lolz may work well in high chunk sizes.

Isn't the oodle precompressor multi threaded already? :confused:

I made the tool in such a way that there no streams being cut off when chunking takes place because the program knows what it is doing and how it should make cuts in the chunks, if for example there is a stream that can be cut off at a certain position, the program self adjusts the position and your chunking method makes no such considerations and gets in the way of how the program should work in the first place which makes it prone to crashes or a drop in ratio. :o

Multi threading a multi threaded program does not make it faster especially if the main program uses all of the cpu, furthermore your MTX does not prepare chunks in advance which could be the reason it doesn't work very well.

Compare the speeds of MTX vs Freearc's 4x4

I'll also leave this here
Attachment 28232

I can't believe this community has been reduced to this, I don't know if I should find this funny or be sad that there are no new inventions will be made. :(

I would totally understand if the community gets excited every time EA Sports releases FIFA every year :D

no more developments. it doesn't seem to be useful.

@mods, plz delete this thread.

I wonder what makes you cease the development of your project, I just pointed out that using MTX on already multi threaded programs may have such an issue of being slower than when run natively. I wouldn't have said anything if you had figured out a smarter way of making the precompressor making sure it stays at 100% because there are times when it doesn't fully utilize the cpu furthermore there is also overutilization because the precompressor would be using all of the cpu while MTX is also making several instances of the program and now you have the problem of system being unresponsive. Your project has its uses just not handling multi threaded programs, you'd need to come up with a better solution for this.

What people thinks development is? Remaking old projects with more bugs and making non sense project is not development. If you can make something new that's development, even if you remake and make something better than before that's development. Posting same thing over and over again by changing a bit of information is spamming, not development.

Sorry for bothering mr.BLACKFIRE, This may be a useful thread but we can't find the download section here. so please reupload it.
Thank you.

Hi,

I tried to use the settings in the snapshot, which should be using precomp048.exe

but when I test a packing, method called is

MTXprecomp048
-> calls method precomp048
-> calls precomp048.exe

however it is reporting error with precomp038.exe in the same directory.

It is MTX 5.0.0.0

https://i.ibb.co/qxF1j5m/2021-02-07-031238.png

Quote:

Originally Posted by github (Post 490321)

Precomp v0.38 requires "packjpg_dll.dll" and "packjpg_dll1.dll" next to the precomp038.exe.

Thanks, that could explain part of it.
But why it would invoke precomp038.exe?
Is MTX parsing every external compressor defined in arc.ini as well as try to load them once to build some kind of lookup table into memory for later use?

Quote:

; ========== for MTX testing ================

[External compressor:rz, razor, rz64]
header = 0
default = -d 128m
; packcmd = "Rz\RZ.exe" a {options} $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
; unpackcmd = "Rz\RZ.exe" e -y $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
; or

packcmd = "Rz\RZ.exe" a {options} MyOutput.out MyInput.in
unpackcmd = "Rz\RZ.exe" e -y MyOutput.out MyInput.in
datafile = MyInput.in
packedfile = MyOutput.out

[External compressor:bcm, bcm32]
header = 0
default = -9
; packcmd = "Bcm\{compressor}" {options} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
; unpackcmd = "Bcm\{compressor}" -d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
; or

packcmd = "Bcm\{compressor}" {options} MyInput.in MyOutput.out
unpackcmd = "Bcm\{compressor}" -d MyOutput.out MyInput.in
datafile = MyInput.in
packedfile = MyOutput.out

[External compressor:lolz, lolz64]
header = 0
default = -dt -dtb1 -dtw1 -dtm1 -dto1 -dtd1 -mtt1 -mt4 -d64m -fba4096
; packcmd = Lolz\lolz_x64.exe {options} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
; unpackcmd = Lolz\lolz_x64_u.exe $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
; or

packcmd = Lolz\lolz_x64.exe {options} MyInput.in MyOutput.out
unpackcmd = Lolz\lolz_x64_u.exe MyOutput.out MyInput.in
datafile = MyInput.in
packedfile = MyOutput.out

[External compressor:RazorX, RazorMTX, razorx]
header = 0
default = -c:64m -t:100% --logs
packcmd = "MTX.exe" a:rz {options} - - <stdin> <stdout>
unpackcmd = "MTX.exe" x:rz -t:100p --logs - - <stdin> <stdout>

[External compressor:BcmX, BcmMTX, bcmx]
header = 0
default = -c:64m -t:100% --logs
packcmd = "MTX.exe" a:bcm {options} - - <stdin> <stdout>
unpackcmd = "MTX.exe" x:bcm -t:100p --logs - - <stdin> <stdout>

[External compressor:XLolz, LolzMTX, xlolz]
header = 0
default = -c:64m -t:100% --logs
packcmd = MTX.exe a:lolz {options} - - <stdin> <stdout>
unpackcmd = MTX.exe x:lolz -t:100p --logs - - <stdin> <stdout>

[External compressor:precomp]
header = 0
default = -cn
packcmd = "Precomp\{compressor}" {options} - - <stdin> <stdout>
unpackcmd = "Precomp\{compressor}" -r -o$$arcdatafile$$.tmp $$arcpackedfile$$.tmp
datafile = $$arcdatafile$$.tmp
packedfile = $$arcpackedfile$$.tmp
; or

;packcmd = "Precomp\{compressor}" {options} -oMyOutput.out MyInput.in
;unpackcmd = "Precomp\{compressor}" -r -oMyInput.in MyOutput.out
;datafile = MyInput.in
;packedfile = MyOutput.out

[External compressor:PrecompX, PrecompMTX, precompx]
header = 0
default = -c:64m -t:50% --logs
packcmd = MTX.exe a:precomp {options} - - <stdin> <stdout>
unpackcmd = MTX.exe x:precomp -t:100p --logs - - <stdin> <stdout>

I gave the default example and some random files a go:

precomp048+srep+lolz64 works
precomp048+srep+LolzMTX ERROR: general (de)compression error in LolzMTX

precomp048+srep+rz64 works
precomp048+srep+RazorMTX ERROR: general (de)compression error in RazorMTX

precomp048+srep+bcm32 ERROR: general (de)compression error in bcm32
precomp048+srep+BcmMTX works

It is a bit strange that, if not using MTX it works, using MTX will make it not working.
If not using MTX doesn't work, using MTX will make it work?
Is it because of the input output in the settings?

Another test for
PrecompMTX+srep+ any normal or MTX final compressor

none works.

The console / log doesn't say why at all.
It seems as soon as MTX invokes precomp, it exits/crashes.
The temp folder for precomp has been created, only the 0 folder has 512KB file, all others are 0

MTX New Update v5.0.0.1

What's new: MTX v5.0.0.1

Some improvements.
Minor bugs fixed.
Exception handling.
MTX.ini is no longer supported by default.
{compressor} keyword is no longer supported.
A few more changes:
1. %exePath% ==> $exePath$
2. %basePath% ==> $basePath$

Example contains:

Bcm
Bsc
Lolz
Mcm
Mpz
Nanozip
Precomp
Razor
Uharc
Zcm
Zstd

^^
Thanks for the MTX update.
But in freearc alpha build struggled with Antivirus:(.

Code:

https://www.virustotal.com/gui/file/...5ac8/detection

If we turnoff all Ok:rolleyes:

Quote:

Originally Posted by ffmla (Post 490528)

^^
Thanks for the MTX update.
But in freearc alpha build struggled with Antivirus:(.

Code:

https://www.virustotal.com/gui/file/...5ac8/detection

If we turnoff all Ok:rolleyes:

Maybe because of UPX. He has compressed all exe files with UPX?

Quote:

Originally Posted by ffmla (Post 490528)

^^
Thanks for the MTX update.
But in freearc alpha build struggled with Antivirus:(.

Code:

https://www.virustotal.com/gui/file/...5ac8/detection

If we turnoff all Ok:rolleyes:

it's common to see false positives due to upx. I didn't include any trojans with them. :p