ADDeX — Advanced Data Deduplication Engine

[panker1992]

ADDeX — Advanced Data Deduplication Engine

I’m working on a new data deduplication and compression tool called ADDeX (Advanced Data Deduplication Engine).

The goal is to experiment with different levels of duplicate-data detection and compression, with a focus on achieving better compression ratios while allowing the user to control the trade-off between speed, memory usage, and compression efficiency.
Current compression modes

M1 — Standard: General-purpose compression with reasonable speed and memory usage.
M2 — Improved: More aggressive matching for better compression ratios.
M3 — Exhaustive: Performs much more intensive matching for maximum compression potential. This mode is significantly more RAM-intensive and can be considerably slower.

Available options

ADDeX.exe [OPTIONS] <command> <input> <output>

Commands:
-m1 <input> <output> Standard compression mode
-m2 <input> <output> Improved compression mode
-m3 <input> <output> Exhaustive compression mode
-d <input> <output> Decompress a file
-d - - Decompress stdin to stdout
-h Display help

Compression options:
-l<N> Minimum match length
Default: 512 | Min: 64 | Max: 1024

-a<N> Alignment
Default: 0 | Min: 0 | Max: 2

-b<N> Block size in MB
Default: 8 MB

Examples

ADDeX.exe -m1 input.dat output.addex

ADDeX.exe -m2 -l64 -a1 data.bin output.addex

ADDeX.exe -m3 -l512 -b16 data.bin output.addex

ADDeX.exe -d compressed.addex original.bin

type compressed.addex | ADDeX.exe -d - - > output.bin

- can be used as a filename to read from stdin or write to stdout, making ADDeX suitable for command-line pipelines.

Version: 0.1 Alpha
Author: Panker1992

This is an early alpha release, so the format, algorithms, command-line options, and compression behavior are still subject to change. Feedback, testing results, and suggestions are welcome.

[shazzla]

For a few days im away from my computer.
So....
Whats the de/compression speed compared to srep?

[panker1992]

i have no clue i never compared to srep, encoding speed should be 15% faster and decode hould be on par

you guys can run this and tell me

[FarhanFaudTahix]

is it possible to provide a 32 bit cls for freearc. and is the cls working without ADDeX.exe.

[panker1992]

32 bit DLL has been included, and yes the DLL can extract alone, but it cant encode...

is has only the decompression code inside.

I will run benchmarks today for people to compare

[kj911]

CLS filters, not works... Max. 16-32MB input data sizes still probably workingly in good scenarios/cases, upto 100MB+ data compressed it, not working.

And different cases, reactions from Win7 x64 SP1:

64bit CLS: Thread blocked indefinitely.
32bit CLS: Few percentage or first ~8MB's or rare cases in below 16MB data extracted, largers not, crashing now after 8MB data readed/writed in.

Only workingly use aDDeX.exe file from testing/decompress. /properly configured comp/decomp commands from arc.ini file./

Tested methods: xZLib+addex:m3+lzma or addex:m3+lzma

Note: xZLib = Old XTool 0.12 32bit versions.

[panker1992]

I am working on my own CLS filter written in delphi, this CLS was kindly provided by a friend

Will be fixed within the day, you can use the addex to decompress your file, and if you have files that failed in srep you can try to see, rumors said that it worked

[Razor12911]

Some benchmarks, tests performed on ramdisk.

input: Bright Memory Infinite (zlib precompressed), 20,6 GB (22 175 074 282 bytes)

Srep

srep (encoding -m3f): Clock 51.695s (CPU 58.467s), Peak RAM usage 1569 MB, Output 15,0 GB (16 112 713 354 bytes)
srep (decoding): Clock 10.785s (CPU 10.468s), Peak RAM usage 1007 MB
lzma compression: Final output 7,26 GB (7 796 298 754 bytes)

Addex

addex (encoding -m1): Clock 85.233s (CPU 92.202s), Peak RAM usage 1797 MB, Output 15,0 GB (16 112 671 046 bytes)
addex (decoding): Clock 9.790s (CPU 9.561s), Peak RAM usage 989 MB
lzma2 compression: Final output 7,25 GB (7 796 022 132 bytes)

procprofile srep

encoding

Process ID       : 26296
Thread ID        : 19848
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          52.421s
Kernel Time      :           6.046s
Process Time     :          58.467s
Clock Time       :          51.695s

Working Set      :         1606240 KB
Paged Pool       :             150 KB
Nonpaged Pool    :              11 KB
Pagefile         :         1599976 KB
Page Fault Count : 432691

IO Read          :        43310706 KB (in            5303 reads )
IO Write         :        15735101 KB (in           38937 writes)
IO Other         :              24 KB (in           31358 others)

decoding 

Process ID       : 11320
Thread ID        : 5852
Process Exit Code: 0
Thread Exit Code : 0

User Time        :           4.843s
Kernel Time      :           5.625s
Process Time     :          10.468s
Clock Time       :          10.785s

Working Set      :         1031588 KB
Paged Pool       :             149 KB
Nonpaged Pool    :             136 KB
Pagefile         :         1034640 KB
Page Fault Count : 259090

IO Read          :        15735170 KB (in            8090 reads )
IO Write         :        21655353 KB (in            9877 writes)
IO Other         :              21 KB (in            7414 others)

procprofile addex

encoding

Process ID       : 24816
Thread ID        : 20988
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          86.859s
Kernel Time      :           5.343s
Process Time     :          92.202s
Clock Time       :          85.233s

Working Set      :         1840540 KB
Paged Pool       :             111 KB
Nonpaged Pool    :              12 KB
Pagefile         :         1846388 KB
Page Fault Count : 502279

IO Read          :        43310691 KB (in            5289 reads )
IO Write         :        15735033 KB (in           15455 writes)
IO Other         :              24 KB (in             156 others)

decoding

Process ID       : 23824
Thread ID        : 3120
Process Exit Code: 0
Thread Exit Code : 0

User Time        :           3.921s
Kernel Time      :           5.640s
Process Time     :           9.561s
Clock Time       :           9.790s

Working Set      :         1011880 KB
Paged Pool       :             110 KB
Nonpaged Pool    :              98 KB
Pagefile         :         1010832 KB
Page Fault Count : 1503488

IO Read          :        15735074 KB (in            7579 reads )
IO Write         :        21655346 KB (in           21158 writes)
IO Other         :               0 KB (in              49 others)

[kj911]

I retested again... Now it's good.

What's certain: The 64bit CLS filter is completely inoperable.

During the afternoon test, during packaging, the 64bit version of XTOOL v0.12 was used (which I wrote as "xZLib"), including the hif/raw*.dll and zlibwapi.dll files. During unpacking, the 32bit version. (From the Xeon PC.)

Furthermore, the unpacking/testing via the "aDDeX.exe" file runs without error, with the 32bit CLS file, although not completely, similarly to the previous case. If there is no xZLib at the beginning of the packaging chain, then there is no error, strange mode. In one of the two smaller images you can see this (test from laptop) + the 64bit CLS error, in the larger image, the other test case.







For those who are experimenting, here is the small prepared package, in the "DATA" folder in the "pack" folder, copy at least 100-500MB of data or even more. Copy/move the created "data.arc" file to the other 3 folders, running the tests one after the other.

For those who don't have enough, you can also copy additional pre- and other compressors to the folders, expanding and editing the "arc.ini" files accordingly, for experimentation. Or add them to the existing test/compressor sets and test them. (eg: Tiny Compressor, Diskspan GUI, etc...)

Plus info: The 32bit CLS-filter, running from XP! (Goood!)