ADDeX — Advanced Data Deduplication Engine
 

ADDeX — Advanced Data Deduplication Engine

I’m working on a new data deduplication and compression tool called ADDeX (Advanced Data Deduplication Engine).

The goal is to experiment with different levels of duplicate-data detection and compression, with a focus on achieving better compression ratios while allowing the user to control the trade-off between speed, memory usage, and compression efficiency.
Current compression modes

M1 — Standard: General-purpose compression with reasonable speed and memory usage.
M2 — Improved: More aggressive matching for better compression ratios.
M3 — Exhaustive: Performs much more intensive matching for maximum compression potential. This mode is significantly more RAM-intensive and can be considerably slower.

Available options

ADDeX.exe [OPTIONS] <command> <input> <output>

Commands:
-m1 <input> <output> Standard compression mode
-m2 <input> <output> Improved compression mode
-m3 <input> <output> Exhaustive compression mode
-d <input> <output> Decompress a file
-d - - Decompress stdin to stdout
-h Display help

Compression options:
-l<N> Minimum match length
Default: 512 | Min: 64 | Max: 1024

-a<N> Alignment
Default: 0 | Min: 0 | Max: 2

-b<N> Block size in MB
Default: 8 MB

Examples

ADDeX.exe -m1 input.dat output.addex

ADDeX.exe -m2 -l64 -a1 data.bin output.addex

ADDeX.exe -m3 -l512 -b16 data.bin output.addex

ADDeX.exe -d compressed.addex original.bin

type compressed.addex | ADDeX.exe -d - - > output.bin

- can be used as a filename to read from stdin or write to stdout, making ADDeX suitable for command-line pipelines.

Version: 0.1 Alpha
Author: Panker1992

This is an early alpha release, so the format, algorithms, command-line options, and compression behavior are still subject to change. Feedback, testing results, and suggestions are welcome.

For a few days im away from my computer.
So....
Whats the de/compression speed compared to srep? :)

i have no clue :) i never compared to srep, encoding speed should be 15% faster and decode hould be on par

you guys can run this and tell me :)

is it possible to provide a 32 bit cls for freearc. and is the cls working without ADDeX.exe.

32 bit DLL has been included, and yes the DLL can extract alone, but it cant encode...

is has only the decompression code inside.

I will run benchmarks today for people to compare

CLS filters, not works... Max. 16-32MB input data sizes still probably workingly in good scenarios/cases, upto 100MB+ data compressed it, not working.

And different cases, reactions from Win7 x64 SP1:

64bit CLS: Thread blocked indefinitely.
32bit CLS: Few percentage or first ~8MB's or rare cases in below 16MB data extracted, largers not, crashing now after 8MB data readed/writed in.

Only workingly use aDDeX.exe file from testing/decompress. /properly configured comp/decomp commands from arc.ini file./

Tested methods: xZLib+addex:m3+lzma or addex:m3+lzma

Note: xZLib = Old XTool 0.12 32bit versions.

I am working on my own CLS filter written in delphi, this CLS was kindly provided by a friend :)

Will be fixed within the day, you can use the addex to decompress your file, and if you have files that failed in srep you can try to see, rumors said that it worked

Some benchmarks, tests performed on ramdisk.

input: Bright Memory Infinite (zlib precompressed), 20,6 GB (22 175 074 282 bytes)

Srep
Addex
procprofile srep

procprofile addex