ADDeX — Advanced Data Deduplication Engine

[panker1992]

ADDeX — Advanced Data Deduplication Engine

I’m working on a new data deduplication and compression tool called ADDeX (Advanced Data Deduplication Engine).

The goal is to experiment with different levels of duplicate-data detection and compression, with a focus on achieving better compression ratios while allowing the user to control the trade-off between speed, memory usage, and compression efficiency.
Current compression modes

M1 — Standard: General-purpose compression with reasonable speed and memory usage.
M2 — Improved: More aggressive matching for better compression ratios.
M3 — Exhaustive: Performs much more intensive matching for maximum compression potential. This mode is significantly more RAM-intensive and can be considerably slower.

Available options

ADDeX.exe [OPTIONS] <command> <input> <output>

Commands:
-m1 <input> <output> Standard compression mode
-m2 <input> <output> Improved compression mode
-m3 <input> <output> Exhaustive compression mode
-d <input> <output> Decompress a file
-d - - Decompress stdin to stdout
-h Display help

Compression options:
-l<N> Minimum match length
Default: 512 | Min: 64 | Max: 1024

-a<N> Alignment
Default: 0 | Min: 0 | Max: 2

-b<N> Block size in MB
Default: 8 MB

Examples

ADDeX.exe -m1 input.dat output.addex

ADDeX.exe -m2 -l64 -a1 data.bin output.addex

ADDeX.exe -m3 -l512 -b16 data.bin output.addex

ADDeX.exe -d compressed.addex original.bin

type compressed.addex | ADDeX.exe -d - - > output.bin

- can be used as a filename to read from stdin or write to stdout, making ADDeX suitable for command-line pipelines.

Version: 0.1 Alpha
Author: Panker1992

This is an early alpha release, so the format, algorithms, command-line options, and compression behavior are still subject to change. Feedback, testing results, and suggestions are welcome.