Best Compression Methods for 'Specific' Games. Q&A

[panker1992]

Originally Posted by shazzla

# arc a -i2 -mdedup p:\ded.arc e:\shdw.xt
FreeArc 0.67 (March 15 2014) creating archive: p:\ded.arc
Compressing 1 file, 31,837,733,330 bytes
Compressing shdw.xt 99.9%
Compressing 31,837,733,330 bytes with dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
100.0%=== DEDUPLICATION (XXH3-128) ===
Pass 1: Scanning for block positions (minimal RAM)
Processing 29.65 GB input file...
Progress: 100.00%
Starting collision-free frequency-based deduplication...
Pass 1 complete: 44006478 blocks, 37991985 unique XXH3 hashes
100.0%Collision-Free Analysis: 6014493 duplicate blocks, 2938568 unique hashes cached
Pass 1.75: Loading ONLY duplicate block data into RAM...
100.0%Loaded 2938568 unique duplicate blocks into RAM
100.0%RAM usage: ~5.3% of original data (duplicates only)
Pass 2: Writing all blocks with XXH3 collision-free hints...
100.0%=== DEDUPLICATION RESULTS (XXH3-128) ===
Original Size: 29.65 GB (31837733330 bytes)
Compressed Size: 26.74 GB (28713904996 bytes)
Compression Ratio: 90.19% (1.11:1)
Space Saved: 2.91 GB (9.8%)
Total Blocks: 44006478
Unique Blocks: 37991985 (86.3%)
Duplicate Blocks: 6014493 (13.7%)
Cached Blocks: 2938568 (7.7% of unique)
Hash Algorithm: XXH3-128 (Collision-Free)
RAM breakdown: Tabs=1.31 GB, FreqMap=869.57 MB, Duplicates=1.58 GB, Total=3.74 GB (12.6% of input)
Collision-free deduplication complete!
100.0%
Errorlevel=0

Compressed 1 file, 31,837,733,330 => 28,713,904,996 bytes. Ratio 90.19%
Compression time: cpu 8.25 sec/real 1506.37 sec = 1%. Speed 21.14 mB/s
All OK

# arc t p:\ded.arc
FreeArc 0.67 (March 15 2014) testing archive: p:\ded.arc
Testing 1 file, 31,837,733,330 bytes. Processed 36.9%
ERROR: write error (disk full?) in compression algorithm dedup

#

[External compressor:dedup]
header = 0
packcmd = dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = dedup.exe d - - <stdin> <stdout>


Win10 Enterprise 21H2
Ryzen 7 9700x
32 GB RAM

ATM no more free time to testing.
Thanks for the program!
(Maybe later...)

Interesting (another dataset):
Microsoft Windows [Version 10.0.19044.5965]
(c) Microsoft Corporation. All rights reserved.

arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 8.3%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.8%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup

Hello, i am trying to understand what you are doing ...it's working fine here