Best Compression For Archiving

[brispuss]

Not sure if this is the right part of this forum to ask this, but?


Trying to save space on HDD by LOSSLESSLY compressing some games for archiving purposes. And later decompressing these games LOSSLESSLY to match the orginal files exactly (bit perfect).


At this stage, concentrating mostly on compressing .dds, .jpg, .ogg, and .png files from games.

So far, compressing *.dds files seem to be quite good using lolz directly with -d128 option and with all other lolz options at default settings.

Here is a sample dds file in case someone wishes to improve broch_02.dds compression?

Now, having problems compressing *.jpg, *.png, and *.ogg files (LOSSLESSLY). Since these files are already compressed (to a degree), it is difficult compressing these files much smaller.

Here are some sample files -

he-crown-solomon117.jpg

common_game_04.png

amb_in_boat_cabin.ogg


Can anyone suggest suitable tools and their settings to losslessly compress these (sample) files, please?

Thank you!

[78372]

packjpg for jpeg images, png contains zlib streams so xtool or reflate should work, oggrec can compress ogg files losslessly.

[brispuss]

Thanks for the post!

I'm experimenting with several compressors now.

But trying to compress the png file common_game_04.png using xtool produces the same sized file output (113 kB)! No compression of this file at all!?

Tried using the command

xtool e:precomp:c32mb,t4:zlib

on this test png file. Why is the png file not being compressed?

[78372]

Try reflate. XTool doesn't work for all png streams.

[brispuss]

Thanks again.

But where do I get the "reflate", and what (command line) settings should be used to run it?

The *.ogg file recompressor oggre doesn't seem to compress the amb_in_boat_cabin.ogg file much unfortunately. Original ogg file size ~ 1004 kB, and after recompressing, the file size becomes only ~ 971 kB. This is only about 3% reduction in file size.

Is there a better program or group of programs that would LOSSLESSLY compress *.ogg files better? Ideally something around 50% or more reduction in file size would be excellent!

[78372]

oggrec is probably the best(or only) lossless compressor for ogg files. Ogg files aren't much recompressible.

[78372]

Here's reflate CLS-Reflate
As it's a cls release, you don't need to edit arc.ini, just paste the files beside arc.exe

[brispuss]

Thanks again for the comments and link.

EDIT: Not sure how to run reflate now?

1) Which version of reflate should be used "k" or "l"? And why?

2) Pasted files from "k" version into Freearc (v0.67) bin sub-directory where the command line executable version of Freearc (Arc.exe) is.

3) Tried running reflate executable under elevated command prompt and nothing seemed to happen.

4) Ran Arc executable under elevated commend prompt, but I don't see an option for "reflate"; unless I missed it?

5) So how do you run reflate?

6) And how do you run reflate ONLY without any precompression and without any compression applied?

[78372]

reflate is a precompressor. Use l version. Program works like -mreflate+srep+lolz

[brispuss]

Thanks, but which command line executable should be run? Arc.exe? Or?

I tried the command "arc -mreflate" only but only get arc showing commands help!? Also tried "arc -mreflate+srep+lolz" but still get arc showing commands help only!?

[Simorq]

@brispuss

FreeArc 0.67 (March 15 2014)  http://freearc.org  March 15 2014
High-performance archiver. Free as well for commercial as for non-commercial use
Usage: Arc command [options...] archive [files... @listfiles...]
Commands:
  a        add files to archive
  c        add comment to archive
  ch       modify archive (recompress, encrypt and so on)
  create   create new archive
  cw       write archive comment to file
  d        delete files from archive
  e        extract files from archive ignoring pathnames
  f        freshen archive
  j        join archives
  k        lock archive
  l        list files in archive
  lb       bare list of files in archive
  lt       technical archive listing
  m        move files and dirs to archive
  mf       move files to archive
  modify   modify archive using +/-/* actions
  r        recover archive using recovery record
  rr       add recovery record to archive
  s        convert archive to SFX
  t        test archive integrity
  u        update files in archive
  v        verbosely list files in archive
  x        extract files from archive
Options:
  --                                        stop processing options
  -ac           --ClearArchiveBit           clear Archive bit on files succesfully (de)archived
  -ad           --adddir                    add arcname to extraction path
  -aeALGORITHM  --encryption=ALGORITHM      encryption ALGORITHM (aes, blowfish, serpent, twofish)
  -agFMT        --autogenerate=FMT          autogenerate archive name with FMT
  -ao           --SelectArchiveBit          select only files with Archive bit set
  -apDIR        --arcpath=DIR               base DIR in archive
  -baMODE       --BrokenArchive=MODE        deal with badly broken archive using MODE
  -cfgFILES     --config=FILES              use configuration FILES (default: arc*.ini)
  -d            --delete                    delete files & dirs after successful archiving
  -df           --delfiles                  delete only files after successful archiving
  -diAMOUNT     --display=AMOUNT            control AMOUNT of information displayed: [hoacmnwrfdtske]*
  -dmMETHOD     --dirmethod=METHOD          compression METHOD for archive directory
  -dpDIR        --diskpath=DIR              base DIR on disk
  -dsORDER      --sort=ORDER                sort files in ORDER
  -ed           --nodirs                    don't add empty dirs to archive
  -envVAR                                   read default options from environment VAR (default: FREEARC)
  -epMODE       --ExcludePath=MODE          Exclude/expand path MODE
  -f            --freshen                   freshen files
  -fn           --fullnames                 match with full names
  -hpPASSWORD   --HeadersPassword=PASSWORD  encrypt/decrypt archive headers and data using PASSWORD
  -iTYPE        --indicator=TYPE            select progress indicator TYPE (0/1/2)
  -ioff         --shutdown                  shutdown computer when operation completed
  -k            --lock                      lock archive
  -kb           --keepbroken                keep broken extracted files
  -kfKEYFILE    --keyfile=KEYFILE           encrypt/decrypt using KEYFILE
  -lcN          --LimitCompMem=N            limit memory usage for compression to N mbytes
  -ldN          --LimitDecompMem=N          limit memory usage for decompression to N mbytes
  -mMETHOD      --method=METHOD             compression METHOD (-m0..-m9, -m1x..-m9x)
  -maLEVEL                                  set filetype detection LEVEL (+/-/1..9)
  -max                                      maximum compression using external precomp, ecm, ppmonstr
  -mc                                       disable compression algorithms (-mcd-, -mc-rep...)
  -mdN          --dictionary=N              set compression dictionary to N mbytes
  -mmMODE       --multimedia=MODE           set multimedia compression to MODE
  -ms           --StoreCompressed           store already compressed files
  -mtTHREADS    --MultiThreaded=THREADS     number of compression THREADS
  -mx                                       maximum internal compression mode
  -nFILESPECS   --include=FILESPECS         include only files matching FILESPECS
  -oMODE        --overwrite=MODE            existing files overwrite MODE (+/-/p)
  -okfKEYFILE   --OldKeyfile=KEYFILE        old KEYFILE used only for decryption
  -opPASSWORD   --OldPassword=PASSWORD      old PASSWORD used only for decryption
  -pPASSWORD    --password=PASSWORD         encrypt/decrypt compressed data using PASSWORD
  -r            --recursive                 recursively collect files
  -rrSIZE       --recovery=SIZE             add recovery information of specified SIZE to archive
  -sGROUPING    --solid=GROUPING            GROUPING for solid compression
  -scCHARSETS   --charset=CHARSETS          CHARSETS used for listfiles and comment files
  -sfxMODULE                                add sfx MODULE ("freearc.sfx" by default)
  -slSIZE       --SizeLess=SIZE             select files smaller than SIZE
  -smSIZE       --SizeMore=SIZE             select files larger than SIZE
  -t            --test                      test archive after operation
  -tTYPE        --type=TYPE                 archive TYPE (arc/zip/rar/...)
  -taTIME       --TimeAfter=TIME            select files modified after specified TIME
  -tbTIME       --TimeBefore=TIME           select files modified before specified TIME
  -tk           --keeptime                  keep original archive time
  -tl           --timetolast                set archive time to latest file
  -tnPERIOD     --TimeNewer=PERIOD          select files newer than specified time PERIOD
  -toPERIOD     --TimeOlder=PERIOD          select files older than specified time PERIOD
  -tpMODE       --pretest=MODE              test archive before operation using MODE
  -u            --update                    update files
  -vSIZE        --volume=SIZE               split archive to volumes each of SIZE bytes
  -wDIRECTORY   --workdir=DIRECTORY         DIRECTORY for temporary files
  -xFILESPECS   --exclude=FILESPECS         exclude FILESPECS from operation
  -y            --yes                       answer Yes to all queries
  -zFILE        --arccmt=FILE               read archive comment from FILE or stdin
                --append                    add new files to the end of archive
                --archive-comment=COMMENT   input archive COMMENT in cmdline
                --bypass=URL                setups proxy bypass list for URL access
                --cache=N                   use N mbytes for read-ahead cache
                --crconly                   save/check CRC, but don't store data
                --create-in-workdir         create archive in workdir and then move to final location
                --dirs                      add empty dirs to archive
                --groups=FILE               name of groups FILE
                --language=FILE             load localisation from FILE
                --logfile=FILE              duplicate all information displayed to this FILE
                --noarcext                  don't add default extension to archive name
                --nodata                    don't store data in archive
                --nodates                   don't store filetimes in archive
                --nodir                     don't write archive directories
                --original=URL              redownload broken parts of archive from the URL
                --pause-before-exit=PAUSE   make a PAUSE just before closing program window
                --print-config              display built-in definitions of compression methods
                --proxy=URL                 setups proxy(s) for URL access
                --queue                     queue operations across multiple FreeArc copies
                --recompress                recompress archive contents
                --save-bad-ranges=FILE      save list of broken archive parts to the FILE
                --sync                      synchronize archive and disk contents
All OK

Batch Script Example

:-ioff         --shutdown
SET exclude=-x@Resources\_Exclude.lst
SET DIR1=M:\Journey
SET MC=SR+lolz:dtb1:d32:mtt1:mt6:mc1023

Resources\Arc.exe create -cfgFA.ini -i2 -ds=pns -ep1 -r -ed -w.\Temp -s; -dp"%DIR1%" -m%MC% %exclude% "Journey.Bin"
RD /S /Q TEMP
pause

[brispuss]

Thanks for the detailed reply!

I've basically got cls-reflate working now by using the command-

arc create -mreflate archive inputfile

Similarly, I've also downloaded and installed cls-srep for Freearc as well. So the command for srep is -

arc create -msrep archive inputfile

As for the sample batch script, that is a little over my head at the moment!

As of now, by experimenting with different (pre)compressors, the best compression for *.png files seems to be by using -

Precomp V0.4.7 (-cn -brute options), and then using lolz V22c4b (-d128 -mc1023 options).

The test file common_game_04.png at 113 kB is compressed to 90 kB. Not too bad, but I would like better compression if possible.

As for the *.ogg files, they are proving to be extremely difficult to compress much further!


EDIT: I've got the common_game_04.png file compressed down to 77 kB by using Precomp and then using fp8 V6 at maximum compression (-8 option).

And by using paq8px V183fix1 (-9beta options) instead of fp8 V6, common_game_04.png is now compressed to 72 kB, but this compression process is VERY SLOW!

[amcic1994]

Originally Posted by brispuss

Thanks, but which command line executable should be run? Arc.exe? Or?

I tried the command "arc -mreflate" only but only get arc showing commands help!? Also pornjk porn800 redtube tried "arc -mreflate+srep+lolz" but still get arc showing commands help only!?

Trying to save space on HDD by LOSSLESSLY compressing some games for archiving purposes. And later decompressing these games LOSSLESSLY to match the orginal files exactly (bit perfect).

[RebeccaFoley]

Thanks alot!