Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials
Register FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old Today, 09:30
sajmon83's Avatar
sajmon83 sajmon83 is offline
Registered User
 
Join Date: Jun 2017
Location: Poland
Posts: 44
Thanks: 252
Thanked 57 Times in 35 Posts
sajmon83 is on a distinguished road
ROFZ Compressor

ROFZ 0.17.0 – LZ compressor with texture / audio / raw-data models

ROFZ is a lossless compressor for repacking games: optimal-parsing LZ77 + range coder with a per-block literal mixer, plus its own models for the data LZ codes badly. It finds that data anywhere in the stream, by header or by content:
  • BC1–BC7 / DXT textures – DDS, IWI, VTF headers, and BC1/3/4/5 without any header. Textures are chained, and a texture can be coded against an earlier one that shares its blocks.
  • PCM and ADPCM audio (WAV / AIFF, 8/16/24-bit, float, IMA / MS ADPCM) – cascade of sign-LMS predictors.
  • Raw multi-channel data – uncompressed images, vertex / animation arrays (adaptive linear predictor + context mixing).
  • x86 filter, multithreaded compression and decompression, stdin/stdout for FreeArc.
Std-only, Windows x64, needs an AVX2 CPU. Streams decode identically with any thread count.

Neural-network models (CPU only, integer arithmetic; trained offline on samples of 75 games than Memoriapolis. In order not to falsify the results)
  • Online mixing networks – the literal mixer, the BC pixel-index mixer and the raw-data model combine their context predictions with a single-layer logistic network that learns while coding (identically in encoder and decoder).
  • Pre-trained start states – the literal mixer and the raw-data model (weights, context cells, bias, APM) start from states learned on game data instead of even odds (~580 KB of tables in the exe), so small blocks and block starts code smaller.
  • Literal parameter network – a one-layer softmax network over 22 cheap statistics of a block (order-0/1 and position-context entropies, share of zeros, record-stride hints) proposes the two most promising literal-context settings for the encoder's test-coding (trained on 600 blocks from 75 game samples).
  • The gains are small (0.02–0.1%) but free for the decoder.

Levels
  • -l1 … -l4: fast-decoding models, kept at about 200 MB/s decompression on 16 threads (-ds N changes it). -l1 compresses fastest, -l4 smallest.
  • -l5 … -l8: full models, smallest output. -l5 compresses fastest, -l8 smallest.
  • -m = memory budget (% of RAM), -t = threads, -ds = decompression speed to keep (MB/s, any level).

Benchmark
Ryzen 7 5800X3D (8C/16T), 64 GB RAM, NVMe SSD, Windows 11. Command line, one run after another on an idle PC, all 16 threads; every output was decompressed and compared byte for byte with the input. MB/s refer to the compressor's input.

Settings
  • LOLZ ULTRA (DiskSpan's LOLZ_ULTRA alias): lolz -dtb1 -d263 -mtt1 -mt16 -mc1023, decoded with CLS-LOLZ_D_x64
  • TARZ 1.0.3 at its best (LOLZ-ULTRA-like settings): tarz c -t16 -d256 -mtt1 -tt16 -mc1023 -ow4096 -fb256, tarz d -t16
  • ROFZ 0.17.0: rofz c -lN -m85%, rofz d (defaults otherwise)

Inputs and precompression
  • Per-type sets: 35 files of up to 16 MB cut from game data samples (files of one kind put together): PCM audio, BC1–BC7 / DXT textures (DDS, VTF, IWI, XBM and header-less), raw images (BMP / TGA / RAW / TIF), text, Unity / CR2W / Source structures, executables, meshes, animations, BSP maps, PAK / VPK archives. No precompression – the same file for every compressor.
  • Memoriapolis (Unity 2022.3, 7 832 files, 7 267 959 185 B): files in Arc order (by extension, then path); the LZ4 / LZ4HC blocks of the 4 713 UnityFS archives restored to raw data, only blocks that recompress bit for bit) → 9 119 658 087 B; then srep 4.90g-wip -hash- -m3f -a0/0 → 5 355 251 639 B, the input of every run.

Code:
audio (PCM WAV / AIFF): 67 108 864 B input
compressor   output (B)   ratio  vs LOLZ  compression   MB/s  decompression  MB/s
LOLZ ULTRA   39 127 417  58.30%        -       50.4 s   1.33         1.62 s    42
TARZ (best)  40 183 818  59.88%   +2.70%      191.2 s   0.35         2.70 s    25
rofz -l1     30 841 176  45.96%  -21.18%        4.2 s  15.81         0.85 s    79
rofz -l2     30 841 153  45.96%  -21.18%        4.5 s  15.01         0.81 s    82
rofz -l3     30 841 117  45.96%  -21.18%        4.7 s  14.34         0.80 s    84
rofz -l4     30 841 062  45.96%  -21.18%        4.6 s  14.57         0.83 s    81
rofz -l5     30 841 056  45.96%  -21.18%        4.6 s  14.54         0.81 s    83
rofz -l6     30 839 955  45.96%  -21.18%        6.8 s   9.91         0.83 s    81
rofz -l7     30 840 239  45.96%  -21.18%        6.8 s   9.84         0.81 s    83
rofz -l8     30 840 254  45.96%  -21.18%        6.9 s   9.76         0.83 s    81

textures (DDS / BC1-BC7 / VTF / IWI / XBM): 150 994 944 B input
compressor   output (B)   ratio  vs LOLZ  compression  MB/s  decompression  MB/s
LOLZ ULTRA   65 330 035  43.27%        -      156.3 s  0.97         3.64 s    41
TARZ (best)  65 011 249  43.06%   -0.49%      285.7 s  0.53         6.87 s    22
rofz -l1     69 948 641  46.33%   +7.07%       51.9 s  2.91         1.72 s    88
rofz -l2     69 036 769  45.72%   +5.67%       61.5 s  2.46         1.74 s    87
rofz -l3     68 975 602  45.68%   +5.58%       75.9 s  1.99         1.74 s    87
rofz -l4     68 830 512  45.58%   +5.36%       88.5 s  1.71         1.73 s    87
rofz -l5     61 524 633  40.75%   -5.82%       69.2 s  2.18         8.54 s    18
rofz -l6     61 091 679  40.46%   -6.49%       95.4 s  1.58         8.93 s    17
rofz -l7     61 030 275  40.42%   -6.58%      110.9 s  1.36         9.27 s    16
rofz -l8     61 004 049  40.40%   -6.62%      135.2 s  1.12         9.09 s    17

raw images (BMP / TGA / RAW / TIF): 83 886 080 B input
compressor   output (B)   ratio  vs LOLZ  compression  MB/s  decompression  MB/s
LOLZ ULTRA   21 323 178  25.42%        -       78.9 s  1.06         1.60 s    52
TARZ (best)  20 755 681  24.74%   -2.66%      437.9 s  0.19         2.56 s    33
rofz -l1     24 610 417  29.34%  +15.42%       21.0 s  3.99         0.70 s   120
rofz -l2     24 464 511  29.16%  +14.73%       26.6 s  3.15         0.70 s   119
rofz -l3     24 362 914  29.04%  +14.26%       37.0 s  2.27         0.72 s   117
rofz -l4     24 280 411  28.94%  +13.87%       50.0 s  1.68         0.71 s   118
rofz -l5     20 374 255  24.29%   -4.45%       35.5 s  2.36         3.01 s    28
rofz -l6     20 180 470  24.06%   -5.36%       48.3 s  1.74         3.24 s    26
rofz -l7     20 150 628  24.02%   -5.50%       62.2 s  1.35         3.13 s    27
rofz -l8     20 118 927  23.98%   -5.65%       90.3 s  0.93         3.24 s    26

text, structures, executables, meshes, maps: 266 556 789 B input
compressor    output (B)   ratio  vs LOLZ  compression  MB/s  decompression  MB/s
LOLZ ULTRA    95 836 113  35.95%        -      267.1 s  1.00         3.27 s    82
TARZ (best)   95 749 915  35.92%   -0.09%     1301.8 s  0.20         4.68 s    57
rofz -l1     103 232 176  38.73%   +7.72%       65.8 s  4.05         2.39 s   111
rofz -l2     102 161 297  38.33%   +6.60%       92.7 s  2.87         2.53 s   105
rofz -l3     101 522 432  38.09%   +5.93%      133.0 s  2.00         2.56 s   104
rofz -l4     100 950 059  37.87%   +5.34%      164.5 s  1.62         2.66 s   100
rofz -l5      99 296 586  37.25%   +3.61%      147.1 s  1.81         5.42 s    49
rofz -l6      94 025 908  35.27%   -1.89%      283.4 s  0.94         9.01 s    30
rofz -l7      93 433 928  35.05%   -2.51%      417.0 s  0.64         9.24 s    29
rofz -l8      93 169 190  34.95%   -2.78%      673.5 s  0.40         9.19 s    29

Memoriapolis, whole game: 5 355 251 639 B input
compressor      output (B)   ratio  vs LOLZ  compression  MB/s  decompression  MB/s  RAM to decompress
LOLZ ULTRA   1 758 818 797  32.84%        -     1615.1 s  3.32        12.00 s   446           5 698 MB
TARZ (best)  1 772 775 068  33.10%   +0.79%     5779.4 s  0.93        20.30 s   264           2 784 MB
rofz -l1     1 858 615 328  34.71%   +5.67%      564.5 s  9.49        24.68 s   217           4 103 MB
rofz -l4     1 801 250 333  33.64%   +2.41%     1302.6 s  4.11        27.01 s   198           4 081 MB
rofz -l5     1 763 133 782  32.92%   +0.25%     1442.6 s  3.71        54.31 s    99           6 377 MB
rofz -l8     1 739 267 961  32.48%   -1.11%     3545.8 s  1.51        82.51 s    65           8 847 MB
Per set (output bytes)]
Code:
set                 LOLZ ULTRA  TARZ (best)     rofz -l1     rofz -l2     rofz -l3     rofz -l4     rofz -l5     rofz -l6     rofz -l7     rofz -l8
lit/text             1 939 745    1 920 604    2 262 808    2 227 316    2 218 327    2 215 268    2 200 847    1 862 894    1 862 845    1 862 066
lit/text_utf16         782 807      768 166      888 579      850 714      818 020      800 900      800 900      732 423      731 687      731 892
lit/struct_unity     4 622 416    4 613 608    4 849 278    4 751 920    4 734 144    4 691 595    4 620 495    4 559 034    4 543 832    4 522 125
lit/struct_cr2w     12 435 801   12 466 536   12 639 172   12 587 096   12 569 746   12 555 434   12 482 358   12 407 004   12 395 590   12 388 942
lit/struct_misc      1 646 195    1 642 642    2 542 813    2 416 790    2 414 836    2 386 836    2 385 877    1 775 473    1 767 250    1 750 461
lit/struct_source   12 877 500   12 860 749   12 995 671   12 980 473   12 903 204   12 852 635   12 642 857   12 426 081   12 393 627   12 373 962
lit/exe64            4 402 081    4 000 834    4 405 168    4 347 824    4 301 554    4 267 053    4 132 456    3 996 796    3 981 898    3 968 525
lit/exe32            9 230 031    9 100 271   11 300 627   11 260 205   11 248 602   11 237 582   11 187 425    9 041 833    9 036 342    9 035 011
lit/exe_net          3 619 668    3 589 282    3 814 151    3 774 163    3 726 540    3 703 505    3 646 090    3 567 644    3 557 735    3 549 889
lit/mesh             7 019 892    7 061 608    8 106 947    8 067 894    8 024 947    8 005 355    7 929 739    7 017 507    6 983 564    6 976 781
lit/anim             5 093 373    5 523 928    5 654 300    5 594 216    5 457 922    5 407 425    5 162 291    4 924 904    4 831 283    4 832 529
lit/map_bsp          2 044 624    2 073 579    2 928 230    2 813 633    2 737 254    2 656 450    2 457 677    2 411 648    2 134 791    2 011 830
lit/terrain_nav      5 181 137    5 001 339    5 356 031    5 261 473    5 188 097    5 139 100    5 010 374    4 977 348    4 948 536    4 926 081
lit/font             3 279 030    3 265 130    3 537 714    3 426 914    3 419 834    3 308 573    3 148 620    3 008 360    2 960 486    2 944 509
lit/shader             294 593      285 262      312 772      298 792      292 592      288 823      288 823      287 478      287 627      284 382
lit/pack_pak         9 394 270    9 534 756    9 561 099    9 531 863    9 516 861    9 508 237    9 450 969    9 316 653    9 312 523    9 310 209
lit/pack_vpk        11 972 950   12 041 621   12 076 816   11 970 011   11 949 952   11 925 288   11 748 788   11 712 828   11 704 312   11 699 996
audio/audio_pcm16   12 621 005   12 980 497    9 844 657    9 844 649    9 844 628    9 844 577    9 844 575    9 843 976    9 844 260    9 844 276
audio/WAV16          9 020 554    9 567 334    7 417 279    7 417 272    7 417 259    7 417 258    7 417 257    7 417 068    7 417 068    7 417 068
audio/WAV24          8 424 908    8 093 706    6 161 722    6 161 720    6 161 714    6 161 711    6 161 710    6 161 510    6 161 510    6 161 510
audio/AIFF           9 060 950    9 542 281    7 417 518    7 417 512    7 417 516    7 417 516    7 417 514    7 417 401    7 417 401    7 417 400
raw/img_raw          4 244 729    4 129 180    4 327 272    4 299 776    4 257 292    4 219 785    4 094 807    3 946 790    3 925 761    3 905 797
raw/BMP              4 286 406    4 176 200    5 026 948    5 005 881    4 994 612    4 979 539    4 096 476    4 076 462    4 072 811    4 067 539
raw/TGA              4 283 867    4 175 178    5 027 335    5 005 923    4 994 678    4 979 656    4 096 535    4 078 837    4 072 918    4 067 784
raw/RAW              4 254 850    4 126 583    5 319 411    5 244 519    5 226 319    5 220 814    4 243 943    4 236 600    4 237 186    4 235 932
raw/TIF              4 253 326    4 148 540    4 909 451    4 908 412    4 890 013    4 880 617    3 842 494    3 841 781    3 841 952    3 841 875
tex/tex_bc1          8 534 089    8 451 644    9 049 159    8 999 048    9 013 355    9 003 513    8 231 756    8 226 511    8 224 326    8 223 206
tex/tex_bc3          4 775 742    4 778 892    5 030 075    5 029 991    5 029 618    5 029 376    4 553 433    4 552 990    4 552 975    4 552 907
tex/tex_bc45         4 987 145    4 979 232    5 524 199    5 376 167    5 346 919    5 327 043    5 185 965    5 075 383    5 042 839    5 031 738
tex/tex_bc7         11 390 057   11 140 226   11 874 864   11 622 766   11 631 007   11 603 221   10 979 436   10 869 072   10 865 053   10 862 826
tex/tex_bc6         12 295 850   12 353 859   13 307 248   13 040 088   13 059 948   13 028 860    9 160 955    9 160 864    9 160 864    9 160 862
tex/dds              1 265 829    1 200 701    1 250 563    1 217 712    1 194 798    1 182 005    1 133 061    1 133 020    1 133 020    1 133 020
tex/tex_vtf          4 831 058    4 865 011    5 163 232    5 110 175    5 100 603    5 090 718    4 861 782    4 855 095    4 854 611    4 853 849
tex/tex_xbm          9 254 180    9 334 486   10 049 317    9 940 778    9 906 648    9 877 155    9 421 032    9 230 276    9 222 629    9 219 057
tex/tex_iwi          7 996 085    7 907 198    8 699 984    8 700 044    8 692 706    8 688 621    7 997 213    7 988 468    7 973 958    7 966 584
FreeArc / DiskSpan (arc.ini):
Code:
[External compressor:rofz]
header    = 0
packcmd   = "rofz.exe" c {options} - - <stdin> <stdout>
unpackcmd = "rofz.exe" d - - <stdin> <stdout>
Use e.g. -mrofz:l4:m65% after your precompressors. Presets: rofz:l2:m45% (fast), rofz:l4:m65% (normal), rofz:l7:m85% (high), rofz:l8:m85% (ultra). In an installer only the unpackcmd is needed (d -t100p - - <stdin> <stdout>).

## Changes
* **0.17.0** First release

Download: rofz-0.17.0-win64.7z
SHA-256: aef0048f694009f7406ba2911c58607f9b13278c59ec5fe592 d4e14b2b679351 (770 269 B)

Feedback and results on other games are welcome, especially where it loses. And 200MB of samples
Attached Files
File Type: 7z rofz-0.17.0-win64.7z (752.2 KB, 6 views)

Last edited by sajmon83; Today at 09:49.
Reply With Quote
The Following User Says Thank You to sajmon83 For This Useful Post:
DomoVoi_96 (Today)
Sponsored Links
  #2  
Old Today, 10:17
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 4,821
Thanks: 1,120
Thanked 7,483 Times in 2,909 Posts
KaktoR is on a distinguished road
adpcm sample
all settings left on default

Code:
rofz
Compressed 1 file, 208,780,556 => 156,157,676 bytes. Ratio 74.80%
Compression time: cpu 0.13 sec/real 37.72 sec = 0%. Speed 5.54 mB/s
All OK
Code:
tarz
Compressed 1 file, 208,780,556 => 149,665,201 bytes. Ratio 71.69%
Compression time: cpu 0.47 sec/real 374.98 sec = 0%. Speed 0.56 mB/s
All OK
__________________
Haters gonna hate
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New Fox Kompressor 1.01c Ultimate felice2011 Conversion Tutorials 31 07-06-2023 15:58
Ultimate Conversion Compressor (UCC) vollachr Conversion Tutorials 55 26-04-2021 10:27
Kitty File Compressor (STDIO Patch) 78372 Conversion Tutorials 1 31-10-2019 08:46
RAYcompressor - the ultimate AIO compressor rayated Conversion Tutorials 38 11-04-2018 01:10
Tiny Compressor 78372 Conversion Tutorials 1 24-11-2017 14:27



All times are GMT -7. The time now is 11:45.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com