Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials
Register FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 04-08-2019, 15:05
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,749
Thanks: 2,170
Thanked 11,206 Times in 2,307 Posts
Razor12911 is on a distinguished road
Update available

1908_R1
- Fixed MT hanging issues
- Fixed reflate issues
- Fixed memory issues
- Improved zlib codec
- Added auto reflate level detector
- Fixed aborting issues when used via stdio

Notes
If there is a file where this xtool fails to precompress better or the same as ztool, pzlib, reflate or precomp, please upload it.

Last edited by Razor12911; 04-08-2019 at 15:08.
Reply With Quote
The Following 13 Users Say Thank You to Razor12911 For This Useful Post:
Andu21 (05-08-2019), Cesar82 (04-08-2019), dixen (04-08-2019), ffmla (05-08-2019), Harsh ojha (04-08-2019), IgorKolesnik (04-08-2019), kenzo34 (05-08-2019), mausschieber (04-08-2019), sajmon83 (04-08-2019), shazzla (04-08-2019), ShivShubh (24-11-2019), Simorq (05-08-2019), ZAZA4EVER (04-08-2019)
Sponsored Links
  #2  
Old 05-08-2019, 11:29
Simorq's Avatar
Simorq Simorq is offline
Registered User
 
Join Date: Mar 2014
Location: Iran
Posts: 642
Thanks: 3,602
Thanked 1,308 Times in 464 Posts
Simorq is on a distinguished road
Test xtool_1908_R1

Streams: ZLib
Middle-Earth - Shadow of Mordor ALL OK
Fallout 4 ALL OK

Wargame Red Dragon
Streams: Reflate
Code:
pZLib,ZTool,XTool CRC Error
Code:
xtool_1908_R1 Good Work
Log:
Code:
Reflate
Creating archive: RG-reflate.Bin using reflate
Compressed 2,586 files, 30,541,141,079 => 44,791,613,736 bytes. Ratio 146.66%
Compression time: cpu 22.98 sec/real 1280.30 sec = 2%. Speed 23.85 mB/s
All OK

Testing archive: RG-reflate.Bin
Tested 2,586 files, 44,791,613,736 => 30,541,141,079 bytes. Ratio 146.66%  
Directory 3,955 => 88,873 bytes. Ratio 4.45%
Testing time: cpu 16.83 sec/real 924.56 sec = 2%. Speed 33.03 mB/s
All OK
Code:
rZLib
Creating archive: RG-rzlib.Bin using rzlib
Compressed 2,586 files, 30,541,141,079 => 44,791,601,829 bytes. Ratio 146.66%
Compression time: cpu 68.80 sec/real 2496.90 sec = 3%. Speed 12.23 mB/s
All OK

Testing archive: RG-rzlib.Bin
Tested 2,586 files, 44,791,601,829 => 30,541,141,079 bytes. Ratio 146.66%  
Directory 3,953 => 88,871 bytes. Ratio 4.45%
Testing time: cpu 17.86 sec/real 910.84 sec = 2%. Speed 33.53 mB/s
All OK
Reply With Quote
The Following 6 Users Say Thank You to Simorq For This Useful Post:
dixen (06-08-2019), ffmla (06-08-2019), Harsh ojha (05-08-2019), oltjon (05-08-2019), Razor12911 (05-08-2019), ZAZA4EVER (05-08-2019)
  #3  
Old 05-08-2019, 15:29
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,749
Thanks: 2,170
Thanked 11,206 Times in 2,307 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Simorq View Post
Log:
Code:
Reflate
Creating archive: RG-reflate.Bin using reflate
Compressed 2,586 files, 30,541,141,079 => 44,791,613,736 bytes. Ratio 146.66%
Compression time: cpu 22.98 sec/real 1280.30 sec = 2%. Speed 23.85 mB/s
All OK

Testing archive: RG-reflate.Bin
Tested 2,586 files, 44,791,613,736 => 30,541,141,079 bytes. Ratio 146.66%  
Directory 3,955 => 88,873 bytes. Ratio 4.45%
Testing time: cpu 16.83 sec/real 924.56 sec = 2%. Speed 33.03 mB/s
All OK
Code:
rZLib
Creating archive: RG-rzlib.Bin using rzlib
Compressed 2,586 files, 30,541,141,079 => 44,791,601,829 bytes. Ratio 146.66%
Compression time: cpu 68.80 sec/real 2496.90 sec = 3%. Speed 12.23 mB/s
All OK

Testing archive: RG-rzlib.Bin
Tested 2,586 files, 44,791,601,829 => 30,541,141,079 bytes. Ratio 146.66%  
Directory 3,953 => 88,871 bytes. Ratio 4.45%
Testing time: cpu 17.86 sec/real 910.84 sec = 2%. Speed 33.53 mB/s
All OK
Can you test these exes on the same input, thanks
Attached Files
File Type: 7z _binaries_fpc.7z (222.4 KB, 53 views)
Reply With Quote
The Following 4 Users Say Thank You to Razor12911 For This Useful Post:
dixen (06-08-2019), ffmla (06-08-2019), ShivShubh (24-11-2019), Simorq (06-08-2019)
  #4  
Old 06-08-2019, 00:13
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 650
Thanks: 772
Thanked 953 Times in 309 Posts
78372 is on a distinguished road
Code:
FreeArc 0.67 (March 15 2014) creating archive: datarzlib.arc
Compressed 1 file, 127,027,167 => 497,974,169 bytes. Ratio 392.02%
Compression time: cpu 0.41 sec/real 76.37 sec = 1%. Speed 1.66 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datarzlib.arc
Tested 1 file, 497,974,169 => 127,027,167 bytes. Ratio 392.02%
Testing time: cpu 0.23 sec/real 27.32 sec = 1%. Speed 4.65 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datazlib.arc
Compressed 1 file, 127,027,167 => 436,735,325 bytes. Ratio 343.81%
Compression time: cpu 0.42 sec/real 47.82 sec = 1%. Speed 2.66 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datazlib.arc
Tested 1 file, 436,735,325 => 127,027,167 bytes. Ratio 343.81%
Testing time: cpu 0.14 sec/real 16.94 sec = 1%. Speed 7.50 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datareflate.arc
Compressed 1 file, 127,027,167 => 498,080,370 bytes. Ratio 392.11%
Compression time: cpu 0.53 sec/real 161.65 sec = 0%. Speed 0.79 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datareflate.arc
Tested 1 file, 498,080,370 => 127,027,167 bytes. Ratio 392.11%
Testing time: cpu 0.27 sec/real 60.78 sec = 0%. Speed 2.09 mB/s
All OK
Code:
FreeArc 0.67 (March 15 2014) creating archive: datarzlib.arc
Compressed 1 file, 111,299,563 => 482,246,557 bytes. Ratio 433.29%
Compression time: cpu 0.45 sec/real 78.94 sec = 1%. Speed 1.41 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datarzlib.arc
Tested 1 file, 482,246,557 => 111,299,563 bytes. Ratio 433.29%
Testing time: cpu 0.23 sec/real 28.13 sec = 1%. Speed 3.96 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datazlib.arc
Compressed 1 file, 111,299,563 => 421,007,713 bytes. Ratio 378.27%
Compression time: cpu 0.39 sec/real 48.68 sec = 1%. Speed 2.29 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datazlib.arc
Tested 1 file, 421,007,713 => 111,299,563 bytes. Ratio 378.27%
Testing time: cpu 0.16 sec/real 16.98 sec = 1%. Speed 6.55 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datareflate.arc
Compressed 1 file, 111,299,563 => 482,352,758 bytes. Ratio 433.38%
Compression time: cpu 0.33 sec/real 162.03 sec = 0%. Speed 0.69 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datareflate.arc
Tested 1 file, 482,352,758 => 111,299,563 bytes. Ratio 433.38%
Testing time: cpu 0.09 sec/real 61.75 sec = 0%. Speed 1.80 mB/s
All OK

Last edited by 78372; 06-08-2019 at 00:24.
Reply With Quote
The Following 3 Users Say Thank You to 78372 For This Useful Post:
ffmla (06-08-2019), Razor12911 (07-08-2019), Simorq (06-08-2019)
  #5  
Old 06-08-2019, 03:17
Simorq's Avatar
Simorq Simorq is offline
Registered User
 
Join Date: Mar 2014
Location: Iran
Posts: 642
Thanks: 3,602
Thanked 1,308 Times in 464 Posts
Simorq is on a distinguished road
_binaries_fpc.7z Test

@Razor12911

x64
Code:
---------------------------
XTool.exe - Application Error
---------------------------
The application was unable to start correctly (0xc000007b). Click OK to close the application. 
---------------------------
OK   
---------------------------
x86
Code:
t100p
ERROR: write error (disk full?) in compression algorithm reflate
ERROR: write error (disk full?) in compression algorithm rzlib
ERROR: write error (disk full?) in compression algorithm zlib
But it works with t1,t2,t3,t4 in pre-compression \ t100p Works on decompression..
AMD RYZEN 5 1600 6-Core

Last edited by Simorq; 06-08-2019 at 03:53.
Reply With Quote
The Following 2 Users Say Thank You to Simorq For This Useful Post:
ffmla (06-08-2019), Razor12911 (07-08-2019)
  #6  
Old 06-08-2019, 04:24
Simorq's Avatar
Simorq Simorq is offline
Registered User
 
Join Date: Mar 2014
Location: Iran
Posts: 642
Thanks: 3,602
Thanked 1,308 Times in 464 Posts
Simorq is on a distinguished road
1908_R1 x86 vs _binaries_fpc x86 - Test

Code:
udun_weather.arch05 2.70 GB
1908_R1 x86 t100p for pre-compression = ERROR: write error (disk full?) in compression algorithm

1908_R1 x86
Code:
===============================================================================
t4
Creating archive: Game_ZLib.Bin using zlib
Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51%     
Compression time: cpu 1.55 sec/real 80.55 sec = 2%. Speed 36.04 mB/s
All OK

t100p
Testing archive: Game_ZLib.Bin
Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51%         
Testing time: cpu 1.11 sec/real 32.64 sec = 3%. Speed 88.95 mB/s
All OK
===============================================================================

===============================================================================
t4
Creating archive: Game_reflate.Bin using reflate
Compressed 1 file, 2,902,848,160 => 4,224,827,147 bytes. Ratio 145.54%     
Compression time: cpu 1.88 sec/real 318.25 sec = 1%. Speed 9.12 mB/s
All OK

t100p
Testing archive: Game_reflate.Bin
Tested 1 file, 4,224,827,147 => 2,902,848,160 bytes. Ratio 145.54%         
Testing time: cpu 1.61 sec/real 55.21 sec = 3%. Speed 52.58 mB/s
All OK
===============================================================================

===============================================================================
t4
Creating archive: Game_rzlib.Bin using rzlib
Compressed 1 file, 2,902,848,160 => 4,223,911,477 bytes. Ratio 145.51%     
Compression time: cpu 1.80 sec/real 79.44 sec = 2%. Speed 36.54 mB/s
All OK

t100p
Testing archive: Game_rzlib.Bin
WARNING: CRC failed in "udun_weather.arch05". File is broken.
Tested 1 file, 4,223,911,477 => 2,902,848,160 bytes. Ratio 145.51%         
Testing time: cpu 1.59 sec/real 43.39 sec = 4%. Speed 66.90 mB/s
There were 1 warning(s)
===============================================================================
_binaries_fpc x86
Code:
===============================================================================
t4
Creating archive: Game_ZLib.Bin using zlib
Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51%     
Compression time: cpu 1.73 sec/real 76.23 sec = 2%. Speed 38.08 mB/s
All OK

t100p
Testing archive: Game_ZLib.Bin
Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51%         
Testing time: cpu 1.44 sec/real 47.60 sec = 3%. Speed 60.98 mB/s
All OK
===============================================================================

===============================================================================
t4
Creating archive: Game_reflate.Bin using reflate
Compressed 1 file, 2,902,848,160 => 4,224,827,147 bytes. Ratio 145.54%     
Compression time: cpu 1.56 sec/real 303.38 sec = 1%. Speed 9.57 mB/s
All OK

t100p
Testing archive: Game_reflate.Bin
Tested 1 file, 4,224,827,147 => 2,902,848,160 bytes. Ratio 145.54%         
Testing time: cpu 1.59 sec/real 51.68 sec = 3%. Speed 56.17 mB/s
All OK
===============================================================================

===============================================================================
t4
Creating archive: Game_rzlib.Bin using rzlib
Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51%     
Compression time: cpu 1.55 sec/real 77.61 sec = 2%. Speed 37.40 mB/s
All OK

t100p
Testing archive: Game_rzlib.Bin
Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51%         
Testing time: cpu 1.47 sec/real 42.34 sec = 3%. Speed 68.56 mB/s
All OK
===============================================================================
__________________________________________________ __
The 1908_R1 x64 version works very stable and well.
Reply With Quote
The Following 2 Users Say Thank You to Simorq For This Useful Post:
ffmla (06-08-2019), Razor12911 (07-08-2019)
  #7  
Old 07-08-2019, 15:20
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,749
Thanks: 2,170
Thanked 11,206 Times in 2,307 Posts
Razor12911 is on a distinguished road
Update available

1908_R2
- Updated history size feature
- Added save/load history file feature
- Improved processing speed

The benefits of the usage history database files

I'll use GTAV as an example, it's a huge game and has a lot of highly compressed streams.

So if you were compressing this game and you messed up the srep/lolz settings or your PC shutdown unexpectedly due to several reasons like powercuts or windows forcing updates and etc, if xtool already stored a database then it will load it up with all the information that was used in previous precompression session on the same input to speed up the process because it will know what needs to be done and what settings should be used.

Here's an example on update.rpf (GTAV)

on first run:
Code:
Compressed 1 file, 814,551,040 => 1,616,722,602 bytes. Ratio 198.48%
Compression time: cpu 1.78 sec/real 77.43 sec = 2%. Speed 10.52 mB/s
since I have processed this input before, xtool loaded up history file:
Code:
Compressed 1 file, 814,551,040 => 1,616,722,602 bytes. Ratio 198.48%
Compression time: cpu 2.50 sec/real 26.25 sec = 10%. Speed 31.03 mB/s
The reason I decided to add this feature was because of games compressed with ridiculously high compression settings resulting in precompression being very slow so then I wanted people to save time by allowing at least one of us to process the game then share their database file with everyone so that they can use it to save time on precompressing the same game.

This xtool is already faster than the old one and I just had to find more ways to give you guys more speed.

Note: It doesn't matter if the input is different from the one that was used to create a specific history file, an example to explain this, if one used GTAV with latest update to make a history file, a person who has the first release of GTAV without any updates can still use the same history file without problems or visa versa, if a game got updated and you wanted to compress it again with the new files, you can use the old history file.
Reply With Quote
The Following 15 Users Say Thank You to Razor12911 For This Useful Post:
78372 (07-08-2019), Behnam2018 (30-01-2020), COPyCAT (23-08-2020), DiCaPrIo (08-08-2019), ffmla (08-08-2019), Harsh ojha (07-08-2019), IgorKolesnik (08-08-2019), murphy78 (12-08-2025), oltjon (10-08-2019), rambohazard (20-08-2019), Sergey3695 (08-08-2019), shazzla (09-08-2019), ShivShubh (24-11-2019), Simorq (07-08-2019), ZAZA4EVER (08-08-2019)
  #8  
Old 08-08-2019, 00:02
Harsh ojha's Avatar
Harsh ojha Harsh ojha is offline
Registered User
 
Join Date: May 2019
Location: INDIA
Posts: 78
Thanks: 527
Thanked 77 Times in 36 Posts
Harsh ojha is on a distinguished road
Test 1908_R2

Game GTA V use x64w.rpf
Size : 893 MB


Creating archive

#1 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: rzlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.78 sec/real 91.65 sec = 2%. Speed 10.23 mB/s
All OK
-------------------------------------------------------------------------------------------------------

#2 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: zlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.67 sec/real 107.49 sec = 2%. Speed 8.72 mB/s
All OK
-----------------------------------------------------------------------------------------------------------

#3reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: reflate_GTAV.arc using reflate
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.80 sec/real 61.87 sec = 3%. Speed 15.15 mB/s
All OK

---------------------------------------------------------------------------------------------------------------------------------

Extracting archive
#1 reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: reflate_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.22 sec/real 27.71 sec = 12%. Speed 33.83 mB/s
All OK

-----------------------------------------------------------------------------------------------------------------------------------

#2 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: rzlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.19 sec/real 59.93 sec = 5%. Speed 15.64 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------

#3 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: zlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.06 sec/real 38.95 sec = 8%. Speed 24.07 mB/s
All OK

end
__________________
Video Creator
Discord - Harsh_Ojha_748 #8782
Reply With Quote
  #9  
Old 08-08-2019, 15:37
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,749
Thanks: 2,170
Thanked 11,206 Times in 2,307 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Harsh ojha View Post
Game GTA V use x64w.rpf
Size : 893 MB


Creating archive

#1 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: rzlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.78 sec/real 91.65 sec = 2%. Speed 10.23 mB/s
All OK
-------------------------------------------------------------------------------------------------------

#2 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: zlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.67 sec/real 107.49 sec = 2%. Speed 8.72 mB/s
All OK
-----------------------------------------------------------------------------------------------------------

#3reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: reflate_GTAV.arc using reflate
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.80 sec/real 61.87 sec = 3%. Speed 15.15 mB/s
All OK

---------------------------------------------------------------------------------------------------------------------------------

Extracting archive
#1 reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: reflate_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.22 sec/real 27.71 sec = 12%. Speed 33.83 mB/s
All OK

-----------------------------------------------------------------------------------------------------------------------------------

#2 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: rzlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.19 sec/real 59.93 sec = 5%. Speed 15.64 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------

#3 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: zlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.06 sec/real 38.95 sec = 8%. Speed 24.07 mB/s
All OK

end
doesn't seem like libraries are loaded

Quote:
Originally Posted by Sergey3695 View Post
reflate didn't work with t100p when packing. add setting lvl. for reflate pls and add option for disable autodetect lvl. xD
Well up to you, if you know how to use reflate lvls
Reply With Quote
  #10  
Old 08-08-2019, 10:34
Sergey3695 Sergey3695 is offline
Registered User
 
Join Date: Mar 2013
Location: Russia
Posts: 86
Thanks: 43
Thanked 76 Times in 42 Posts
Sergey3695 is on a distinguished road
reflate didn't work with t100p when packing. add setting lvl. for reflate pls and add option for disable autodetect lvl. xD
Reply With Quote
  #11  
Old 08-08-2019, 11:28
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 265
Thanks: 190
Thanked 325 Times in 119 Posts
elit is on a distinguished road
There is much better way to detect cmp level than any current tools do(including precomp) and is simple to implement:

When you find first potential stream, first use all levels until match(as you probably do), but instead of fixing it for rest of data as granted, do at least say 9 more(up to 10). If 80%(or 8 of 10) of those streams show same specific level(say -9 or -6), then keep that for the rest of data.

If less than 80%, do all levels check for another 10 streams and average with previous 10. Now if average(of 20 streams) is 80% specific level then all is good for the rest of data, otherwise go full check again another 10 streams etc..

You can set minimum threshold to 70% if 80% threshold is too much but I think it should be fine. In other words, with this simple to implement algo you dont ever need option to manually set specific level, it will automatically determine whether streams are too mixed and it needs to test each on all levels(extremely unlikely), more likely it will eventually get right one that is in majority. This will also prevent false detection unless majority of first 10 streams are wrong levels, if thats common then you could do in steps of 100's instead of 10 and so on.
Reply With Quote
  #12  
Old 08-08-2019, 16:09
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,749
Thanks: 2,170
Thanked 11,206 Times in 2,307 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by elit View Post
There is much better way to detect cmp level than any current tools do(including precomp) and is simple to implement:

When you find first potential stream, first use all levels until match(as you probably do), but instead of fixing it for rest of data as granted, do at least say 9 more(up to 10). If 80%(or 8 of 10) of those streams show same specific level(say -9 or -6), then keep that for the rest of data.

If less than 80%, do all levels check for another 10 streams and average with previous 10. Now if average(of 20 streams) is 80% specific level then all is good for the rest of data, otherwise go full check again another 10 streams etc..

You can set minimum threshold to 70% if 80% threshold is too much but I think it should be fine. In other words, with this simple to implement algo you dont ever need option to manually set specific level, it will automatically determine whether streams are too mixed and it needs to test each on all levels(extremely unlikely), more likely it will eventually get right one that is in majority. This will also prevent false detection unless majority of first 10 streams are wrong levels, if thats common then you could do in steps of 100's instead of 10 and so on.
I'd like to see this method fly

especially with the case of xtool because of multi threading, precomp might find this useful because it is single threaded but xtool? no. It's not something I have not thought about before but look at it from this perspective. you're processing DiRT Rally, a game with large streams, so to detect level according to you is to process the first stream, check it level by level. Take note when it comes to deflate, there are not just 9 options (1-9), but there are 81 options, each level has memory settings (9 of them as well) with also affect the checksum of the stream if you are doing trial and error approach, so you're telling me it's best to try 81 levels on the first stream, so the next 9 streams can use the same method, during those 81 trial and errors, the other threads in xtool, what would the other threads be doing during that time? , a user selected to use 8 threads but it's still just 1 thread that needs to figure out what options to use and then after that is done, they can proceed. As those threads process the streams using the level they found from the first stream, if it's incorrect, they all now have to go trial and error for all those streams whichever way you look at it, this method has drawbacks in terms of time the other threads spend by not doing anything and just waiting, other drawback is, it's not really guaranteed that all those 81 options you'll be trying will give back perfect checksum so you can be making the other threads waiting for something that will never come.

xtool is fast not just because of multi threading, you can even compare it with precomp using 1 thread. I've thought about level detection and etc.

Here's how xtool works, the scanner itself can even give ideas of what level was used, zlib has a 2 byte header that is generated based on what level was used to compress a specific stream but sometimes this is not enough because some streams are headerless. This is where I decided to add a statistics system that stores information as all threads process the streams, this statistic system I introduced allows xtool to determine which cmp levels popped up the most and which ones were less, so if it were a case of trial and error, it wouldn't be trying from level 1-9, it would be the level that appeared the most using previous stream data(could even be in the order (6,5,9,8,1,2,3,4..). There are more strategies I've added for cmp level detection and they are all quick because they use stats
Reply With Quote
  #13  
Old 08-08-2019, 12:35
Sergey3695 Sergey3695 is offline
Registered User
 
Join Date: Mar 2013
Location: Russia
Posts: 86
Thanks: 43
Thanked 76 Times in 42 Posts
Sergey3695 is on a distinguished road
rly? lol. i'm look also on unpack time. example: better low lvl for save 1 min. my time or ~4 mb. better compression?

Last edited by Sergey3695; 08-08-2019 at 12:56.
Reply With Quote
  #14  
Old 08-08-2019, 15:41
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,749
Thanks: 2,170
Thanked 11,206 Times in 2,307 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Sergey3695 View Post
rly? lol. i'm look also on unpack time. example: better low lvl for save 1 min. my time or ~4 mb. better compression?
and you want me to remove auto detect.

This is why I add it in the first place, especially when it comes to reflate because most people don't know how it works. Low level does not mean you save time and at times, if you choose incorrect levels you end up with negative ratio. Example is GTAV, people used level 1 just because it was faster, resulted in horrible results then at some point people just decided to use level 9 on everything because "most" games are highly compressed, then game mad max which was level 1 compressed, they got shitty results again so yea, if you want level settings, I guess it's not a problem to add it but I added autodetect for a reason, because of people messing up
Reply With Quote
  #15  
Old 08-08-2019, 20:51
Sergey3695 Sergey3695 is offline
Registered User
 
Join Date: Mar 2013
Location: Russia
Posts: 86
Thanks: 43
Thanked 76 Times in 42 Posts
Sergey3695 is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
and you want me to remove auto detect.
add setting - turn off only/
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 15:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 18:16



All times are GMT -7. The time now is 15:35.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com