|
|
|
#1
|
||||
|
||||
|
Update available
1908_R1 - Fixed MT hanging issues - Fixed reflate issues - Fixed memory issues - Improved zlib codec - Added auto reflate level detector - Fixed aborting issues when used via stdio Notes If there is a file where this xtool fails to precompress better or the same as ztool, pzlib, reflate or precomp, please upload it. Last edited by Razor12911; 04-08-2019 at 15:08. |
| The Following 13 Users Say Thank You to Razor12911 For This Useful Post: | ||
Andu21 (05-08-2019), Cesar82 (04-08-2019), dixen (04-08-2019), ffmla (05-08-2019), Harsh ojha (04-08-2019), IgorKolesnik (04-08-2019), kenzo34 (05-08-2019), mausschieber (04-08-2019), sajmon83 (04-08-2019), shazzla (04-08-2019), ShivShubh (24-11-2019), Simorq (05-08-2019), ZAZA4EVER (04-08-2019) | ||
| Sponsored Links |
|
#2
|
||||
|
||||
|
Test xtool_1908_R1
Streams: ZLib
Middle-Earth - Shadow of Mordor ALL OK Fallout 4 ALL OK Wargame Red Dragon Streams: Reflate Code:
pZLib,ZTool,XTool CRC Error Code:
xtool_1908_R1 Good Work Code:
Reflate Creating archive: RG-reflate.Bin using reflate Compressed 2,586 files, 30,541,141,079 => 44,791,613,736 bytes. Ratio 146.66% Compression time: cpu 22.98 sec/real 1280.30 sec = 2%. Speed 23.85 mB/s All OK Testing archive: RG-reflate.Bin Tested 2,586 files, 44,791,613,736 => 30,541,141,079 bytes. Ratio 146.66% Directory 3,955 => 88,873 bytes. Ratio 4.45% Testing time: cpu 16.83 sec/real 924.56 sec = 2%. Speed 33.03 mB/s All OK Code:
rZLib Creating archive: RG-rzlib.Bin using rzlib Compressed 2,586 files, 30,541,141,079 => 44,791,601,829 bytes. Ratio 146.66% Compression time: cpu 68.80 sec/real 2496.90 sec = 3%. Speed 12.23 mB/s All OK Testing archive: RG-rzlib.Bin Tested 2,586 files, 44,791,601,829 => 30,541,141,079 bytes. Ratio 146.66% Directory 3,953 => 88,871 bytes. Ratio 4.45% Testing time: cpu 17.86 sec/real 910.84 sec = 2%. Speed 33.53 mB/s All OK |
| The Following 6 Users Say Thank You to Simorq For This Useful Post: | ||
dixen (06-08-2019), ffmla (06-08-2019), Harsh ojha (05-08-2019), oltjon (05-08-2019), Razor12911 (05-08-2019), ZAZA4EVER (05-08-2019) | ||
|
#3
|
||||
|
||||
|
Quote:
|
|
#4
|
|||
|
|||
|
Code:
FreeArc 0.67 (March 15 2014) creating archive: datarzlib.arc Compressed 1 file, 127,027,167 => 497,974,169 bytes. Ratio 392.02% Compression time: cpu 0.41 sec/real 76.37 sec = 1%. Speed 1.66 mB/s All OK FreeArc 0.67 (March 15 2014) testing archive: datarzlib.arc Tested 1 file, 497,974,169 => 127,027,167 bytes. Ratio 392.02% Testing time: cpu 0.23 sec/real 27.32 sec = 1%. Speed 4.65 mB/s All OK FreeArc 0.67 (March 15 2014) creating archive: datazlib.arc Compressed 1 file, 127,027,167 => 436,735,325 bytes. Ratio 343.81% Compression time: cpu 0.42 sec/real 47.82 sec = 1%. Speed 2.66 mB/s All OK FreeArc 0.67 (March 15 2014) testing archive: datazlib.arc Tested 1 file, 436,735,325 => 127,027,167 bytes. Ratio 343.81% Testing time: cpu 0.14 sec/real 16.94 sec = 1%. Speed 7.50 mB/s All OK FreeArc 0.67 (March 15 2014) creating archive: datareflate.arc Compressed 1 file, 127,027,167 => 498,080,370 bytes. Ratio 392.11% Compression time: cpu 0.53 sec/real 161.65 sec = 0%. Speed 0.79 mB/s All OK FreeArc 0.67 (March 15 2014) testing archive: datareflate.arc Tested 1 file, 498,080,370 => 127,027,167 bytes. Ratio 392.11% Testing time: cpu 0.27 sec/real 60.78 sec = 0%. Speed 2.09 mB/s All OK Code:
FreeArc 0.67 (March 15 2014) creating archive: datarzlib.arc Compressed 1 file, 111,299,563 => 482,246,557 bytes. Ratio 433.29% Compression time: cpu 0.45 sec/real 78.94 sec = 1%. Speed 1.41 mB/s All OK FreeArc 0.67 (March 15 2014) testing archive: datarzlib.arc Tested 1 file, 482,246,557 => 111,299,563 bytes. Ratio 433.29% Testing time: cpu 0.23 sec/real 28.13 sec = 1%. Speed 3.96 mB/s All OK FreeArc 0.67 (March 15 2014) creating archive: datazlib.arc Compressed 1 file, 111,299,563 => 421,007,713 bytes. Ratio 378.27% Compression time: cpu 0.39 sec/real 48.68 sec = 1%. Speed 2.29 mB/s All OK FreeArc 0.67 (March 15 2014) testing archive: datazlib.arc Tested 1 file, 421,007,713 => 111,299,563 bytes. Ratio 378.27% Testing time: cpu 0.16 sec/real 16.98 sec = 1%. Speed 6.55 mB/s All OK FreeArc 0.67 (March 15 2014) creating archive: datareflate.arc Compressed 1 file, 111,299,563 => 482,352,758 bytes. Ratio 433.38% Compression time: cpu 0.33 sec/real 162.03 sec = 0%. Speed 0.69 mB/s All OK FreeArc 0.67 (March 15 2014) testing archive: datareflate.arc Tested 1 file, 482,352,758 => 111,299,563 bytes. Ratio 433.38% Testing time: cpu 0.09 sec/real 61.75 sec = 0%. Speed 1.80 mB/s All OK Last edited by 78372; 06-08-2019 at 00:24. |
| The Following 3 Users Say Thank You to 78372 For This Useful Post: | ||
|
#5
|
||||
|
||||
|
_binaries_fpc.7z Test
@Razor12911
x64 Code:
--------------------------- XTool.exe - Application Error --------------------------- The application was unable to start correctly (0xc000007b). Click OK to close the application. --------------------------- OK --------------------------- Code:
t100p ERROR: write error (disk full?) in compression algorithm reflate ERROR: write error (disk full?) in compression algorithm rzlib ERROR: write error (disk full?) in compression algorithm zlib AMD RYZEN 5 1600 6-Core Last edited by Simorq; 06-08-2019 at 03:53. |
| The Following 2 Users Say Thank You to Simorq For This Useful Post: | ||
ffmla (06-08-2019), Razor12911 (07-08-2019) | ||
|
#6
|
||||
|
||||
|
1908_R1 x86 vs _binaries_fpc x86 - Test
Code:
udun_weather.arch05 2.70 GB 1908_R1 x86 Code:
=============================================================================== t4 Creating archive: Game_ZLib.Bin using zlib Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51% Compression time: cpu 1.55 sec/real 80.55 sec = 2%. Speed 36.04 mB/s All OK t100p Testing archive: Game_ZLib.Bin Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51% Testing time: cpu 1.11 sec/real 32.64 sec = 3%. Speed 88.95 mB/s All OK =============================================================================== =============================================================================== t4 Creating archive: Game_reflate.Bin using reflate Compressed 1 file, 2,902,848,160 => 4,224,827,147 bytes. Ratio 145.54% Compression time: cpu 1.88 sec/real 318.25 sec = 1%. Speed 9.12 mB/s All OK t100p Testing archive: Game_reflate.Bin Tested 1 file, 4,224,827,147 => 2,902,848,160 bytes. Ratio 145.54% Testing time: cpu 1.61 sec/real 55.21 sec = 3%. Speed 52.58 mB/s All OK =============================================================================== =============================================================================== t4 Creating archive: Game_rzlib.Bin using rzlib Compressed 1 file, 2,902,848,160 => 4,223,911,477 bytes. Ratio 145.51% Compression time: cpu 1.80 sec/real 79.44 sec = 2%. Speed 36.54 mB/s All OK t100p Testing archive: Game_rzlib.Bin WARNING: CRC failed in "udun_weather.arch05". File is broken. Tested 1 file, 4,223,911,477 => 2,902,848,160 bytes. Ratio 145.51% Testing time: cpu 1.59 sec/real 43.39 sec = 4%. Speed 66.90 mB/s There were 1 warning(s) =============================================================================== Code:
=============================================================================== t4 Creating archive: Game_ZLib.Bin using zlib Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51% Compression time: cpu 1.73 sec/real 76.23 sec = 2%. Speed 38.08 mB/s All OK t100p Testing archive: Game_ZLib.Bin Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51% Testing time: cpu 1.44 sec/real 47.60 sec = 3%. Speed 60.98 mB/s All OK =============================================================================== =============================================================================== t4 Creating archive: Game_reflate.Bin using reflate Compressed 1 file, 2,902,848,160 => 4,224,827,147 bytes. Ratio 145.54% Compression time: cpu 1.56 sec/real 303.38 sec = 1%. Speed 9.57 mB/s All OK t100p Testing archive: Game_reflate.Bin Tested 1 file, 4,224,827,147 => 2,902,848,160 bytes. Ratio 145.54% Testing time: cpu 1.59 sec/real 51.68 sec = 3%. Speed 56.17 mB/s All OK =============================================================================== =============================================================================== t4 Creating archive: Game_rzlib.Bin using rzlib Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51% Compression time: cpu 1.55 sec/real 77.61 sec = 2%. Speed 37.40 mB/s All OK t100p Testing archive: Game_rzlib.Bin Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51% Testing time: cpu 1.47 sec/real 42.34 sec = 3%. Speed 68.56 mB/s All OK =============================================================================== The 1908_R1 x64 version works very stable and well.
|
| The Following 2 Users Say Thank You to Simorq For This Useful Post: | ||
ffmla (06-08-2019), Razor12911 (07-08-2019) | ||
|
#7
|
||||
|
||||
|
Update available
1908_R2 - Updated history size feature - Added save/load history file feature - Improved processing speed The benefits of the usage history database files I'll use GTAV as an example, it's a huge game and has a lot of highly compressed streams. So if you were compressing this game and you messed up the srep/lolz settings or your PC shutdown unexpectedly due to several reasons like powercuts or windows forcing updates and etc, if xtool already stored a database then it will load it up with all the information that was used in previous precompression session on the same input to speed up the process because it will know what needs to be done and what settings should be used. Here's an example on update.rpf (GTAV) on first run: Code:
Compressed 1 file, 814,551,040 => 1,616,722,602 bytes. Ratio 198.48% Compression time: cpu 1.78 sec/real 77.43 sec = 2%. Speed 10.52 mB/s Code:
Compressed 1 file, 814,551,040 => 1,616,722,602 bytes. Ratio 198.48% Compression time: cpu 2.50 sec/real 26.25 sec = 10%. Speed 31.03 mB/s This xtool is already faster than the old one and I just had to find more ways to give you guys more speed. ![]() Note: It doesn't matter if the input is different from the one that was used to create a specific history file, an example to explain this, if one used GTAV with latest update to make a history file, a person who has the first release of GTAV without any updates can still use the same history file without problems or visa versa, if a game got updated and you wanted to compress it again with the new files, you can use the old history file. |
| The Following 15 Users Say Thank You to Razor12911 For This Useful Post: | ||
78372 (07-08-2019), Behnam2018 (30-01-2020), COPyCAT (23-08-2020), DiCaPrIo (08-08-2019), ffmla (08-08-2019), Harsh ojha (07-08-2019), IgorKolesnik (08-08-2019), murphy78 (12-08-2025), oltjon (10-08-2019), rambohazard (20-08-2019), Sergey3695 (08-08-2019), shazzla (09-08-2019), ShivShubh (24-11-2019), Simorq (07-08-2019), ZAZA4EVER (08-08-2019) | ||
|
#8
|
||||
|
||||
|
Test 1908_R2
Game GTA V use x64w.rpf
Size : 893 MB Creating archive #1 rzlib FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log Creating archive: rzlib_GTAV.arc using rzlib Memory for compression 0b, decompression 0b, cache 16mb Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00% Compression time: cpu 1.78 sec/real 91.65 sec = 2%. Speed 10.23 mB/s All OK ------------------------------------------------------------------------------------------------------- #2 zlib FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log Creating archive: zlib_GTAV.arc using rzlib Memory for compression 0b, decompression 0b, cache 16mb Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00% Compression time: cpu 1.67 sec/real 107.49 sec = 2%. Speed 8.72 mB/s All OK ----------------------------------------------------------------------------------------------------------- #3reflate FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log Creating archive: reflate_GTAV.arc using reflate Memory for compression 0b, decompression 0b, cache 16mb Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00% Compression time: cpu 1.80 sec/real 61.87 sec = 3%. Speed 15.15 mB/s All OK --------------------------------------------------------------------------------------------------------------------------------- Extracting archive #1 reflate FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log Extracting archive: reflate_GTAV.arc Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00% Extraction time: cpu 3.22 sec/real 27.71 sec = 12%. Speed 33.83 mB/s All OK ----------------------------------------------------------------------------------------------------------------------------------- #2 rzlib FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log Extracting archive: rzlib_GTAV.arc Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00% Extraction time: cpu 3.19 sec/real 59.93 sec = 5%. Speed 15.64 mB/s All OK ------------------------------------------------------------------------------------------------------------------------------------- #3 zlib FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log Extracting archive: zlib_GTAV.arc Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00% Extraction time: cpu 3.06 sec/real 38.95 sec = 8%. Speed 24.07 mB/s All OK end
__________________
Video Creator Discord - Harsh_Ojha_748 #8782 |
|
#9
|
||||
|
||||
|
Quote:
![]() Well up to you, if you know how to use reflate lvls |
|
#10
|
|||
|
|||
|
reflate didn't work with t100p when packing. add setting lvl. for reflate pls and add option for disable autodetect lvl. xD
|
|
#11
|
|||
|
|||
|
There is much better way to detect cmp level than any current tools do(including precomp) and is simple to implement:
When you find first potential stream, first use all levels until match(as you probably do), but instead of fixing it for rest of data as granted, do at least say 9 more(up to 10). If 80%(or 8 of 10) of those streams show same specific level(say -9 or -6), then keep that for the rest of data. If less than 80%, do all levels check for another 10 streams and average with previous 10. Now if average(of 20 streams) is 80% specific level then all is good for the rest of data, otherwise go full check again another 10 streams etc.. You can set minimum threshold to 70% if 80% threshold is too much but I think it should be fine. In other words, with this simple to implement algo you dont ever need option to manually set specific level, it will automatically determine whether streams are too mixed and it needs to test each on all levels(extremely unlikely), more likely it will eventually get right one that is in majority. This will also prevent false detection unless majority of first 10 streams are wrong levels, if thats common then you could do in steps of 100's instead of 10 and so on. |
|
#12
|
||||
|
||||
|
Quote:
![]() especially with the case of xtool because of multi threading, precomp might find this useful because it is single threaded but xtool? no. It's not something I have not thought about before but look at it from this perspective. you're processing DiRT Rally, a game with large streams, so to detect level according to you is to process the first stream, check it level by level. Take note when it comes to deflate, there are not just 9 options (1-9), but there are 81 options, each level has memory settings (9 of them as well) with also affect the checksum of the stream if you are doing trial and error approach, so you're telling me it's best to try 81 levels on the first stream, so the next 9 streams can use the same method, during those 81 trial and errors, the other threads in xtool, what would the other threads be doing during that time? , a user selected to use 8 threads but it's still just 1 thread that needs to figure out what options to use and then after that is done, they can proceed. As those threads process the streams using the level they found from the first stream, if it's incorrect, they all now have to go trial and error for all those streams whichever way you look at it, this method has drawbacks in terms of time the other threads spend by not doing anything and just waiting, other drawback is, it's not really guaranteed that all those 81 options you'll be trying will give back perfect checksum so you can be making the other threads waiting for something that will never come.xtool is fast not just because of multi threading, you can even compare it with precomp using 1 thread. I've thought about level detection and etc. Here's how xtool works, the scanner itself can even give ideas of what level was used, zlib has a 2 byte header that is generated based on what level was used to compress a specific stream but sometimes this is not enough because some streams are headerless. This is where I decided to add a statistics system that stores information as all threads process the streams, this statistic system I introduced allows xtool to determine which cmp levels popped up the most and which ones were less, so if it were a case of trial and error, it wouldn't be trying from level 1-9, it would be the level that appeared the most using previous stream data(could even be in the order (6,5,9,8,1,2,3,4..). There are more strategies I've added for cmp level detection and they are all quick because they use stats |
|
#13
|
|||
|
|||
|
rly? lol. i'm look also on unpack time. example: better low lvl for save 1 min. my time or ~4 mb. better compression?
Last edited by Sergey3695; 08-08-2019 at 12:56. |
|
#14
|
||||
|
||||
|
Quote:
![]() ![]() and you want me to remove auto detect.This is why I add it in the first place, especially when it comes to reflate because most people don't know how it works. Low level does not mean you save time and at times, if you choose incorrect levels you end up with negative ratio. Example is GTAV, people used level 1 just because it was faster, resulted in horrible results then at some point people just decided to use level 9 on everything because "most" games are highly compressed, then game mad max which was level 1 compressed, they got shitty results again so yea, if you want level settings, I guess it's not a problem to add it but I added autodetect for a reason, because of people messing up |
|
#15
|
|||
|
|||
|
add setting - turn off only/
|
![]() |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| [Dev]XTool | Razor12911 | Conversion Tutorials | 180 | 23-10-2020 06:26 |
| Project Cars Digital Edition (3xDVD5) (srep+lzma) | GTX590 | PC Games - CD/DVD Conversions | 10 | 28-08-2017 08:34 |
| Project IGI Anthology 1xCD700 CIUV2 2039 | mausschieber | PC Games - CD/DVD Conversions | 0 | 24-07-2017 15:12 |
| Space Channel 5 Part 2 Translation Project | Christuserloeser | DC Games | 0 | 21-06-2004 18:16 |