FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   XTool 2020 (Main Project) (https://fileforums.com/showthread.php?t=102832)

Razor12911 28-07-2019 17:14

Can you run xtool outside freearc, like via console.

Code:

@echo off
xtool.exe precomp:rzlib:c16mb,t100p %1 %1.out
pause

I'm interested in the exception/error message

Simorq 28-07-2019 17:21

@Razor12911
good
http://s8.picofile.com/file/8368124442/rrrr.jpg

-------------------------
I think the main problem is - -.
- - <stdin> <stdout>:confused:

Razor12911 28-07-2019 17:31

try this then

Code:

@echo off
xtool.exe precomp:rzlib:c32mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res
pause


Simorq 28-07-2019 17:38

@Razor12911
no work
stuck cpu
http://s9.picofile.com/file/8368124692/rrr.jpg

Razor12911 28-07-2019 17:42

Nice ghost bugs, I'll check what I missed in the source and thanks for the tests.

Sergey3695 29-07-2019 00:19

I have a problem using t100p.

Sergey3695 30-07-2019 05:37

add pls lvl setting for reflate 1-9. thx.

Razor12911 31-07-2019 03:00

Quote:

Originally Posted by Sergey3695 (Post 481930)
add pls lvl setting for reflate 1-9. thx.

Have patience

IgorKolesnik 31-07-2019 23:56

using these parameters, the expansion reaches 2.4% and freezes.

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp:rzlib,zlib,reflate:c16mb,t2 - - <stdin> <stdout>


I am using the x64 version.

FitGirl 01-08-2019 07:41

Quote:

Originally Posted by IgorKolesnik (Post 481942)
using these parameters, the expansion reaches 2.4% and freezes.

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp:rzlib,zlib,reflate:c16mb,t2 - - <stdin> <stdout>


I am using the x64 version.

And why do you use all three methods at the same time?

IgorKolesnik 01-08-2019 07:44

Quote:

Originally Posted by FitGirl (Post 481943)
And why do you use all three methods at the same time?

если я оставлю только один метод, ошибка та же.

----

if I leave only one method, the error is the same.

IgorKolesnik 01-08-2019 07:46

But what is the difference between rzlib and zlib?

FitGirl 01-08-2019 08:19

Quote:

Originally Posted by IgorKolesnik (Post 481945)
But what is the difference between rzlib and zlib?

Don't be that lazy.
https://fileforums.com/showpost.php?...4&postcount=35

As for decompression error - it's been already reported and Razor answered.

And don't use Russian here, it's English-only forum.

Razor12911 01-08-2019 15:17

There are internal errors I've come across, so for now until next update, don't use xtool. the expansion stops because of memory management issues, xtool keeps increasing memory usage over and over again until you run out of memory, I noticed this when I was running some tests.

As of right now, I have fixed this issue but am running more tests to make sure all is well.

FitGirl mentioned you must pick one codec in the zlib lineup, rzlib is what is best and it's the same one used in the old xtool, I just added the other ones (zlib and reflate) for special cases where a specific codec is required.

Razor12911 04-08-2019 15:05

Update available

1908_R1
- Fixed MT hanging issues
- Fixed reflate issues
- Fixed memory issues
- Improved zlib codec
- Added auto reflate level detector
- Fixed aborting issues when used via stdio

Notes
If there is a file where this xtool fails to precompress better or the same as ztool, pzlib, reflate or precomp, please upload it.

Simorq 05-08-2019 11:29

Test xtool_1908_R1
 
Streams: ZLib
Middle-Earth - Shadow of Mordor ALL OK
Fallout 4 ALL OK

Wargame Red Dragon
Streams: Reflate
Code:

pZLib,ZTool,XTool CRC Error
Code:

xtool_1908_R1 Good Work
Log:
Code:

Reflate
Creating archive: RG-reflate.Bin using reflate
Compressed 2,586 files, 30,541,141,079 => 44,791,613,736 bytes. Ratio 146.66%
Compression time: cpu 22.98 sec/real 1280.30 sec = 2%. Speed 23.85 mB/s
All OK

Testing archive: RG-reflate.Bin
Tested 2,586 files, 44,791,613,736 => 30,541,141,079 bytes. Ratio 146.66% 
Directory 3,955 => 88,873 bytes. Ratio 4.45%
Testing time: cpu 16.83 sec/real 924.56 sec = 2%. Speed 33.03 mB/s
All OK

Code:

rZLib
Creating archive: RG-rzlib.Bin using rzlib
Compressed 2,586 files, 30,541,141,079 => 44,791,601,829 bytes. Ratio 146.66%
Compression time: cpu 68.80 sec/real 2496.90 sec = 3%. Speed 12.23 mB/s
All OK

Testing archive: RG-rzlib.Bin
Tested 2,586 files, 44,791,601,829 => 30,541,141,079 bytes. Ratio 146.66% 
Directory 3,953 => 88,871 bytes. Ratio 4.45%
Testing time: cpu 17.86 sec/real 910.84 sec = 2%. Speed 33.53 mB/s
All OK


Razor12911 05-08-2019 15:29

1 Attachment(s)
Quote:

Originally Posted by Simorq (Post 481980)
Log:
Code:

Reflate
Creating archive: RG-reflate.Bin using reflate
Compressed 2,586 files, 30,541,141,079 => 44,791,613,736 bytes. Ratio 146.66%
Compression time: cpu 22.98 sec/real 1280.30 sec = 2%. Speed 23.85 mB/s
All OK

Testing archive: RG-reflate.Bin
Tested 2,586 files, 44,791,613,736 => 30,541,141,079 bytes. Ratio 146.66% 
Directory 3,955 => 88,873 bytes. Ratio 4.45%
Testing time: cpu 16.83 sec/real 924.56 sec = 2%. Speed 33.03 mB/s
All OK

Code:

rZLib
Creating archive: RG-rzlib.Bin using rzlib
Compressed 2,586 files, 30,541,141,079 => 44,791,601,829 bytes. Ratio 146.66%
Compression time: cpu 68.80 sec/real 2496.90 sec = 3%. Speed 12.23 mB/s
All OK

Testing archive: RG-rzlib.Bin
Tested 2,586 files, 44,791,601,829 => 30,541,141,079 bytes. Ratio 146.66% 
Directory 3,953 => 88,871 bytes. Ratio 4.45%
Testing time: cpu 17.86 sec/real 910.84 sec = 2%. Speed 33.53 mB/s
All OK


Can you test these exes on the same input, thanks

78372 06-08-2019 00:13

Code:

FreeArc 0.67 (March 15 2014) creating archive: datarzlib.arc
Compressed 1 file, 127,027,167 => 497,974,169 bytes. Ratio 392.02%
Compression time: cpu 0.41 sec/real 76.37 sec = 1%. Speed 1.66 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datarzlib.arc
Tested 1 file, 497,974,169 => 127,027,167 bytes. Ratio 392.02%
Testing time: cpu 0.23 sec/real 27.32 sec = 1%. Speed 4.65 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datazlib.arc
Compressed 1 file, 127,027,167 => 436,735,325 bytes. Ratio 343.81%
Compression time: cpu 0.42 sec/real 47.82 sec = 1%. Speed 2.66 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datazlib.arc
Tested 1 file, 436,735,325 => 127,027,167 bytes. Ratio 343.81%
Testing time: cpu 0.14 sec/real 16.94 sec = 1%. Speed 7.50 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datareflate.arc
Compressed 1 file, 127,027,167 => 498,080,370 bytes. Ratio 392.11%
Compression time: cpu 0.53 sec/real 161.65 sec = 0%. Speed 0.79 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datareflate.arc
Tested 1 file, 498,080,370 => 127,027,167 bytes. Ratio 392.11%
Testing time: cpu 0.27 sec/real 60.78 sec = 0%. Speed 2.09 mB/s
All OK

Code:

FreeArc 0.67 (March 15 2014) creating archive: datarzlib.arc
Compressed 1 file, 111,299,563 => 482,246,557 bytes. Ratio 433.29%
Compression time: cpu 0.45 sec/real 78.94 sec = 1%. Speed 1.41 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datarzlib.arc
Tested 1 file, 482,246,557 => 111,299,563 bytes. Ratio 433.29%
Testing time: cpu 0.23 sec/real 28.13 sec = 1%. Speed 3.96 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datazlib.arc
Compressed 1 file, 111,299,563 => 421,007,713 bytes. Ratio 378.27%
Compression time: cpu 0.39 sec/real 48.68 sec = 1%. Speed 2.29 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datazlib.arc
Tested 1 file, 421,007,713 => 111,299,563 bytes. Ratio 378.27%
Testing time: cpu 0.16 sec/real 16.98 sec = 1%. Speed 6.55 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datareflate.arc
Compressed 1 file, 111,299,563 => 482,352,758 bytes. Ratio 433.38%
Compression time: cpu 0.33 sec/real 162.03 sec = 0%. Speed 0.69 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datareflate.arc
Tested 1 file, 482,352,758 => 111,299,563 bytes. Ratio 433.38%
Testing time: cpu 0.09 sec/real 61.75 sec = 0%. Speed 1.80 mB/s
All OK


Simorq 06-08-2019 03:17

_binaries_fpc.7z Test
 
@Razor12911

x64
Code:

---------------------------
XTool.exe - Application Error
---------------------------
The application was unable to start correctly (0xc000007b). Click OK to close the application.
---------------------------
OK 
---------------------------

x86
Code:

t100p
ERROR: write error (disk full?) in compression algorithm reflate
ERROR: write error (disk full?) in compression algorithm rzlib
ERROR: write error (disk full?) in compression algorithm zlib

But it works with t1,t2,t3,t4 in pre-compression \ t100p Works on decompression..
AMD RYZEN 5 1600 6-Core

Simorq 06-08-2019 04:24

1908_R1 x86 vs _binaries_fpc x86 - Test
 
Code:

udun_weather.arch05 2.70 GB
1908_R1 x86 t100p for pre-compression = ERROR: write error (disk full?) in compression algorithm

1908_R1 x86
Code:

===============================================================================
t4
Creating archive: Game_ZLib.Bin using zlib
Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51%   
Compression time: cpu 1.55 sec/real 80.55 sec = 2%. Speed 36.04 mB/s
All OK

t100p
Testing archive: Game_ZLib.Bin
Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51%       
Testing time: cpu 1.11 sec/real 32.64 sec = 3%. Speed 88.95 mB/s
All OK
===============================================================================

===============================================================================
t4
Creating archive: Game_reflate.Bin using reflate
Compressed 1 file, 2,902,848,160 => 4,224,827,147 bytes. Ratio 145.54%   
Compression time: cpu 1.88 sec/real 318.25 sec = 1%. Speed 9.12 mB/s
All OK

t100p
Testing archive: Game_reflate.Bin
Tested 1 file, 4,224,827,147 => 2,902,848,160 bytes. Ratio 145.54%       
Testing time: cpu 1.61 sec/real 55.21 sec = 3%. Speed 52.58 mB/s
All OK
===============================================================================

===============================================================================
t4
Creating archive: Game_rzlib.Bin using rzlib
Compressed 1 file, 2,902,848,160 => 4,223,911,477 bytes. Ratio 145.51%   
Compression time: cpu 1.80 sec/real 79.44 sec = 2%. Speed 36.54 mB/s
All OK

t100p
Testing archive: Game_rzlib.Bin
WARNING: CRC failed in "udun_weather.arch05". File is broken.
Tested 1 file, 4,223,911,477 => 2,902,848,160 bytes. Ratio 145.51%       
Testing time: cpu 1.59 sec/real 43.39 sec = 4%. Speed 66.90 mB/s
There were 1 warning(s)
===============================================================================

_binaries_fpc x86
Code:

===============================================================================
t4
Creating archive: Game_ZLib.Bin using zlib
Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51%   
Compression time: cpu 1.73 sec/real 76.23 sec = 2%. Speed 38.08 mB/s
All OK

t100p
Testing archive: Game_ZLib.Bin
Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51%       
Testing time: cpu 1.44 sec/real 47.60 sec = 3%. Speed 60.98 mB/s
All OK
===============================================================================

===============================================================================
t4
Creating archive: Game_reflate.Bin using reflate
Compressed 1 file, 2,902,848,160 => 4,224,827,147 bytes. Ratio 145.54%   
Compression time: cpu 1.56 sec/real 303.38 sec = 1%. Speed 9.57 mB/s
All OK

t100p
Testing archive: Game_reflate.Bin
Tested 1 file, 4,224,827,147 => 2,902,848,160 bytes. Ratio 145.54%       
Testing time: cpu 1.59 sec/real 51.68 sec = 3%. Speed 56.17 mB/s
All OK
===============================================================================

===============================================================================
t4
Creating archive: Game_rzlib.Bin using rzlib
Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51%   
Compression time: cpu 1.55 sec/real 77.61 sec = 2%. Speed 37.40 mB/s
All OK

t100p
Testing archive: Game_rzlib.Bin
Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51%       
Testing time: cpu 1.47 sec/real 42.34 sec = 3%. Speed 68.56 mB/s
All OK
===============================================================================

__________________________________________________ __
The 1908_R1 x64 version works very stable and well.:D

Razor12911 07-08-2019 15:20

Update available

1908_R2
- Updated history size feature
- Added save/load history file feature
- Improved processing speed

The benefits of the usage history database files

I'll use GTAV as an example, it's a huge game and has a lot of highly compressed streams.

So if you were compressing this game and you messed up the srep/lolz settings or your PC shutdown unexpectedly due to several reasons like powercuts or windows forcing updates and etc, if xtool already stored a database then it will load it up with all the information that was used in previous precompression session on the same input to speed up the process because it will know what needs to be done and what settings should be used.

Here's an example on update.rpf (GTAV)

on first run:
Code:

Compressed 1 file, 814,551,040 => 1,616,722,602 bytes. Ratio 198.48%
Compression time: cpu 1.78 sec/real 77.43 sec = 2%. Speed 10.52 mB/s

since I have processed this input before, xtool loaded up history file:
Code:

Compressed 1 file, 814,551,040 => 1,616,722,602 bytes. Ratio 198.48%
Compression time: cpu 2.50 sec/real 26.25 sec = 10%. Speed 31.03 mB/s

The reason I decided to add this feature was because of games compressed with ridiculously high compression settings resulting in precompression being very slow so then I wanted people to save time by allowing at least one of us to process the game then share their database file with everyone so that they can use it to save time on precompressing the same game.

This xtool is already faster than the old one and I just had to find more ways to give you guys more speed. :D

Note: It doesn't matter if the input is different from the one that was used to create a specific history file, an example to explain this, if one used GTAV with latest update to make a history file, a person who has the first release of GTAV without any updates can still use the same history file without problems or visa versa, if a game got updated and you wanted to compress it again with the new files, you can use the old history file.

Harsh ojha 08-08-2019 00:02

Test 1908_R2
 
Game GTA V use x64w.rpf
Size : 893 MB


Creating archive

#1 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: rzlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.78 sec/real 91.65 sec = 2%. Speed 10.23 mB/s
All OK
-------------------------------------------------------------------------------------------------------

#2 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: zlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.67 sec/real 107.49 sec = 2%. Speed 8.72 mB/s
All OK
-----------------------------------------------------------------------------------------------------------

#3reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: reflate_GTAV.arc using reflate
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.80 sec/real 61.87 sec = 3%. Speed 15.15 mB/s
All OK

---------------------------------------------------------------------------------------------------------------------------------

Extracting archive
#1 reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: reflate_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.22 sec/real 27.71 sec = 12%. Speed 33.83 mB/s
All OK

-----------------------------------------------------------------------------------------------------------------------------------

#2 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: rzlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.19 sec/real 59.93 sec = 5%. Speed 15.64 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------

#3 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: zlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.06 sec/real 38.95 sec = 8%. Speed 24.07 mB/s
All OK

end :)

Sergey3695 08-08-2019 10:34

reflate didn't work with t100p when packing. add setting lvl. for reflate pls and add option for disable autodetect lvl. xD

elit 08-08-2019 11:28

There is much better way to detect cmp level than any current tools do(including precomp) and is simple to implement:

When you find first potential stream, first use all levels until match(as you probably do), but instead of fixing it for rest of data as granted, do at least say 9 more(up to 10). If 80%(or 8 of 10) of those streams show same specific level(say -9 or -6), then keep that for the rest of data.

If less than 80%, do all levels check for another 10 streams and average with previous 10. Now if average(of 20 streams) is 80% specific level then all is good for the rest of data, otherwise go full check again another 10 streams etc..

You can set minimum threshold to 70% if 80% threshold is too much but I think it should be fine. In other words, with this simple to implement algo you dont ever need option to manually set specific level, it will automatically determine whether streams are too mixed and it needs to test each on all levels(extremely unlikely), more likely it will eventually get right one that is in majority. This will also prevent false detection unless majority of first 10 streams are wrong levels, if thats common then you could do in steps of 100's instead of 10 and so on.

Sergey3695 08-08-2019 12:35

rly? lol. i'm look also on unpack time. example: better low lvl for save 1 min. my time or ~4 mb. better compression?

Razor12911 08-08-2019 15:37

Quote:

Originally Posted by Harsh ojha (Post 482013)
Game GTA V use x64w.rpf
Size : 893 MB


Creating archive

#1 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: rzlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.78 sec/real 91.65 sec = 2%. Speed 10.23 mB/s
All OK
-------------------------------------------------------------------------------------------------------

#2 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: zlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.67 sec/real 107.49 sec = 2%. Speed 8.72 mB/s
All OK
-----------------------------------------------------------------------------------------------------------

#3reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: reflate_GTAV.arc using reflate
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.80 sec/real 61.87 sec = 3%. Speed 15.15 mB/s
All OK

---------------------------------------------------------------------------------------------------------------------------------

Extracting archive
#1 reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: reflate_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.22 sec/real 27.71 sec = 12%. Speed 33.83 mB/s
All OK

-----------------------------------------------------------------------------------------------------------------------------------

#2 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: rzlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.19 sec/real 59.93 sec = 5%. Speed 15.64 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------

#3 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: zlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.06 sec/real 38.95 sec = 8%. Speed 24.07 mB/s
All OK

end :)

doesn't seem like libraries are loaded :)

Quote:

Originally Posted by Sergey3695 (Post 482025)
reflate didn't work with t100p when packing. add setting lvl. for reflate pls and add option for disable autodetect lvl. xD

Well up to you, if you know how to use reflate lvls

Razor12911 08-08-2019 15:41

Quote:

Originally Posted by Sergey3695 (Post 482027)
rly? lol. i'm look also on unpack time. example: better low lvl for save 1 min. my time or ~4 mb. better compression?

:eek::eek::eek: and you want me to remove auto detect.

This is why I add it in the first place, especially when it comes to reflate because most people don't know how it works. Low level does not mean you save time and at times, if you choose incorrect levels you end up with negative ratio. Example is GTAV, people used level 1 just because it was faster, resulted in horrible results then at some point people just decided to use level 9 on everything because "most" games are highly compressed, then game mad max which was level 1 compressed, they got shitty results again so yea, if you want level settings, I guess it's not a problem to add it but I added autodetect for a reason, because of people messing up

Razor12911 08-08-2019 16:09

Quote:

Originally Posted by elit (Post 482026)
There is much better way to detect cmp level than any current tools do(including precomp) and is simple to implement:

When you find first potential stream, first use all levels until match(as you probably do), but instead of fixing it for rest of data as granted, do at least say 9 more(up to 10). If 80%(or 8 of 10) of those streams show same specific level(say -9 or -6), then keep that for the rest of data.

If less than 80%, do all levels check for another 10 streams and average with previous 10. Now if average(of 20 streams) is 80% specific level then all is good for the rest of data, otherwise go full check again another 10 streams etc..

You can set minimum threshold to 70% if 80% threshold is too much but I think it should be fine. In other words, with this simple to implement algo you dont ever need option to manually set specific level, it will automatically determine whether streams are too mixed and it needs to test each on all levels(extremely unlikely), more likely it will eventually get right one that is in majority. This will also prevent false detection unless majority of first 10 streams are wrong levels, if thats common then you could do in steps of 100's instead of 10 and so on.

I'd like to see this method fly :D

especially with the case of xtool because of multi threading, precomp might find this useful because it is single threaded but xtool? no. It's not something I have not thought about before but look at it from this perspective. you're processing DiRT Rally, a game with large streams, so to detect level according to you is to process the first stream, check it level by level. Take note when it comes to deflate, there are not just 9 options (1-9), but there are 81 options, each level has memory settings (9 of them as well) with also affect the checksum of the stream if you are doing trial and error approach, so you're telling me it's best to try 81 levels on the first stream, so the next 9 streams can use the same method, during those 81 trial and errors, the other threads in xtool, what would the other threads be doing during that time? :D, a user selected to use 8 threads but it's still just 1 thread that needs to figure out what options to use and then after that is done, they can proceed. As those threads process the streams using the level they found from the first stream, if it's incorrect, they all now have to go trial and error for all those streams whichever way you look at it, this method has drawbacks in terms of time the other threads spend by not doing anything and just waiting, other drawback is, it's not really guaranteed that all those 81 options you'll be trying will give back perfect checksum so you can be making the other threads waiting for something that will never come.

xtool is fast not just because of multi threading, you can even compare it with precomp using 1 thread. I've thought about level detection and etc.

Here's how xtool works, the scanner itself can even give ideas of what level was used, zlib has a 2 byte header that is generated based on what level was used to compress a specific stream but sometimes this is not enough because some streams are headerless. This is where I decided to add a statistics system that stores information as all threads process the streams, this statistic system I introduced allows xtool to determine which cmp levels popped up the most and which ones were less, so if it were a case of trial and error, it wouldn't be trying from level 1-9, it would be the level that appeared the most using previous stream data(could even be in the order (6,5,9,8,1,2,3,4..). There are more strategies I've added for cmp level detection and they are all quick because they use stats

Sergey3695 08-08-2019 20:51

Quote:

Originally Posted by Razor12911 (Post 482031)
:eek::eek::eek: and you want me to remove auto detect.

add setting - turn off only/

elit 09-08-2019 15:17

Razor,

you mentioned xtool does auto detection on the first stream. Does it iterate through all 81 levels until match is found, or only few known to be used most often?

As for my idea,
It don't need to go through all 81 every time, only until match is found and then next stream can be tested with that level found in previous stream first. This means if streams are not mixed there will be no slowdown because first try is a match already. Also your own test implementation on first stream seem fast, why not use same technique, whatever is it doing?

I have never noticed ztool/xtool being slow at beginning, it seems to start immediately. Would there really be such big difference if it had to try 10 streams instead of 1, especially with optimization I mentioned above(starting with previously found level)? Perhaps up to few seconds of delay should not be much problem in scope of thousand streams and GB's of data?

As for MT, that's not a big deal we don't want to test whole data only first couple of streams(again assuming it should not take long).

You mentioned using stats, that sounds actually very similar to what I am talking about. In fact, if I understand correctly you may have already implemented what I am talking about here :). But interestingly, you mentioned collecting stats from threads, that pretty much equals to cmp levels. So you in fact are collecting from multiple streams, not just first?

Razor12911 09-08-2019 16:22

"you mentioned xtool does auto detection on the first stream. Does it iterate through all 81 levels until match is found, or only few known to be used most often?"
- it does for all streams, all 81 options but very quickly, takes milliseconds per stream to go through all 81 options.

"It don't need to go through all 81 every time"
- it doesn't go through all 81, sometimes even the first option it tries is correct and it stops, if it isn't correct then the 2nd one is most likely correct because of stats. XTool requests mode (value that appears most often), if that option fails, it requests mode in 2nd order (value that appears the 2nd most often) and so forth... which is why trials on all streams is necessary because it keeps the stats function filled with updated information about the current input.

"next stream can be tested with that level found in previous stream first."
- this method is the same as stat but it stores information limited to one stream which happens to be the previous stream, the approach of xtool is it stores a number of previous stream information, default is 256 streams.
Let's say for example
if the first stream was gave back level 9, the next stream is determined to be level 9, roughly the same as what you suggested, but what if the third stream is level 6? you have go through all the trials, ok fine, now you think then next stream is level 6 because of previous stream but what if that level 6 stream was just png image within the game data but instead the entire game was level 9? you try level 6 on the next stream it fails, then again you have to go through all 81 options again. XTool apporach thanks to stats, is level 9, level 9 and level 6 is what was stored, so for the next stream, it doesn't matter if the last one was level 6, it will try level 9 because it was detected twice compared to level 6, if it wasn't level 9, it will try that level 6 before it tries all 81 options because it was at least detected once, if that fails. it then tries 81 - 2 (6-9 from stats already failed) it does this for the past 256 streams and tries options in mode orders.

"I have never noticed ztool/xtool being slow at beginning, it seems to start immediately"
- It's because of stats plus other methods I've developed that make this possible.

"Perhaps up to few seconds of delay should not be much problem in scope of thousand streams and GB's of data?"
You'll notice not just "few seconds of delay", yes few seconds for a set of streams but minutes of overall delay :D if you were to use this method for big streams, try this on dirt rally or doom 2016 and you'll see some incredibly low CPU utilization when using a lot of threads because other threads are obliged to wait for the first stream.

"As for MT, that's not a big deal we don't want to test whole data only first couple of streams(again assuming it should not take long)."

with large streams, it is guaranteed to take long as mentioned before, which is why xtool faces too many problems with multi threading, I've instructed the threads to try 81 levels while using stats from all the streams they can find, as a result they end up fighting and crash each other :D it's hilarious but you get the idea, it's just best to try all levels on all streams but let stats decide in what order they should be presented in to speed up the process.

"You mentioned using stats, that sounds actually very similar to what I am talking about."
- Yes with the exception that not the previous stream option is considered but the previous 256 stream options are considered. If from those 256 options, there are 128 level 6s, 10 level 3s , 1 level 5 and 20 level 9s, it will try them in the order: 6 >> 9 >> 3 >> 5, then since all that failed, it begins trials, 1 - 9, it will skip, 3,5,6 and 9 because those failed, after that the new discovered level is added to stats, if it was 2, it will try the new order as 6 >> 9 >> 3 >> 5 >> 2, the reason all streams have to try all options is to get rid of non appearing options, example is when the data out of nowhere started to become level 4 out of nowhere, as stats are added, the number of level 9s will decrease because of these new levels 4s until they reach a certain threshold, then the order will now be 4 >> 6 >> 9 >> 3 >> 5 >> 2, as you can see level 4 over took level 6 but level 6 is still considered because before all of the level 4s there were a lot of level 6s so yes, I really did think deep about how to speed up xtool :)

"I understand correctly you may have already implemented what I am talking about here"
- Yep

"So you in fact are collecting from multiple streams, not just first?"
- Yes, I had to carefully do this else the threads crash themselves while fighting for the same stream or the same information, causes headaches but I find it funny when this happens, this is what causes most problems in pzlib/ztool and old xtool but if you properly implement it, you get to enjoy all the speed that comes with it.

elit 09-08-2019 18:05

Finally, now(!) I am in picture. Thanks for taking your time, it does make a difference when I understand internals.

Believe or not until now I thought (z)xtool simply scan first stream and.. that's it - use same info for *all* rest of data lol :). Reason I thought so was that game packs are likely using single specific level and that could in theory be enough(to pack the game, for which zxtools were designed).

Since that is not the case and your approach is clearly even much better than mine we can rest this case. Thanks again.

Simorq 10-08-2019 15:25

1908_R2_x86 > ERROR: write error (disk full?) in compression algorithm
1908_R2_x64 > good work

decode has a bug.

Code:

decode:t1
Testing archive: data.arc
Tested 2 files, 4,023,182,817 => 2,147,912,601 bytes. Ratio 187.31%       
Testing time: cpu 1.09 sec/real 23.73 sec = 5%. Speed 90.52 mB/s
All OK

decode:t100p
Testing archive: data.arc
Tested 2 files, 4,023,182,817 => 2,147,912,601 bytes. Ratio 187.31%
Testing time: cpu 1.16 sec/real 22.47 sec = 5%. Speed 95.61 mB/s
All OK

Even with t1 it uses all cores!

doofoo24 11-08-2019 08:50

1 Attachment(s)
Quote:

Originally Posted by Simorq (Post 482047)
1908_R2_x86 > ERROR: write error (disk full?) in compression algorithm
1908_R2_x64 > good work

also with x64 you can get "ERROR: write error (disk full?) in compression algorithm"...
i tested on doom with reflate t100p with c384mb i get ( ERROR: write error (disk full?))...
i changed to t4 it work...

for the decode it work i tested with t1 and t100p, for t1 use 11% of the cpu and for t100p it use all core...
***it's so important to emphasis do not use t100p for decode if you don't have a good cpu cooler, better to use t50p...

Razor12911 11-08-2019 11:16

Guys, I’ll repeat. If you encounter an error, use xtool without fa so it produces an exception message so I know what the problem is

Simorq 11-08-2019 14:02

reflate (XTool 1908_R2_x86)
 
#############################################
# reflate (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

Code:

XTool x86 Command Line

xtool.exe precomp:reflate:c16mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

Test 1 = OK
Test 2 = OK
Test 3 = OK
Test 4 = OK

Code:

XTool x86 Command Line

xtool.exe precomp:reflate:c32mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

Test 1

P: OK
D: OK (FC: no differences encountered)

Test 2

P: EAccessViolation: Access violation at address 5378614D in module 'xtool.exe'. Read of address 5378614D
XTool is created by Razor12911

D: EReadError: Stream read error


Test 3

P: Exception EAccessViolation in module xtool.exe at 000BCEC7.
Access violation at address 004BCEC7 in module 'xtool.exe'. Read of address 00611D34.
D: EReadError: Stream read error

reflate works well with c16m in x86.:rolleyes:

Simorq 11-08-2019 14:11

zlib (XTool 1908_R2_x86)
 
#############################################
# zlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

Code:

XTool x86 Command Line

zlib
xtool.exe precomp:zlib:c32mb and c16mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res


TEST 1
P: OK
D: CRC Error (0F2C892C: 09 D7)

TEST 2
P: Error (EReadError: Stream read error)

TEST 3
P: Error (EReadError: Stream read error)

TEST 4
P: EAccessViolation: Access violation at address 00000000 in module 'xtool.exe'. Read of address 00000000

D: EReadError: Stream read error

zlib is very unstable, once successful, failed several times:(

Simorq 11-08-2019 14:15

rzlib (XTool 1908_R2_x86)
 
#############################################
# rzlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

Code:

XTool x86 Command Line

rzlib
xtool.exe precomp:rzlib:c16mb/c32mb/c64mb/c128mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

TEST 1
P: Error (EReadError: Stream read error)

TEST 2
P: Error (EReadError: Stream read error)

TEST 3
P: Error (EReadError: Stream read error)

TEST 4
P: Error (EReadError: Stream read error)

TEST 5
P: Error (EReadError: Stream read error)

rzlib always fails.:(

Razor12911 12-08-2019 15:26

@Simorq

Looks like I'll have to go through debugging road once more :D

@everyone

Update available

1908_R3
- Compatiblity broken (old history db and inputs cannot work in this version)
- Added plugin support
- Added LZ4 codec placeholders (lz4,lz4hc)
- Added LZO codec placeholders (lzo1c,lzo1x,lzo2a)
- Added ZSTD codec placeholders (zstd)
- Added Oodle codec placeholders (lzna,kraken,mermaid,selkie,leviathan)
- Added LZX codec placeholders (lzx)
- rzlib codec removed (use zlib)
- Added x86 memory limit

Notes
Placeholder does not mean the codec is ready to be used universally, it's intended for plugins which can be found here

ZAZA4EVER 12-08-2019 17:52

xtool_1908_R3 X64 ZLIB CODEC
eFootball Pes2020 Demo ... dt00_4K_x64.cpk
Code:

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Test.bin
Compressed 1 file, 15,252,601 => 72,893,019 bytes. Ratio 477.91%
Compression time: cpu 0.05 sec/real 37.50 sec = 0%. Speed 0.41 mB/s
All OK

Zlib Codec is Stable .. i tested it Three Times ...:D
/////////////////////////////////////////////////////////////
xtool_1908_R3 X64 oodle Codec
Wolfenstein Youngblood ... chunkbase_9_pc.resources

Code:

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Test.bin
Compressed 1 file, 100,552,642 => 100,552,714 bytes. Ratio 100.00%
Compression time: cpu 0.33 sec/real 3.38 sec = 10%. Speed 29.79 mB/s
All OK

oodle leviathan Stream Dont Work :o


All times are GMT -7. The time now is 20:39.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com