[Dev]XTool
 

..

Hmm, a hint? Hopefully! :D

I can't wait for april fools day

And I will be ready again to fool someone:p:p:p

i think its xcore or may be it will be a april fool

Now you have destroyed the whole anticipation :D

XD, People be thanking two dots ".."

they thank you because you have a reputation. no the two dots ....

Good to know, I need these tested.

Test #1

* I want to know whether xtool performs give better output. (better output size is irrelevant, compress the output files individually and check which one was compressed better)
*For speed comparison, run tests on ramdisk or SSD

Games/Files to test on:
Video files (they may freeze xtool which is what I want)
DOOM 2016
Dishonored 2
And all other zlib games where pzlib failed to process

Post results like (Include times if you ran on ramdisk):
Xtool: 825GB > 1.69GB (Gave better output) took 12.592s
Ztool197: 825GB > 1.24GB (Gave better output) took 12.592s
Ztool198: 825GB > 1.24GB (Gave better output) took 12.592s

Don't change the parameters of ztool, they are made to match xtool

Test #2

* For this test, I want to see if the memory improvements worked, so just simply post the entire cmd output of the test you have run
* This test must be run on ramdisk or ssd

Games\Files to test on:
Anything that does not have deflate/zlib streams (e.g. Video files)

No Ram disk, but if u want i can perform on HDD.

I cant open test1.7z, winrar and 7zip says test 1 is corrupted. i downloaded twice

use latest version of winrar

Dishonored_DeathOfTheOutsider game2_001.resources 36.1 MB
TEST1

game2_001.resources.out1 97.2 МБ (101 969 487)
game2_001.resources.out2 36.1 MB (37 924 455)
game2_001.resources.out3 97.1 MB (101 827 433)

TEST2 - M.2 SSD
Process ID : 127384
Thread ID : 133524
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.078s
Process Time : 0.765s
Clock Time : 0.575s

Working Set : 164796 KB
Paged Pool : 101 KB
Nonpaged Pool : 12 KB
Pagefile : 221256 KB
Page Fault Count : 83384

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1724 others)


Process ID : 128596
Thread ID : 129300
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.062s
Process Time : 0.749s
Clock Time : 0.550s

Working Set : 175800 KB
Paged Pool : 101 KB
Nonpaged Pool : 11 KB
Pagefile : 249272 KB
Page Fault Count : 72029

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1688 others)

Thanks, so. The issue of not being able to detect DH2 streams under normal scan is fixed.
And data loader uses less memory.

Test 3

What is required here is running test on data with deflate/zlib streams by replacing zlibwapi.dll with different other zlib dlls that you can find, submit the dlls that do not work.

zlibwapi.dll
zlib1.dll
zlib.dll

are the dlls to look for to see whether xtool can handle all and at least give inflation.
N.B. exe is x86, so get x86 dll and not x64

Anything below version 1.2.11.0 doesn't work on my test file, it gives 0 bytes. It only happens if i change original name to "zlibwapi", strange enough xtool works without zlibwapi.dll.

Thanks for the test, xtool can work without zlibwapi, but only if zlib1.dll or zlib.dll is detected elsewhere and since Freearc registers dlls when it installs, probably it used the one in FA's directory.

:)

used it on Darksiders 2 file, test 1 recompressed it to 1gb but apperror occured, test 3 successfully worked

Preocmp gave better output, 1.12gb whereas 1.06 with xtool on above mentioned file.

Hmm, interesting. Did you compress the file afterwards, both outputs of xtool and precomp to see which one did a good precompression job?

working on it, afai tested previously, precomp is the lead winner against ztool and pzlibv3, its slow but efficient

Same here, ran many tests and somehow precomp seems to find streams that I have no idea as to where it finds them but compression after the precompressed data tells the true story.

i am still thinking about it!:confused:

@Razor In my case precomp and test3 produced same results. i tried files from halo 2, tom raider 2013 etc.

XTool+srep+lzma = 459mb = 1.06gb = 257mb
Precomp+srep+lzma = 459mb = 1.12gb = 239mb
Even after this I am damn impressed with XTool, its 10x faster compared to precomp, but the difference is about 16mb and it will be more if used on bigger files. Still ill ran some more tests. Also about Ztool it had the same issue while recompressing a zip file.

I want that 16mb, can you upload that file so I can see what's really going on.

Dev] XTool I want to use or try this application but I do not know anything, can you help?

Second to this Razor. Also, pdfs aren't properly detected either.

Very sorry bro, i wish i could but my internet speed sucks, it will take me 24 hours or more to upload it. This might help, I used offzip on it and the output size extracted was about 1.12gb, same as precomp output.

Cool, I think I know what the issue is then. Thanks for participating in tests.
Will roll out another test where perhaps this issue will be fixed then move onto lz4 codec.

I had a lot of work to do last days (f.e. compressing 2 games at the same time is a really pain). On weekend i will participate into test some usefull stuff with xtool.

Cheers and good work, as always ;)

Welcome and I look forward to it

Another test result here used on characters.vpp_pc from Saints Row Gat out of Hell Game. Ztool(pzlib) and precomp doesn't detect and recompress this file so the Competitors are Xtool, Pzlib3, reflate and Pzlib3+reflate compressed with srep+lzma. Here are the results

Thanks, I'm now thinking of pushing forward the first working codec in xtool.

Since the new rewritten scanner seems improved.

A few ideas have popped up since the last time I posted the tests so I may post a few more tests, regarding the processing part.

Welcome and bring it on, it's a pleasure to help out.

Test 4

The rewritten new xtool project is really surprising lol, I didn't know I can draw even more speed from just rewriting the code... Unless if I made a mess in the older versions, I wish I had the old sources just to see what the hell I wrote in the code editor.

Here are my benchmarks:
Attachment 21587

What I need from this test is you guys to post the cmd output.

if possible compress the outputs made my all the test programs and post results.

Eh, i guess just drag&drop file to batch? Because it doesn't work "The command ProcProfile64.exe is wrong" or something.

^Yes.

FF-C03C_MAIN.ZIP from Freedom Fighters:

PACK5.000.tiger:

All files compressed using lolz.

x64w.rpf

arcrowd01.pac (size 95 207 424) from WWE2K18


Process ID : 309132
Thread ID : 309128
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.750s
Kernel Time : 0.203s
Process Time : 16.953s
Clock Time : 8.460s

Working Set : 226684 KB
Paged Pool : 98 KB
Nonpaged Pool : 11 KB
Pagefile : 251260 KB
Page Fault Count : 112870

IO Read : 92976 KB (in 1454 reads )
IO Write : 166352 KB (in 2600 writes)
IO Other : 2 KB (in 1299 others)


Process ID : 309272
Thread ID : 309256
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.156s
Kernel Time : 0.328s
Process Time : 16.484s
Clock Time : 8.180s

Working Set : 294064 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 336720 KB
Page Fault Count : 206861

IO Read : 92976 KB (in 1455 reads )
IO Write : 166354 KB (in 2601 writes)
IO Other : 2 KB (in 886 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 90.8 MB > 162 MB (179%)
Speed: 8.05 MB/s (read) 14.4 MB/s (write)
Duration: 11281 ms

Process ID : 309356
Thread ID : 309352
Process Exit Code: 0
Thread Exit Code : 0

User Time : 22.375s
Kernel Time : 0.250s
Process Time : 22.625s
Clock Time : 11.308s

Working Set : 272072 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 308976 KB
Page Fault Count : 169115

IO Read : 92976 KB (in 369 reads )
IO Write : 166354 KB (in 696 writes)
IO Other : 20 KB (in 657 others)