FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   [Dev]XTool (https://fileforums.com/showthread.php?t=101613)

Razor12911 16-03-2018 15:20

[Dev]XTool
 
..

JustFun 16-03-2018 16:05

Hmm, a hint? Hopefully! :D

doofoo24 16-03-2018 16:45

I can't wait for april fools day

DedSec 16-03-2018 20:16

Quote:

Originally Posted by doofoo24 (Post 469102)
I can't wait for april fools day

And I will be ready again to fool someone:p:p:p

Jiva newstone 17-03-2018 00:46

i think its xcore or may be it will be a april fool

KaktoR 17-03-2018 03:07

Quote:

Originally Posted by DedSec (Post 469105)
And I will be ready again to fool someone:p:p:p

Now you have destroyed the whole anticipation :D

Razor12911 17-03-2018 14:53

Quote:

Originally Posted by Razor12911 (Post 469099)
..

XD, People be thanking two dots ".."

oltjon 17-03-2018 14:57

Quote:

Originally Posted by Razor12911 (Post 469152)
XD, People be thanking two dots ".."


they thank you because you have a reputation. no the two dots ....

Razor12911 17-03-2018 15:16

2 Attachment(s)
Good to know, I need these tested.

Test #1

* I want to know whether xtool performs give better output. (better output size is irrelevant, compress the output files individually and check which one was compressed better)
*For speed comparison, run tests on ramdisk or SSD

Games/Files to test on:
Video files (they may freeze xtool which is what I want)
DOOM 2016
Dishonored 2
And all other zlib games where pzlib failed to process

Post results like (Include times if you ran on ramdisk):
Xtool: 825GB > 1.69GB (Gave better output) took 12.592s
Ztool197: 825GB > 1.24GB (Gave better output) took 12.592s
Ztool198: 825GB > 1.24GB (Gave better output) took 12.592s

Don't change the parameters of ztool, they are made to match xtool

Test #2

* For this test, I want to see if the memory improvements worked, so just simply post the entire cmd output of the test you have run
* This test must be run on ramdisk or ssd

Games\Files to test on:
Anything that does not have deflate/zlib streams (e.g. Video files)

ZakirAhmad 17-03-2018 18:38

No Ram disk, but if u want i can perform on HDD.

ZakirAhmad 18-03-2018 01:44

I cant open test1.7z, winrar and 7zip says test 1 is corrupted. i downloaded twice

-NORO- 18-03-2018 02:34

Quote:

Originally Posted by ZakirAhmad (Post 469179)
I cant open test1.7z, winrar and 7zip says test 1 is corrupted. i downloaded twice

use latest version of winrar

Mini 18-03-2018 03:20

Dishonored_DeathOfTheOutsider game2_001.resources 36.1 MB
TEST1

Quote:


Process ID : 126624
Thread ID : 126816
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.640s
Kernel Time : 0.046s
Process Time : 0.686s
Clock Time : 0.583s

Working Set : 187852 KB
Paged Pool : 99 KB
Nonpaged Pool : 12 KB
Pagefile : 256260 KB
Page Fault Count : 88384

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 2 KB (in 1505 others)


Process ID : 130628
Thread ID : 125304
Process Exit Code: 0
Thread Exit Code : 0

User Time : 2.000s
Kernel Time : 0.062s
Process Time : 2.062s
Clock Time : 1.175s

Working Set : 78864 KB
Paged Pool : 96 KB
Nonpaged Pool : 9 KB
Pagefile : 85704 KB
Page Fault Count : 32166

IO Read : 37035 KB (in 581 reads )
IO Write : 37035 KB (in 580 writes)
IO Other : 2 KB (in 983 others)


Process ID : 129012
Thread ID : 132776
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.609s
Kernel Time : 0.187s
Process Time : 0.796s
Clock Time : 0.527s

Working Set : 205708 KB
Paged Pool : 98 KB
Nonpaged Pool : 8 KB
Pagefile : 229604 KB
Page Fault Count : 106539

IO Read : 37035 KB (in 581 reads )
IO Write : 99441 KB (in 1555 writes)
IO Other : 2 KB (in 955 others)
game2_001.resources.out1 97.2 МБ (101 969 487)
game2_001.resources.out2 36.1 MB (37 924 455)
game2_001.resources.out3 97.1 MB (101 827 433)

TEST2 - M.2 SSD
Process ID : 127384
Thread ID : 133524
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.078s
Process Time : 0.765s
Clock Time : 0.575s

Working Set : 164796 KB
Paged Pool : 101 KB
Nonpaged Pool : 12 KB
Pagefile : 221256 KB
Page Fault Count : 83384

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1724 others)


Process ID : 128596
Thread ID : 129300
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.062s
Process Time : 0.749s
Clock Time : 0.550s

Working Set : 175800 KB
Paged Pool : 101 KB
Nonpaged Pool : 11 KB
Pagefile : 249272 KB
Page Fault Count : 72029

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1688 others)

Razor12911 18-03-2018 17:12

Quote:

Originally Posted by Mini (Post 469183)
Dishonored_DeathOfTheOutsider game2_001.resources 36.1 MB
TEST1



game2_001.resources.out1 97.2 МБ (101 969 487)
game2_001.resources.out2 36.1 MB (37 924 455)
game2_001.resources.out3 97.1 MB (101 827 433)

TEST2 - M.2 SSD
Process ID : 127384
Thread ID : 133524
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.078s
Process Time : 0.765s
Clock Time : 0.575s

Working Set : 164796 KB
Paged Pool : 101 KB
Nonpaged Pool : 12 KB
Pagefile : 221256 KB
Page Fault Count : 83384

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1724 others)


Process ID : 128596
Thread ID : 129300
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.062s
Process Time : 0.749s
Clock Time : 0.550s

Working Set : 175800 KB
Paged Pool : 101 KB
Nonpaged Pool : 11 KB
Pagefile : 249272 KB
Page Fault Count : 72029

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1688 others)

Thanks, so. The issue of not being able to detect DH2 streams under normal scan is fixed.
And data loader uses less memory.

Razor12911 18-03-2018 17:17

1 Attachment(s)
Test 3

What is required here is running test on data with deflate/zlib streams by replacing zlibwapi.dll with different other zlib dlls that you can find, submit the dlls that do not work.

zlibwapi.dll
zlib1.dll
zlib.dll

are the dlls to look for to see whether xtool can handle all and at least give inflation.
N.B. exe is x86, so get x86 dll and not x64

Andu21 19-03-2018 05:34

1 Attachment(s)
Anything below version 1.2.11.0 doesn't work on my test file, it gives 0 bytes. It only happens if i change original name to "zlibwapi", strange enough xtool works without zlibwapi.dll.

Razor12911 19-03-2018 14:09

Quote:

Originally Posted by Andu21 (Post 469278)
Anything below version 1.2.11.0 doesn't work on my test file, it gives 0 bytes. It only happens if i change original name to "zlibwapi", strange enough xtool works without zlibwapi.dll.

Thanks for the test, xtool can work without zlibwapi, but only if zlib1.dll or zlib.dll is detected elsewhere and since Freearc registers dlls when it installs, probably it used the one in FA's directory.

:)

Webster17 19-03-2018 18:12

1 Attachment(s)
used it on Darksiders 2 file, test 1 recompressed it to 1gb but apperror occured, test 3 successfully worked

Webster17 19-03-2018 18:52

Preocmp gave better output, 1.12gb whereas 1.06 with xtool on above mentioned file.

Razor12911 19-03-2018 19:02

Hmm, interesting. Did you compress the file afterwards, both outputs of xtool and precomp to see which one did a good precompression job?

Webster17 19-03-2018 19:27

working on it, afai tested previously, precomp is the lead winner against ztool and pzlibv3, its slow but efficient

Razor12911 19-03-2018 19:56

Same here, ran many tests and somehow precomp seems to find streams that I have no idea as to where it finds them but compression after the precompressed data tells the true story.

PsYcHo_RaGE 19-03-2018 21:56

i am still thinking about it!:confused:

ZakirAhmad 19-03-2018 23:16

@Razor In my case precomp and test3 produced same results. i tried files from halo 2, tom raider 2013 etc.

Webster17 20-03-2018 00:29

XTool+srep+lzma = 459mb = 1.06gb = 257mb
Precomp+srep+lzma = 459mb = 1.12gb = 239mb
Even after this I am damn impressed with XTool, its 10x faster compared to precomp, but the difference is about 16mb and it will be more if used on bigger files. Still ill ran some more tests. Also about Ztool it had the same issue while recompressing a zip file.

Razor12911 20-03-2018 20:43

Quote:

Originally Posted by Webster17 (Post 469340)
XTool+srep+lzma = 459mb = 1.06gb = 257mb
Precomp+srep+lzma = 459mb = 1.12gb = 239mb
Even after this I am damn impressed with XTool, its 10x faster compared to precomp, but the difference is about 16mb and it will be more if used on bigger files. Still ill ran some more tests. Also about Ztool it had the same issue while recompressing a zip file.

I want that 16mb, can you upload that file so I can see what's really going on.

masen485 20-03-2018 20:59

Dev] XTool I want to use or try this application but I do not know anything, can you help?

Andu21 21-03-2018 04:12

Quote:

Originally Posted by Webster17 (Post 469340)
XTool+srep+lzma = 459mb = 1.06gb = 257mb
Precomp+srep+lzma = 459mb = 1.12gb = 239mb
Even after this I am damn impressed with XTool, its 10x faster compared to precomp, but the difference is about 16mb and it will be more if used on bigger files. Still ill ran some more tests. Also about Ztool it had the same issue while recompressing a zip file.

Second to this Razor. Also, pdfs aren't properly detected either.

Webster17 21-03-2018 07:52

Quote:

Originally Posted by Razor12911 (Post 469368)
I want that 16mb, can you upload that file so I can see what's really going on.

Very sorry bro, i wish i could but my internet speed sucks, it will take me 24 hours or more to upload it. This might help, I used offzip on it and the output size extracted was about 1.12gb, same as precomp output.

Razor12911 21-03-2018 14:06

Cool, I think I know what the issue is then. Thanks for participating in tests.
Will roll out another test where perhaps this issue will be fixed then move onto lz4 codec.

KaktoR 21-03-2018 15:01

I had a lot of work to do last days (f.e. compressing 2 games at the same time is a really pain). On weekend i will participate into test some usefull stuff with xtool.

Cheers and good work, as always ;)

Webster17 21-03-2018 17:48

Quote:

Originally Posted by Razor12911 (Post 469442)
Cool, I think I know what the issue is then. Thanks for participating in tests.
Will roll out another test where perhaps this issue will be fixed then move onto lz4 codec.

Welcome and I look forward to it

Webster17 27-03-2018 20:50

1 Attachment(s)
Another test result here used on characters.vpp_pc from Saints Row Gat out of Hell Game. Ztool(pzlib) and precomp doesn't detect and recompress this file so the Competitors are Xtool, Pzlib3, reflate and Pzlib3+reflate compressed with srep+lzma. Here are the results

Razor12911 29-03-2018 03:15

Quote:

Originally Posted by Webster17 (Post 469682)
Another test result here used on characters.vpp_pc from Saints Row Gat out of Hell Game. Ztool(pzlib) and precomp doesn't detect and recompress this file so the Competitors are Xtool, Pzlib3, reflate and Pzlib3+reflate compressed with srep+lzma. Here are the results

Thanks, I'm now thinking of pushing forward the first working codec in xtool.

Since the new rewritten scanner seems improved.

A few ideas have popped up since the last time I posted the tests so I may post a few more tests, regarding the processing part.

Webster17 29-03-2018 19:33

Welcome and bring it on, it's a pleasure to help out.

Razor12911 03-04-2018 15:32

2 Attachment(s)
Test 4

The rewritten new xtool project is really surprising lol, I didn't know I can draw even more speed from just rewriting the code... Unless if I made a mess in the older versions, I wish I had the old sources just to see what the hell I wrote in the code editor.

Here are my benchmarks:
Attachment 21587

What I need from this test is you guys to post the cmd output.

if possible compress the outputs made my all the test programs and post results.

KaktoR 03-04-2018 15:55

Eh, i guess just drag&drop file to batch? Because it doesn't work "The command ProcProfile64.exe is wrong" or something.

Andu21 03-04-2018 16:22

^Yes.

FF-C03C_MAIN.ZIP from Freedom Fighters:

Code:

Process ID      : 2604
Thread ID        : 1524
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          2.137s
Kernel Time      :          0.078s
Process Time    :          2.215s
Clock Time      :          1.310s

Working Set      :          80056 KB
Paged Pool      :              84 KB
Nonpaged Pool    :              8 KB
Pagefile        :          167364 KB
Page Fault Count : 34069

IO Read          :          16161 KB (in            254 reads )
IO Write        :          35065 KB (in            548 writes)
IO Other        :              1 KB (in            188 others)


Process ID      : 2904
Thread ID        : 1372
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          1.794s
Kernel Time      :          0.062s
Process Time    :          1.856s
Clock Time      :          1.718s

Working Set      :          61408 KB
Paged Pool      :              84 KB
Nonpaged Pool    :              9 KB
Pagefile        :          66436 KB
Page Fault Count : 26514

IO Read          :          16161 KB (in            254 reads )
IO Write        :          35065 KB (in            548 writes)
IO Other        :              1 KB (in            137 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 15.8 MB > 34.1 MB (216%)
Speed: 7.61 MB/s (read) 16.5 MB/s (write)
Duration: 2075 ms

Process ID      : 2312
Thread ID        : 1180
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          4.368s
Kernel Time      :          0.031s
Process Time    :          4.399s
Clock Time      :          2.584s

Working Set      :          61504 KB
Paged Pool      :              84 KB
Nonpaged Pool    :              9 KB
Pagefile        :          67600 KB
Page Fault Count : 26718

IO Read          :          16161 KB (in              68 reads )
IO Write        :          35065 KB (in            147 writes)
IO Other        :              4 KB (in              97 others)

PACK5.000.tiger:

Code:

Process ID      : 2624
Thread ID        : 1328
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          31.028s
Kernel Time      :          0.514s
Process Time    :          31.542s
Clock Time      :          14.948s

Working Set      :          247680 KB
Paged Pool      :              84 KB
Nonpaged Pool    :              11 KB
Pagefile        :          281144 KB
Page Fault Count : 248746

IO Read          :          288344 KB (in            4507 reads )
IO Write        :          472134 KB (in            7378 writes)
IO Other        :              2 KB (in            379 others)


Process ID      : 2784
Thread ID        : 792
Process Exit Code: 0
Thread Exit Code : 0

User Time        :        149.963s
Kernel Time      :          3.572s
Process Time    :        153.535s
Clock Time      :          42.476s

Working Set      :          296384 KB
Paged Pool      :              84 KB
Nonpaged Pool    :              11 KB
Pagefile        :          333916 KB
Page Fault Count : 2684676

IO Read          :          288342 KB (in            4507 reads )
IO Write        :          472199 KB (in            7379 writes)
IO Other        :              2 KB (in            316 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 282 MB > 461 MB (164%)
Speed: 4.80 MB/s (read) 7.87 MB/s (write)
Duration: 58609 ms

Process ID      : 2264
Thread ID        : 2292
Process Exit Code: 0
Thread Exit Code : 0

User Time        :        218.744s
Kernel Time      :          0.514s
Process Time    :        219.258s
Clock Time      :          58.854s

Working Set      :          280776 KB
Paged Pool      :              84 KB
Nonpaged Pool    :              11 KB
Pagefile        :          322656 KB
Page Fault Count : 181916

IO Read          :          288341 KB (in            1132 reads )
IO Write        :          472199 KB (in            2145 writes)
IO Other        :              35 KB (in            268 others)

All files compressed using lolz.
Code:

FF-C03C_MAIN.ZIP.out1                                34.2 MB (35,907,044 bytes) > 10.3 MB (10,840,373 bytes)
FF-C03C_MAIN.ZIP.out2                                34.2 MB (35,907,118 bytes) > 10.3 MB (10,837,830 bytes)
FF-C03C_MAIN.ZIP.out3                                34.2 MB (35,907,121 bytes) > 10.3 MB (10,840,977 bytes)
FF-C03C_MAIN.pcf(precomp047:intense:brute)        35.2 MB (36,985,675 bytes) > 10.1 MB (10,675,558 bytes)

Code:

PACK5.000.tiger.out1                        461 MB (483,465,220 bytes) > 188 MB (197,647,249 bytes)
PACK5.000.tiger.out2                        461 MB (483,532,609 bytes) > 189 MB (199,192,457 bytes)
PACK5.000.tiger.out3                        461 MB (483,532,612 bytes) > 190 MB (199,349,867 bytes)
PACK5.000.pcf(precomp047:intense:brute)        461 MB (483,591,682 bytes) > 189 MB (198,587,760 bytes)


Gupta 03-04-2018 17:41

x64w.rpf

Code:


Process ID      : 9128
Thread ID        : 7836
Process Exit Code: 0
Thread Exit Code : 0

User Time        :        278.125s
Kernel Time      :          1.375s
Process Time    :        279.500s
Clock Time      :          87.822s

Working Set      :          384048 KB
Paged Pool      :              94 KB
Nonpaged Pool    :              12 KB
Pagefile        :          428284 KB
Page Fault Count : 406712

IO Read          :          915372 KB (in          14304 reads )
IO Write        :        2078038 KB (in          32470 writes)
IO Other        :              4 KB (in            396 others)


Process ID      : 4284
Thread ID        : 3984
Process Exit Code: 0
Thread Exit Code : 0

User Time        :        281.593s
Kernel Time      :          1.500s
Process Time    :        283.093s
Clock Time      :          81.404s

Working Set      :          397436 KB
Paged Pool      :              93 KB
Nonpaged Pool    :              12 KB
Pagefile        :          477376 KB
Page Fault Count : 436067

IO Read          :          915372 KB (in          14305 reads )
IO Write        :        2073414 KB (in          32399 writes)
IO Other        :              4 KB (in            255 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 894 MB > 1.98 GB (226%)
Speed: 6.82 MB/s (read) 15.5 MB/s (write)
Duration: 131016 ms

Process ID      : 11244
Thread ID        : 5772
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          33.581s
Kernel Time      :          1.203s
Process Time    :          34.784s
Clock Time      :        131.038s

Working Set      :          397756 KB
Paged Pool      :              93 KB
Nonpaged Pool    :              13 KB
Pagefile        :          439408 KB
Page Fault Count : 252851

IO Read          :          915372 KB (in            3579 reads )
IO Write        :        2073358 KB (in            9056 writes)
IO Other        :            183 KB (in            918 others)

Code:

x64w.rpf.out1.lzma 594185541
x64w.rpf.out2.lzma 597874786
x64w.rpf.out3.lzma 597841685


Mini 03-04-2018 23:57

arcrowd01.pac (size 95 207 424) from WWE2K18


Process ID : 309132
Thread ID : 309128
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.750s
Kernel Time : 0.203s
Process Time : 16.953s
Clock Time : 8.460s

Working Set : 226684 KB
Paged Pool : 98 KB
Nonpaged Pool : 11 KB
Pagefile : 251260 KB
Page Fault Count : 112870

IO Read : 92976 KB (in 1454 reads )
IO Write : 166352 KB (in 2600 writes)
IO Other : 2 KB (in 1299 others)


Process ID : 309272
Thread ID : 309256
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.156s
Kernel Time : 0.328s
Process Time : 16.484s
Clock Time : 8.180s

Working Set : 294064 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 336720 KB
Page Fault Count : 206861

IO Read : 92976 KB (in 1455 reads )
IO Write : 166354 KB (in 2601 writes)
IO Other : 2 KB (in 886 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 90.8 MB > 162 MB (179%)
Speed: 8.05 MB/s (read) 14.4 MB/s (write)
Duration: 11281 ms

Process ID : 309356
Thread ID : 309352
Process Exit Code: 0
Thread Exit Code : 0

User Time : 22.375s
Kernel Time : 0.250s
Process Time : 22.625s
Clock Time : 11.308s

Working Set : 272072 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 308976 KB
Page Fault Count : 169115

IO Read : 92976 KB (in 369 reads )
IO Write : 166354 KB (in 696 writes)
IO Other : 20 KB (in 657 others)


All times are GMT -7. The time now is 12:43.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com