[Dev]XTool - Page 3

KaktoR · #31 21-03-2018, 16:01

I had a lot of work to do last days (f.e. compressing 2 games at the same time is a really pain). On weekend i will participate into test some usefull stuff with xtool.

Cheers and good work, as always

Webster17 · #32 21-03-2018, 18:48

Quote:

Originally Posted by Razor12911

Cool, I think I know what the issue is then. Thanks for participating in tests.
Will roll out another test where perhaps this issue will be fixed then move onto lz4 codec.

Welcome and I look forward to it

Webster17 · #33 27-03-2018, 21:50

Another test result here used on characters.vpp_pc from Saints Row Gat out of Hell Game. Ztool(pzlib) and precomp doesn't detect and recompress this file so the Competitors are Xtool, Pzlib3, reflate and Pzlib3+reflate compressed with srep+lzma. Here are the results

Razor12911 · #34 29-03-2018, 04:15

Quote:

Originally Posted by Webster17

Another test result here used on characters.vpp_pc from Saints Row Gat out of Hell Game. Ztool(pzlib) and precomp doesn't detect and recompress this file so the Competitors are Xtool, Pzlib3, reflate and Pzlib3+reflate compressed with srep+lzma. Here are the results

Thanks, I'm now thinking of pushing forward the first working codec in xtool.

Since the new rewritten scanner seems improved.

A few ideas have popped up since the last time I posted the tests so I may post a few more tests, regarding the processing part.

Webster17 · #35 29-03-2018, 20:33

Welcome and bring it on, it's a pleasure to help out.

Razor12911 · #36 03-04-2018, 16:32

Test 4

The rewritten new xtool project is really surprising lol, I didn't know I can draw even more speed from just rewriting the code... Unless if I made a mess in the older versions, I wish I had the old sources just to see what the hell I wrote in the code editor.

Here are my benchmarks:
Capture113.PNG

What I need from this test is you guys to post the cmd output.

if possible compress the outputs made my all the test programs and post results.

KaktoR · #37 03-04-2018, 16:55

Eh, i guess just drag&drop file to batch? Because it doesn't work "The command ProcProfile64.exe is wrong" or something.

Andu21 · #38 03-04-2018, 17:22

^Yes.

FF-C03C_MAIN.ZIP from Freedom Fighters:

Code:

Process ID       : 2604
Thread ID        : 1524
Process Exit Code: 0
Thread Exit Code : 0

User Time        :           2.137s
Kernel Time      :           0.078s
Process Time     :           2.215s
Clock Time       :           1.310s

Working Set      :           80056 KB
Paged Pool       :              84 KB
Nonpaged Pool    :               8 KB
Pagefile         :          167364 KB
Page Fault Count : 34069

IO Read          :           16161 KB (in             254 reads )
IO Write         :           35065 KB (in             548 writes)
IO Other         :               1 KB (in             188 others)


Process ID       : 2904
Thread ID        : 1372
Process Exit Code: 0
Thread Exit Code : 0

User Time        :           1.794s
Kernel Time      :           0.062s
Process Time     :           1.856s
Clock Time       :           1.718s

Working Set      :           61408 KB
Paged Pool       :              84 KB
Nonpaged Pool    :               9 KB
Pagefile         :           66436 KB
Page Fault Count : 26514

IO Read          :           16161 KB (in             254 reads )
IO Write         :           35065 KB (in             548 writes)
IO Other         :               1 KB (in             137 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 15.8 MB > 34.1 MB (216%)
Speed: 7.61 MB/s (read) 16.5 MB/s (write)
Duration: 2075 ms

Process ID       : 2312
Thread ID        : 1180
Process Exit Code: 0
Thread Exit Code : 0

User Time        :           4.368s
Kernel Time      :           0.031s
Process Time     :           4.399s
Clock Time       :           2.584s

Working Set      :           61504 KB
Paged Pool       :              84 KB
Nonpaged Pool    :               9 KB
Pagefile         :           67600 KB
Page Fault Count : 26718

IO Read          :           16161 KB (in              68 reads )
IO Write         :           35065 KB (in             147 writes)
IO Other         :               4 KB (in              97 others)

PACK5.000.tiger:

Code:

Process ID       : 2624
Thread ID        : 1328
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          31.028s
Kernel Time      :           0.514s
Process Time     :          31.542s
Clock Time       :          14.948s

Working Set      :          247680 KB
Paged Pool       :              84 KB
Nonpaged Pool    :              11 KB
Pagefile         :          281144 KB
Page Fault Count : 248746

IO Read          :          288344 KB (in            4507 reads )
IO Write         :          472134 KB (in            7378 writes)
IO Other         :               2 KB (in             379 others)


Process ID       : 2784
Thread ID        : 792
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         149.963s
Kernel Time      :           3.572s
Process Time     :         153.535s
Clock Time       :          42.476s

Working Set      :          296384 KB
Paged Pool       :              84 KB
Nonpaged Pool    :              11 KB
Pagefile         :          333916 KB
Page Fault Count : 2684676

IO Read          :          288342 KB (in            4507 reads )
IO Write         :          472199 KB (in            7379 writes)
IO Other         :               2 KB (in             316 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 282 MB > 461 MB (164%)
Speed: 4.80 MB/s (read) 7.87 MB/s (write)
Duration: 58609 ms

Process ID       : 2264
Thread ID        : 2292
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         218.744s
Kernel Time      :           0.514s
Process Time     :         219.258s
Clock Time       :          58.854s

Working Set      :          280776 KB
Paged Pool       :              84 KB
Nonpaged Pool    :              11 KB
Pagefile         :          322656 KB
Page Fault Count : 181916

IO Read          :          288341 KB (in            1132 reads )
IO Write         :          472199 KB (in            2145 writes)
IO Other         :              35 KB (in             268 others)

All files compressed using lolz.

Code:

FF-C03C_MAIN.ZIP.out1				34.2 MB (35,907,044 bytes) > 10.3 MB (10,840,373 bytes)
FF-C03C_MAIN.ZIP.out2				34.2 MB (35,907,118 bytes) > 10.3 MB (10,837,830 bytes)
FF-C03C_MAIN.ZIP.out3				34.2 MB (35,907,121 bytes) > 10.3 MB (10,840,977 bytes)
FF-C03C_MAIN.pcf(precomp047:intense:brute)	35.2 MB (36,985,675 bytes) > 10.1 MB (10,675,558 bytes)

Code:

PACK5.000.tiger.out1			461 MB (483,465,220 bytes) > 188 MB (197,647,249 bytes)
PACK5.000.tiger.out2			461 MB (483,532,609 bytes) > 189 MB (199,192,457 bytes)
PACK5.000.tiger.out3			461 MB (483,532,612 bytes) > 190 MB (199,349,867 bytes)
PACK5.000.pcf(precomp047:intense:brute)	461 MB (483,591,682 bytes) > 189 MB (198,587,760 bytes)

Gupta · #39 03-04-2018, 18:41

x64w.rpf

Code:

Process ID       : 9128
Thread ID        : 7836
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         278.125s
Kernel Time      :           1.375s
Process Time     :         279.500s
Clock Time       :          87.822s

Working Set      :          384048 KB
Paged Pool       :              94 KB
Nonpaged Pool    :              12 KB
Pagefile         :          428284 KB
Page Fault Count : 406712

IO Read          :          915372 KB (in           14304 reads )
IO Write         :         2078038 KB (in           32470 writes)
IO Other         :               4 KB (in             396 others)


Process ID       : 4284
Thread ID        : 3984
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         281.593s
Kernel Time      :           1.500s
Process Time     :         283.093s
Clock Time       :          81.404s

Working Set      :          397436 KB
Paged Pool       :              93 KB
Nonpaged Pool    :              12 KB
Pagefile         :          477376 KB
Page Fault Count : 436067

IO Read          :          915372 KB (in           14305 reads )
IO Write         :         2073414 KB (in           32399 writes)
IO Other         :               4 KB (in             255 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 894 MB > 1.98 GB (226%)
Speed: 6.82 MB/s (read) 15.5 MB/s (write)
Duration: 131016 ms

Process ID       : 11244
Thread ID        : 5772
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          33.581s
Kernel Time      :           1.203s
Process Time     :          34.784s
Clock Time       :         131.038s

Working Set      :          397756 KB
Paged Pool       :              93 KB
Nonpaged Pool    :              13 KB
Pagefile         :          439408 KB
Page Fault Count : 252851

IO Read          :          915372 KB (in            3579 reads )
IO Write         :         2073358 KB (in            9056 writes)
IO Other         :             183 KB (in             918 others)

Code:

x64w.rpf.out1.lzma 594185541
x64w.rpf.out2.lzma 597874786
x64w.rpf.out3.lzma 597841685

Mini · #40 04-04-2018, 00:57

arcrowd01.pac (size 95 207 424) from WWE2K18

Process ID : 309132
Thread ID : 309128
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.750s
Kernel Time : 0.203s
Process Time : 16.953s
Clock Time : 8.460s

Working Set : 226684 KB
Paged Pool : 98 KB
Nonpaged Pool : 11 KB
Pagefile : 251260 KB
Page Fault Count : 112870

IO Read : 92976 KB (in 1454 reads )
IO Write : 166352 KB (in 2600 writes)
IO Other : 2 KB (in 1299 others)

Process ID : 309272
Thread ID : 309256
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.156s
Kernel Time : 0.328s
Process Time : 16.484s
Clock Time : 8.180s

Working Set : 294064 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 336720 KB
Page Fault Count : 206861

IO Read : 92976 KB (in 1455 reads )
IO Write : 166354 KB (in 2601 writes)
IO Other : 2 KB (in 886 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 90.8 MB > 162 MB (179%)
Speed: 8.05 MB/s (read) 14.4 MB/s (write)
Duration: 11281 ms

Process ID : 309356
Thread ID : 309352
Process Exit Code: 0
Thread Exit Code : 0

User Time : 22.375s
Kernel Time : 0.250s
Process Time : 22.625s
Clock Time : 11.308s

Working Set : 272072 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 308976 KB
Page Fault Count : 169115

IO Read : 92976 KB (in 369 reads )
IO Write : 166354 KB (in 696 writes)
IO Other : 20 KB (in 657 others)

KaktoR · #41 04-04-2018, 03:40

GameData.pak from KCD
(268.562.432 Bytes)

Code:

Process ID       : 6964
Thread ID        : 8532
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          31.703s
Kernel Time      :           0.484s
Process Time     :          32.187s
Clock Time       :          10.829s

Working Set      :          350452 KB
Paged Pool       :              98 KB
Nonpaged Pool    :              12 KB
Pagefile         :          380008 KB
Page Fault Count : 258439

IO Read          :          262266 KB (in            4099 reads )
IO Write         :          500723 KB (in            7824 writes)
IO Other         :               2 KB (in             304 others)


Process ID       : 8396
Thread ID        : 7768
Process Exit Code: -1
Thread Exit Code : -1

User Time        :          78.218s
Kernel Time      :           0.734s
Process Time     :          78.952s
Clock Time       :          25.580s

Working Set      :          381624 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              12 KB
Pagefile         :          433332 KB
Page Fault Count : 402705

IO Read          :          262266 KB (in            4100 reads )
IO Write         :          408768 KB (in            6388 writes)
IO Other         :               2 KB (in             153 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 256 MB > 488 MB (191%)
Speed: 5.43 MB/s (read) 10.4 MB/s (write)
Duration: 47172 ms

Process ID       : 9104
Thread ID        : 6936
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         145.437s
Kernel Time      :           0.562s
Process Time     :         145.999s
Clock Time       :          47.723s

Working Set      :          372160 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              13 KB
Pagefile         :          424788 KB
Page Fault Count : 224668

IO Read          :          262266 KB (in            1027 reads )
IO Write         :          500082 KB (in            2123 writes)
IO Other         :              49 KB (in             381 others)

Panky · #42 04-04-2018, 12:06

animations-part1.pak (KCD)

Code:

Process ID       : 4208
Thread ID        : 12228
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          49.296s
Kernel Time      :           0.890s
Process Time     :          50.186s
Clock Time       :          28.901s

Working Set      :          516576 KB
Paged Pool       :              98 KB
Nonpaged Pool    :              13 KB
Pagefile         :          590036 KB
Page Fault Count : 401659

IO Read          :          342891 KB (in            5359 reads )
IO Write         :          552344 KB (in            8631 writes)
IO Other         :               2 KB (in             303 others)


Process ID       : 5184
Thread ID        : 12152
Process Exit Code: -1
Thread Exit Code : -1

User Time        :          16.406s
Kernel Time      :           1.218s
Process Time     :          17.624s
Clock Time       :           5.726s

Working Set      :          797276 KB
Paged Pool       :              97 KB
Nonpaged Pool    :              15 KB
Pagefile         :          913920 KB
Page Fault Count : 605176

IO Read          :          147520 KB (in            2306 reads )
IO Write         :           61952 KB (in             969 writes)
IO Other         :               2 KB (in             236 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 335 MB > 541 MB (161%)
Speed: 11.0 MB/s (read) 17.7 MB/s (write)
Duration: 30578 ms

Process ID       : 1532
Thread ID        : 5836
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          53.265s
Kernel Time      :           0.812s
Process Time     :          54.077s
Clock Time       :          30.574s

Working Set      :          536468 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              12 KB
Pagefile         :          591260 KB
Page Fault Count : 325641

IO Read          :          342891 KB (in            1342 reads )
IO Write         :          553956 KB (in            2315 writes)
IO Other         :              53 KB (in             425 others)

Razor12911 · #43 04-04-2018, 16:24

Quote:

Originally Posted by Mini

arcrowd01.pac (size 95 207 424) from WWE2K18

Process ID : 309132
Thread ID : 309128
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.750s
Kernel Time : 0.203s
Process Time : 16.953s
Clock Time : 8.460s

Working Set : 226684 KB
Paged Pool : 98 KB
Nonpaged Pool : 11 KB
Pagefile : 251260 KB
Page Fault Count : 112870

IO Read : 92976 KB (in 1454 reads )
IO Write : 166352 KB (in 2600 writes)
IO Other : 2 KB (in 1299 others)

Process ID : 309272
Thread ID : 309256
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.156s
Kernel Time : 0.328s
Process Time : 16.484s
Clock Time : 8.180s

Working Set : 294064 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 336720 KB
Page Fault Count : 206861

IO Read : 92976 KB (in 1455 reads )
IO Write : 166354 KB (in 2601 writes)
IO Other : 2 KB (in 886 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 90.8 MB > 162 MB (179%)
Speed: 8.05 MB/s (read) 14.4 MB/s (write)
Duration: 11281 ms

Process ID : 309356
Thread ID : 309352
Process Exit Code: 0
Thread Exit Code : 0

User Time : 22.375s
Kernel Time : 0.250s
Process Time : 22.625s
Clock Time : 11.308s

Working Set : 272072 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 308976 KB
Page Fault Count : 169115

IO Read : 92976 KB (in 369 reads )
IO Write : 166354 KB (in 696 writes)
IO Other : 20 KB (in 657 others)

0.3 seconds slower

I guess I have to work harder on this

KaktoR · #44 05-04-2018, 10:11

FFXII_TZA.vbf
(31.188.934.656 Bytes)

Code:

Process ID       : 4180
Thread ID        : 6696
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         341.998s
Kernel Time      :          43.109s
Process Time     :         385.107s
Clock Time       :        2455.929s

Working Set      :         1156248 KB
Paged Pool       :              98 KB
Nonpaged Pool    :              11 KB
Pagefile         :         1376796 KB
Page Fault Count : 2663686

IO Read          :        30457942 KB (in          475907 reads )
IO Write         :        57503890 KB (in          898499 writes)
IO Other         :               2 KB (in             338 others)


Process ID       : 744
Thread ID        : 6400
Process Exit Code: -1
Thread Exit Code : -1

User Time        :          48.187s
Kernel Time      :           0.859s
Process Time     :          49.046s
Clock Time       :          31.565s

Working Set      :          168704 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              11 KB
Pagefile         :          191520 KB
Page Fault Count : 102259

IO Read          :          671808 KB (in           10498 reads )
IO Write         :          618240 KB (in            9661 writes)
IO Other         :               2 KB (in             248 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 29.0 GB > 54.8 GB (189%)
Speed: 9.10 MB/s (read) 17.2 MB/s (write)
Duration: 3269438 ms

Process ID       : 8420
Thread ID        : 5912
Process Exit Code: 0
Thread Exit Code : 0

User Time        :           9.981s
Kernel Time      :          44.671s
Process Time     :          54.652s
Clock Time       :        3269.772s

Working Set      :         1370460 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              13 KB
Pagefile         :         1543420 KB
Page Fault Count : 876193

IO Read          :        30457942 KB (in          118980 reads )
IO Write         :        57500176 KB (in          250895 writes)
IO Other         :            4959 KB (in           19455 others)

As you can see, ztool just gave up on this, at least given options

Razor12911 · #45 05-04-2018, 16:09

Thanks for the tests guys. Here's another one, I added reflate lib support on this test, the inputs I'm interested in is data where reflate is required otherwise it will be just the same results as test number 4.

Png images, pdfs, linux isos and zip files contain streams that can be processed by reflate, games however that's DOOM and Dishonored 2 afaik.

So just like test 4, post cmd output.

I just like to point out, I'm not interested in the size that much but memory used and the time it took to perform the test.

These are the results of the tests I have performed:
Capture117.PNG

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

The Following 3 Users Say Thank You to Webster17 For This Useful Post:
Jiva newstone (28-03-2018), Razor12911 (29-03-2018), Simorq (28-03-2018)

The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
ShivShubh (24-11-2019), Simorq (31-03-2018)

The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
shazzla (03-04-2018), ShivShubh (24-11-2019)

The Following User Says Thank You to Andu21 For This Useful Post:
Razor12911 (03-04-2018)

The Following User Says Thank You to Gupta For This Useful Post:
Razor12911 (03-04-2018)

The Following User Says Thank You to Mini For This Useful Post:
Razor12911 (04-04-2018)

The Following User Says Thank You to KaktoR For This Useful Post:
Razor12911 (04-04-2018)

The Following User Says Thank You to Panky For This Useful Post:
Razor12911 (04-04-2018)

The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
doofoo24 (05-04-2018), ShivShubh (24-11-2019)

The Following User Says Thank You to KaktoR For This Useful Post:
Razor12911 (05-04-2018)

The Following User Says Thank You to Razor12911 For This Useful Post:
ShivShubh (24-11-2019)