FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   XTool 2020 (Main Project) (https://fileforums.com/showthread.php?t=102832)

Razor12911 16-06-2019 04:53

XTool 2020 (Main Project)
 
1 Attachment(s)
XTool is program made specifically repackaging games by providing a full suite of useful features such as data precompression, archiving, encryption and etc.

With that being said, nothing restricts it from being used on everyday files such as documents, pictures and media but with few limitations.

Read the documentation to find out how it works and how to use it.

Link for older releases.

FitGirl 17-06-2019 18:58

Съешь ещё этих мягких французских булок, да выпей же чаю.
Google this if you want to know what it is and how it relates to the topic ;)

Razor12911 20-06-2019 15:10

Good thing I know a bit of Russian (and by a bit, I mean a drop in an ocean) but from what I can tell, all Cyrillic letters (Russian) have been used :)

ShivShubh 21-06-2019 05:21

Nice to see you still working for the community, much respect !

shiyamrrc 23-06-2019 03:36

Waiting for the precompressor.
Respect

FitGirl 23-06-2019 17:53

Quote:

Originally Posted by Razor12911 (Post 481426)
Good thing I know a bit of Russian (and by a bit, I mean a drop in an ocean) but from what I can tell, all Cyrillic letters (Russian) have been used :)

Correct answer, here's your ONE MILLION USD! :)

Razor12911 24-06-2019 09:22

Where??? :D:D:D

FitGirl 27-06-2019 20:40

Quote:

Originally Posted by Razor12911 (Post 481480)
Where??? :D:D:D

Here in Lagos, the capital of Nigeria. It's stashed in our Global Repackers Bank, all you need is just send me 500 USD in Western Union and I will gladly help you to get all amount from the bank. Trust me, I never lie!

Grumpy 28-06-2019 06:19

Quote:

Originally Posted by FitGirl (Post 481537)
Here in Lagos, the capital of Nigeria. It's stashed in our Global Repackers Bank, all you need is just send me 500 USD in Western Union and I will gladly help you to get all amount from the bank. Trust me, I never lie!

It gave me a good laugh :D

Razor12911 29-06-2019 16:13

Yea right... sounds like a scam :(

@everyone
progress of the project is about 80%, should have been done by now if weren't steam sale.

KaktoR 29-06-2019 16:47

Quote:

Originally Posted by Razor12911 (Post 481553)
steam sale.

I know that feel bro.

And should I tell you something... Most of my games I never played. I just bought them in a sale or from keysellers. I don't know why I buy games and never play them lol.

doofoo24 30-06-2019 14:25

did you fix far cry bug ?
and did you add support for for watch dog/watch dog2/crash bandicoot :o:o:o

elit 05-07-2019 16:12

Quote:

Originally Posted by FitGirl (Post 481537)
Here in Lagos, the capital of Nigeria. It's stashed in our Global Repackers Bank, all you need is just send me 500 USD in Western Union and I will gladly help you to get all amount from the bank. Trust me, I never lie!

I almost fell for it back then, was already in the bank opening secondary account for said incoming 1m$, then he said in the phone he need 500$ first for a lawyer to "finish formalities". Being hardcore "Jew" when it comes to money, I said I don't want 1m$ if it cost me 500$ to get them. And that was it.

PS: Razor, your tool broke PornHub. (nuked) :)

Razor12911 13-07-2019 15:12

1 Attachment(s)
Here's a test for stability

simply drag and drop a sample file that has deflate streams, do this at least 5 times, the program should print out numbers in ascending order, these are the number of streams, not really important but what you must test is in those 5 tests, the same numbers are produced

As for the project's progress, well all seems good but I'm just making sure xtool never faces errors and if it does it should try to recover before it hangs or quits.

Harsh ojha 13-07-2019 21:08

Thx @Razor12911
I will test

shazzla 14-07-2019 02:08

Did it five times ,as you requested.

Layers of fear 2 - pakchunk2-WindowsNoEditor.pak - OK !
Legacy of the void - data.006 - OK !
X4 Foundations - 05.dat - OK !

Interesting : in all testfiles the last two numbers were the same/file :

05.dat ends : 11372 ,11372.
data.006 : 110507,110507.
pakchunk2-WindowsNoEditor.pak : 34990,34990.
Maybe is it a bug ?!

Thanks for your work !

KaktoR 14-07-2019 04:12

Did it aswell with Kingdom Come Deliverance and a few pdf files.

All seems to be the way it should be!

Razor12911 14-07-2019 15:36

Quote:

Originally Posted by shazzla (Post 481790)
Did it five times ,as you requested.

Layers of fear 2 - pakchunk2-WindowsNoEditor.pak - OK !
Legacy of the void - data.006 - OK !
X4 Foundations - 05.dat - OK !

Interesting : in all testfiles the last two numbers were the same/file :

05.dat ends : 11372 ,11372.
data.006 : 110507,110507.
pakchunk2-WindowsNoEditor.pak : 34990,34990.
Maybe is it a bug ?!

Thanks for your work !

I actually saw those last repeating numbers, I already fixed the problem after uploading the attachment, it's not really a big deal but thanks for the test.

@everyone
there are more tests to come.

doofoo24 14-07-2019 15:41

Quote:

Originally Posted by Razor12911 (Post 481800)
@everyone
there are more tests to come.

lz4,lzo,zstd...
is there any speed improvement in raw2hif_dll/hif2raw_dll, for exmaple xtool is slow on doom/cod ghost ?

Razor12911 15-07-2019 16:22

well that is the thing, I didn't make those libraries, if they are slow, they are slow nothing I can do unless if you contact the creator here:
https://encode.ru/threads/1399-refla...e-recompressor

as for doom, well I had plans for that game, there will be a plugin that will come with xtool called idtech6.dll or something similar. The plugin is specially designed to work best for games that use that engine (also dishonored 2, doto), handling zlib specifically for them unlike the universal zlib that tries all possible methods to give you the best output, the plugin will know beforehand what to do and how to handle the streams from those games and speed up the process.

xtool.exe e:precomp:t100p:idtech6 - -
is how you will use it, there are several games that if they require a special touch, they will just come separate from xtool in plugin form like dunia, unity and etc, which is why I created a separate thread here because I reckon there is going to be many.

Razor12911 17-07-2019 15:19

1 Attachment(s)
Here's a speed test

prepare two types of inputs, one with deflate streams and one without deflate streams, can be a video file.

drag and drop the test files onto test.bat, there will be a console output, upload those results as test report.

The test is between xtool v012 and the upcoming release.

Note: Your input shouldn't be from the game that use idTech6 engine (DOOM, Dishonored 2 and Death of the Outsider)

doofoo24 17-07-2019 16:54

1 Attachment(s)
test on HellbladeSenuasSacrifice (HellbladeGame-WindowsNoEditor.pak)...

Process ID : 5172
Thread ID : 3824
Process Exit Code: 0
Thread Exit Code : 0

User Time : 153.018s
Kernel Time : 28.031s
Process Time : 181.049s
Clock Time : 129.848s

Working Set : 881712 KB
Paged Pool : 109 KB
Nonpaged Pool : 18 KB
Pagefile : 1133300 KB
Page Fault Count : 541119

IO Read : 16928698 KB (in 264520 reads )
IO Write : 26142474 KB (in 3999156 writes)
IO Other : 2 KB (in 130 others)

WOW Super fast 360mb/s with half cpu usage...

Andu21 17-07-2019 20:34

Echo
 
CI_Sc090.mp4

Code:

xtool_new.exe:

Process ID      : 3920
Thread ID        : 2620
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          20.779s
Kernel Time      :          0.265s
Process Time    :          21.044s
Clock Time      :          6.797s

Working Set      :          104788 KB
Paged Pool      :              91 KB
Nonpaged Pool    :              7 KB
Pagefile        :          163964 KB
Page Fault Count : 26277

IO Read          :          354842 KB (in            5548 reads )
IO Write        :          354826 KB (in              46 writes)
IO Other        :              5 KB (in            240 others)

xtool_old.exe:

Process ID      : 4104
Thread ID        : 4888
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          40.856s
Kernel Time      :          0.187s
Process Time    :          41.043s
Clock Time      :          11.781s

Working Set      :          106836 KB
Paged Pool      :              93 KB
Nonpaged Pool    :              7 KB
Pagefile        :          164968 KB
Page Fault Count : 26918

IO Read          :          354843 KB (in            5547 reads )
IO Write        :          354826 KB (in            5545 writes)
IO Other        :              1 KB (in            220 others)

Echo-WindowsNoEditor.pak

Code:

xtool_new.exe:

Process ID      : 4832
Thread ID        : 2644
Process Exit Code: 0
Thread Exit Code : 0

User Time        :        196.046s
Kernel Time      :          3.322s
Process Time    :        199.368s
Clock Time      :        489.965s

Working Set      :          213032 KB
Paged Pool      :              91 KB
Nonpaged Pool    :              9 KB
Pagefile        :          282924 KB
Page Fault Count : 90883

IO Read          :        4233301 KB (in          66149 reads )
IO Write        :        4476621 KB (in          171112 writes)
IO Other        :              6 KB (in            240 others)

xtool_old.exe:

Process ID      : 2736
Thread ID        : 4384
Process Exit Code: 0
Thread Exit Code : 259

User Time        :        376.648s
Kernel Time      :          2.792s
Process Time    :        379.440s
Clock Time      :        464.932s

Working Set      :          214460 KB
Paged Pool      :              93 KB
Nonpaged Pool    :              9 KB
Pagefile        :          287852 KB
Page Fault Count : 100289

IO Read          :        4233303 KB (in          66148 reads )
IO Write        :        4446197 KB (in          69472 writes)
IO Other        :              1 KB (in            232 others)


dixen 17-07-2019 21:32

bundle_pc.ipk (Rayman Origins)

Quote:

xtool_new.exe:

Process ID : 9272
Thread ID : 1356
Process Exit Code: 0
Thread Exit Code : 0

User Time : 60.453s
Kernel Time : 1.546s
Process Time : 61.999s
Clock Time : 21.767s

Working Set : 448404 KB
Paged Pool : 110 KB
Nonpaged Pool : 11 KB
Pagefile : 540260 KB
Page Fault Count : 211743

IO Read : 2046443 KB (in 31979 reads )
IO Write : 3038292 KB (in 5711 writes)
IO Other : 6 KB (in 202 others)

xtool_old.exe:


Process ID : 1220
Thread ID : 4636
Process Exit Code: 0
Thread Exit Code : 0

User Time : 130.625s
Kernel Time : 2.593s
Process Time : 133.218s
Clock Time : 44.855s

Working Set : 2084988 KB
Paged Pool : 111 KB
Nonpaged Pool : 10 KB
Pagefile : 2291268 KB
Page Fault Count : 1055801

IO Read : 2046448 KB (in 31978 reads )
IO Write : 3038286 KB (in 47474 writes)
IO Other : 3 KB (in 195 others)

Razor12911 18-07-2019 15:10

Thanks for the tests, 95% complete. the new release will first ship with 1 codec (zlib) just to see if everything works as expected

IgorKolesnik 19-07-2019 06:49

Quote:

Originally Posted by Razor12911 (Post 481842)
Thanks for the tests, 95% complete. the new release will first ship with 1 codec (zlib) just to see if everything works as expected

can update game file scaner? thank

Razor12911 20-07-2019 17:37

Quote:

Originally Posted by IgorKolesnik (Post 481844)
can update game file scaner? thank

and add what feature? :rolleyes:

@everyone

here's a benchmark

some file from Pro Evolution Soccer

old xtool = 275MB >> 381MB took 11.63 seconds using 4 threads
new xtool = 275MB >> 381MB took 06.09 seconds using 4 threads

same memory usage and I/O

note: I only reworked the encoder to use the cpu effectively so you should expect roughly x1.25-2 more speed from old version while using same resources and this applies to all codecs reflate, oodle and etc

Simorq 21-07-2019 01:35

Hi Razor12911
Code:

v: skip verification
This option works for oodle:D, crilayla:D, lzo:D.
Please do not delete this option in the new version.

KaktoR 21-07-2019 02:10

Quote:

Originally Posted by Razor12911 (Post 481868)
and add what feature? :rolleyes:

Just fix the hang bug :D
Quote:

Originally Posted by Razor12911 (Post 481868)
@everyone

here's a benchmark

some file from Pro Evolution Soccer

old xtool = 275MB >> 381MB took 11.63 seconds using 4 threads
new xtool = 275MB >> 381MB took 06.09 seconds using 4 threads

same memory usage and I/O

note: I only reworked the encoder to use the cpu effectively so you should expect roughly x1.25-2 more speed from old version while using same resources and this applies to all codecs reflate, oodle and etc

This is great. :cool:

dixen 21-07-2019 04:43

Quote:

Originally Posted by Razor12911 (Post 481868)
and add what feature? :rolleyes:

oodle detect, for example:D

shazzla 21-07-2019 13:20

If game uses oodle,then you can find its DLLs in game folder somewhere.
Imho : oodle stream scanner is obsolete ?!

Razor12911 21-07-2019 15:29

Quote:

Originally Posted by Simorq (Post 481873)
Hi Razor12911
Code:

v: skip verification
This option works for oodle:D, crilayla:D, lzo:D.
Please do not delete this option in the new version.

The option will be there

Quote:

Originally Posted by dixen (Post 481875)
oodle detect, for example:D

I remember including an oodle scanner, but fine I'll think of something for the scanner.

Quote:

Originally Posted by KaktoR (Post 481874)
This is great. :cool:

how about this?

Quote:

There will be a new feature in xtool called history data, what it does is simply store processed stream information and be able to recall it at the later stage.

If you are wondering how this helps, well there are certain games made by lazy developers who repeat game resources instead of just reusing them, resulting in bigger game size for no absolute reason, apart from that, if they are compressed streams it means that xtool will have to process the same stream over and over again, storing history data allows it to prevent the same stream from being reprocessed instead, it will just look at the history and find out how the previous copy was processed then automatically assume it's the same stream and then apply the same settings, resulting in more speed.

I ran a test on a file I repeated on purpose and these were the results.

old xtool took 196.671 seconds and used 335MB ram
new xtool took 154.174 seconds and used 319MB ram
new xtool with history feature took 72.223 seconds and used 439MB ram

the cost of the feature depends on how much history you want to be stored but with default settings, it should use an additonal 192mb ram usage than without, you can always disable it or order it to store more history, depends on how much memory you have to spare.
Note:
In a practical situation, you can shave off a few seconds but then again, depends on the input

KaktoR 23-07-2019 07:18

This is also good :D

Did you have to make changes to the libraries too? Or just the source of xtool executable? Just corious :)

Razor12911 23-07-2019 15:04

Only the source of xtool, all I did was optimize optimize and optimize, the CPU is forever doing something, I tried to minimize the gaps between reads and writes by making xtool process stuff while it is reading and writing so yeah, the threads hardly get rests now.

Razor12911 26-07-2019 16:55

1 Attachment(s)
Project status = Welp...

I hope this isn't strike 2 in terms losing my sources, my PC died about 2 days ago, motherboard and GPU got fried this time XD. As for my sources well they were stored on my main PC (the dead one) and I don't know if they are safe since I don't have a spare PC to boot up the drive to grab important files and store them to my backup drive, so I can't really tell if the only components that died were only the board and GPU and not my HDD, but I guess that's a story for some other time. If you are wondering what this means in terms of xtool 2019...

Well since I normally code from desktop because I don't need to go far to do things and etc, it was different for xtool somehow... because I was coding from a backup drive only for that project like I knew some shit like this would happen so yes the source is safe (lucky af), if you are wondering about the development now that my main PC is busted.

Well I just had to dust an old laptop and continued development there, issue is it's incredibly slow but it will have to do. If you are wondering why I only provided x86 binaries, well. OS is 32-bit :D, somehow can't compile for x64.

Anyways enough about that, this was suppose to be the last test before the first release, but before I uploaded, I noticed some shitty error in made in the reflate code and I have to rewrite some parts of it so after fixing that, I'll post the final test and if all is good, I'll be making a release.

there are 3 codecs in xtool right now (zlib, reflate, rzlib). The bug was in reflate in multi threaded mode when decoding, but if you want to try it out anyways, open the test bat file. replace zlib with reflate or rzlib but use 1 thread to decode.

zlib = xtool uses zlib1.dll/zlibwapi.dll to process data, fails on some streams
reflate = xtool uses reflate dlls to process data, processes all streams but slow
rzlib = xtool blends zlib and reflate dlls to process data, processes all streams but with better size and speed.

PS: Even the archive is zip, don't have 7-Zip installed for my normal 7z attachments :o
PPS: Nothing is stopping the development of this project XD... I hope cause you never know

FitGirl 26-07-2019 18:33

Shit, mate! You're lucky to have not loosing sources this time! Gotta use some repo soft next time for more security, no? :)

Anyways, it's nice to see the project is coming to release.

Harsh ojha 26-07-2019 21:01

@Razor12911
Download Windows 10 64 bit Archived file and install

https://youtu.be/zGQ9xXYgVJM
Since i also lost my data

Well we are all depends our pc and laptops

Razor12911 28-07-2019 15:14

First release

Notes

- No idea whether x64 works or not, I just somehow managed to compile. That's up to you to find out.
- Reflate currently is set to use level 6, if you are looking for better output, best to use the old xtool. The reason it's like this is because I'm still working on a way to improve the horrible slow code I used before.
- Skip verification option is currently unavailable.

What I can say, run a lot of tests on this release. I've written some dangerous memory saving code and it may cause errors but if all is well then I can move on to add other codecs such as oodle, lzo, lz4 and the rest. BTW, crilayla will NOT be added, that piece of crap is incredibly slow :D

Simorq 28-07-2019 16:11

Image003.png
 
xtool_1907_R1 64Bit
Code:

Creating archive: data.arc using zlib
Compressed 1 file, 668,113 => 668,225 bytes. Ratio 100.02%               
Compression time: cpu 0.02 sec/real 0.18 sec = 9%. Speed 3.71 mB/s
All OK

Testing archive: data.arc
Tested 1 file, 668,225 => 668,113 bytes. Ratio 100.02%                   
Testing time: cpu 0.00 sec/real 0.06 sec = 0%. Speed 10.77 mB/s
All OK

Code:

Creating archive: data.arc using reflate
Compressed 1 file, 668,113 => 1,767,491 bytes. Ratio 264.55%             
Compression time: cpu 0.00 sec/real 0.19 sec = 0%. Speed 3.44 mB/s
All OK

Testing archive: data.arc
Tested 1 file, 1,767,491 => 668,113 bytes. Ratio 264.55%                 
Testing time: cpu 0.00 sec/real 0.16 sec = 0%. Speed 4.31 mB/s
All OK

Code:

Creating archive: data.arc using rzlib
Compressed 1 file, 668,113 => 1,767,491 bytes. Ratio 264.55%             
Compression time: cpu 0.00 sec/real 0.28 sec = 0%. Speed 2.40 mB/s
All OK

Testing archive: data.arc
Tested 1 file, 1,767,491 => 668,113 bytes. Ratio 264.55%                 
Testing time: cpu 0.00 sec/real 0.16 sec = 0%. Speed 4.25 mB/s
All OK


Simorq 28-07-2019 16:30

xtool_1907_R1 Test
 
Does not work with ba2/csg files.
Code:

Fallout4 - Textures7.ba2 / DLCCoast - Geometry.csg
32bit= ERROR: write error (disk full?) in compression algorithm zlib/reflate/rzlib
64bit= ERROR: write error (disk full?) in compression algorithm zlib/reflate/rzlib or stuck cpu
http://s8.picofile.com/file/8368123326/zlib.jpg

-------------------------------------------------------------------

Works with this setting.
Code:

[External compressor:rzlib,zlib,reflate]
header    = 0
packcmd  = xtool.exe precomp:rzlib:c32mb,t1 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = xtool.exe decode:t50p - - <stdin> <stdout>

Code:

Fallout4 - Textures7.ba2 520 MB
ZLIB
Code:

Creating archive: data.arc using zlib
Compressed 1 file, 545,959,220 => 1,237,654,873 bytes. Ratio 226.69%     
Compression time: cpu 0.52 sec/real 84.11 sec = 1%. Speed 6.49 mB/s
All OK

zlib: decode:t100p
Testing archive: data.arc
Tested 1 file, 1,237,654,873 => 545,959,220 bytes. Ratio 226.69%         
Testing time: cpu 0.45 sec/real 9.53 sec = 5%. Speed 57.29 mB/s
All OK

REFLATE
Code:

Creating archive: data.arc using reflate
Compressed 1 file, 545,959,220 => 1,237,783,515 bytes. Ratio 226.72%     
Compression time: cpu 0.42 sec/real 113.43 sec = 0%. Speed 4.81 mB/s
All OK

reflate: decode:t100p
Testing archive: data.arc
Tested 1 file, 1,237,783,515 => 545,959,220 bytes. Ratio 226.72%         
Testing time: cpu 0.36 sec/real 12.23 sec = 3%. Speed 44.62 mB/s
All OK

RZLIB
Code:

Creating archive: data.arc using rzlib
Compressed 1 file, 545,959,220 => 1,237,654,873 bytes. Ratio 226.69%     
Compression time: cpu 0.53 sec/real 85.22 sec = 1%. Speed 6.41 mB/s
All OK

rzlib: decode:t2
Testing archive: data.arc
Tested 1 file, 1,237,654,873 => 545,959,220 bytes. Ratio 226.69%         
Testing time: cpu 0.98 sec/real 73.38 sec = 1%. Speed 7.44 mB/s
All OK

rzlib: decode:t100p
Testing archive: data.arc
Tested 1 file, 1,237,654,873 => 545,959,220 bytes. Ratio 226.69%         
Testing time: cpu 0.77 sec/real 73.54 sec = 1%. Speed 7.42 mB/s
All OK



All times are GMT -7. The time now is 05:43.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com