XTool 2020 (Main Project)
 

XTool is program made specifically repackaging games by providing a full suite of useful features such as data precompression, archiving, encryption and etc.

With that being said, nothing restricts it from being used on everyday files such as documents, pictures and media but with few limitations.

Read the documentation to find out how it works and how to use it.

Link for older releases.

Съешь ещё этих мягких французских булок, да выпей же чаю.
Google this if you want to know what it is and how it relates to the topic ;)

Good thing I know a bit of Russian (and by a bit, I mean a drop in an ocean) but from what I can tell, all Cyrillic letters (Russian) have been used :)

Nice to see you still working for the community, much respect !

Waiting for the precompressor.
Respect

Correct answer, here's your ONE MILLION USD! :)

Where??? :D:D:D

Here in Lagos, the capital of Nigeria. It's stashed in our Global Repackers Bank, all you need is just send me 500 USD in Western Union and I will gladly help you to get all amount from the bank. Trust me, I never lie!

It gave me a good laugh :D

Yea right... sounds like a scam :(

@everyone
progress of the project is about 80%, should have been done by now if weren't steam sale.

I know that feel bro.

And should I tell you something... Most of my games I never played. I just bought them in a sale or from keysellers. I don't know why I buy games and never play them lol.

did you fix far cry bug ?
and did you add support for for watch dog/watch dog2/crash bandicoot :o:o:o

I almost fell for it back then, was already in the bank opening secondary account for said incoming 1m$, then he said in the phone he need 500$ first for a lawyer to "finish formalities". Being hardcore "Jew" when it comes to money, I said I don't want 1m$ if it cost me 500$ to get them. And that was it.

PS: Razor, your tool broke PornHub. (nuked) :)

Here's a test for stability

simply drag and drop a sample file that has deflate streams, do this at least 5 times, the program should print out numbers in ascending order, these are the number of streams, not really important but what you must test is in those 5 tests, the same numbers are produced

As for the project's progress, well all seems good but I'm just making sure xtool never faces errors and if it does it should try to recover before it hangs or quits.

Thx @Razor12911
I will test

Did it five times ,as you requested.

Layers of fear 2 - pakchunk2-WindowsNoEditor.pak - OK !
Legacy of the void - data.006 - OK !
X4 Foundations - 05.dat - OK !

Interesting : in all testfiles the last two numbers were the same/file :

05.dat ends : 11372 ,11372.
data.006 : 110507,110507.
pakchunk2-WindowsNoEditor.pak : 34990,34990.
Maybe is it a bug ?!

Thanks for your work !

Did it aswell with Kingdom Come Deliverance and a few pdf files.

All seems to be the way it should be!

I actually saw those last repeating numbers, I already fixed the problem after uploading the attachment, it's not really a big deal but thanks for the test.

@everyone
there are more tests to come.

lz4,lzo,zstd...
is there any speed improvement in raw2hif_dll/hif2raw_dll, for exmaple xtool is slow on doom/cod ghost ?

well that is the thing, I didn't make those libraries, if they are slow, they are slow nothing I can do unless if you contact the creator here:
https://encode.ru/threads/1399-refla...e-recompressor

as for doom, well I had plans for that game, there will be a plugin that will come with xtool called idtech6.dll or something similar. The plugin is specially designed to work best for games that use that engine (also dishonored 2, doto), handling zlib specifically for them unlike the universal zlib that tries all possible methods to give you the best output, the plugin will know beforehand what to do and how to handle the streams from those games and speed up the process.

xtool.exe e:precomp:t100p:idtech6 - -
is how you will use it, there are several games that if they require a special touch, they will just come separate from xtool in plugin form like dunia, unity and etc, which is why I created a separate thread here because I reckon there is going to be many.

Here's a speed test

prepare two types of inputs, one with deflate streams and one without deflate streams, can be a video file.

drag and drop the test files onto test.bat, there will be a console output, upload those results as test report.

The test is between xtool v012 and the upcoming release.

Note: Your input shouldn't be from the game that use idTech6 engine (DOOM, Dishonored 2 and Death of the Outsider)

test on HellbladeSenuasSacrifice (HellbladeGame-WindowsNoEditor.pak)...

Process ID : 5172
Thread ID : 3824
Process Exit Code: 0
Thread Exit Code : 0

User Time : 153.018s
Kernel Time : 28.031s
Process Time : 181.049s
Clock Time : 129.848s

Working Set : 881712 KB
Paged Pool : 109 KB
Nonpaged Pool : 18 KB
Pagefile : 1133300 KB
Page Fault Count : 541119

IO Read : 16928698 KB (in 264520 reads )
IO Write : 26142474 KB (in 3999156 writes)
IO Other : 2 KB (in 130 others)

WOW Super fast 360mb/s with half cpu usage...

Echo
 

CI_Sc090.mp4

Echo-WindowsNoEditor.pak

bundle_pc.ipk (Rayman Origins)

Thanks for the tests, 95% complete. the new release will first ship with 1 codec (zlib) just to see if everything works as expected

can update game file scaner? thank

and add what feature? :rolleyes:

@everyone

here's a benchmark

some file from Pro Evolution Soccer

old xtool = 275MB >> 381MB took 11.63 seconds using 4 threads
new xtool = 275MB >> 381MB took 06.09 seconds using 4 threads

same memory usage and I/O

note: I only reworked the encoder to use the cpu effectively so you should expect roughly x1.25-2 more speed from old version while using same resources and this applies to all codecs reflate, oodle and etc

Hi Razor12911
This option works for oodle:D, crilayla:D, lzo:D.
Please do not delete this option in the new version.

Just fix the hang bug :D
This is great. :cool:

oodle detect, for example:D

If game uses oodle,then you can find its DLLs in game folder somewhere.
Imho : oodle stream scanner is obsolete ?!

The option will be there

I remember including an oodle scanner, but fine I'll think of something for the scanner.

how about this?

Note:
In a practical situation, you can shave off a few seconds but then again, depends on the input

This is also good :D

Did you have to make changes to the libraries too? Or just the source of xtool executable? Just corious :)

Only the source of xtool, all I did was optimize optimize and optimize, the CPU is forever doing something, I tried to minimize the gaps between reads and writes by making xtool process stuff while it is reading and writing so yeah, the threads hardly get rests now.

Project status = Welp...

I hope this isn't strike 2 in terms losing my sources, my PC died about 2 days ago, motherboard and GPU got fried this time XD. As for my sources well they were stored on my main PC (the dead one) and I don't know if they are safe since I don't have a spare PC to boot up the drive to grab important files and store them to my backup drive, so I can't really tell if the only components that died were only the board and GPU and not my HDD, but I guess that's a story for some other time. If you are wondering what this means in terms of xtool 2019...

Well since I normally code from desktop because I don't need to go far to do things and etc, it was different for xtool somehow... because I was coding from a backup drive only for that project like I knew some shit like this would happen so yes the source is safe (lucky af), if you are wondering about the development now that my main PC is busted.

Well I just had to dust an old laptop and continued development there, issue is it's incredibly slow but it will have to do. If you are wondering why I only provided x86 binaries, well. OS is 32-bit :D, somehow can't compile for x64.

Anyways enough about that, this was suppose to be the last test before the first release, but before I uploaded, I noticed some shitty error in made in the reflate code and I have to rewrite some parts of it so after fixing that, I'll post the final test and if all is good, I'll be making a release.

there are 3 codecs in xtool right now (zlib, reflate, rzlib). The bug was in reflate in multi threaded mode when decoding, but if you want to try it out anyways, open the test bat file. replace zlib with reflate or rzlib but use 1 thread to decode.

zlib = xtool uses zlib1.dll/zlibwapi.dll to process data, fails on some streams
reflate = xtool uses reflate dlls to process data, processes all streams but slow
rzlib = xtool blends zlib and reflate dlls to process data, processes all streams but with better size and speed.

PS: Even the archive is zip, don't have 7-Zip installed for my normal 7z attachments :o
PPS: Nothing is stopping the development of this project XD... I hope cause you never know

Shit, mate! You're lucky to have not loosing sources this time! Gotta use some repo soft next time for more security, no? :)

Anyways, it's nice to see the project is coming to release.

@Razor12911
Download Windows 10 64 bit Archived file and install

https://youtu.be/zGQ9xXYgVJM
Since i also lost my data

Well we are all depends our pc and laptops

First release

Notes

- No idea whether x64 works or not, I just somehow managed to compile. That's up to you to find out.
- Reflate currently is set to use level 6, if you are looking for better output, best to use the old xtool. The reason it's like this is because I'm still working on a way to improve the horrible slow code I used before.
- Skip verification option is currently unavailable.

What I can say, run a lot of tests on this release. I've written some dangerous memory saving code and it may cause errors but if all is well then I can move on to add other codecs such as oodle, lzo, lz4 and the rest. BTW, crilayla will NOT be added, that piece of crap is incredibly slow :D

Image003.png
 

xtool_1907_R1 64Bit

xtool_1907_R1 Test
 

Does not work with ba2/csg files.
32bit= ERROR: write error (disk full?) in compression algorithm zlib/reflate/rzlib
64bit= ERROR: write error (disk full?) in compression algorithm zlib/reflate/rzlib or stuck cpu
http://s8.picofile.com/file/8368123326/zlib.jpg

-------------------------------------------------------------------

Works with this setting.
ZLIB
REFLATE
RZLIB