XTool 2020 (Main Project)

[CR2032]

Originally Posted by Masquerade

No, it's just a final compresison option. A bit like how precomp has built in LZMA compressor

@Razor thanks for the continued updates!

Hello.
Exactly like that! Like a Precomp compression with the key -c.

[Razor12911]

Update available

Changes

- added feature to inject libraries to main executable

Notes

You may notice that the libraries folder is getting filled with a lot of dll files which xtool uses so reduce this cumbersomeness you might want to embed all of these dlls within the main executable and placing the dlls near xtool.exe is no longer needed as they will become part of the executable.

This feature is added to promote portable mode where all you have is the files you want to process and xtool.exe with no libraries nearby.

Usage

xtool.exe inject dll_file

More notes

Only inject lz4, zstd and oodle when you are sure that your input will never need library swaps as these libraries depending on version determine precompression ratio. zlib, reflate and some other libraries do not as every version produces the same results.

[Cesar82]

@Razor12911, thank you very much for the constant updates.

1) Is it possible to update the previously injected libraries?
2) XTool 0.3.21 has been removed from the main post... Will we have updates for the remaining plugins soon?
3) Is it necessary to include the "fast-lzma2.dll" library to use xtool's lzma2 internal method?
4) What xtool method is the library "xdelta3_dll.dll" from the _libraries folder used for?

[CR2032]

Originally Posted by Razor12911

Update available

Changes

- added feature to inject libraries to main executable

Notes

You may notice that the libraries folder is getting filled with a lot of dll files which xtool uses so reduce this cumbersomeness you might want to embed all of these dlls within the main executable and placing the dlls near xtool.exe is no longer needed as they will become part of the executable.

This feature is added to promote portable mode where all you have is the files you want to process and xtool.exe with no libraries nearby.

Usage

xtool.exe inject dll_file

More notes

Only inject lz4, zstd and oodle when you are sure that your input will never need library swaps as these libraries depending on version determine precompression ratio. zlib, reflate and some other libraries do not as every version produces the same results.

Thank You So Much, Razor12911!
I think this is the really great solution.

[CR2032]

Originally Posted by Cesar82

@Razor12911, thank you very much for the constant updates.

1) Is it possible to update the previously injected libraries?
2) XTool 0.3.21 has been removed from the main post... Will we have updates for the remaining plugins soon?
3) Is it necessary to include the "fast-lzma2.dll" library to use xtool's lzma2 internal method?
4) What xtool method is the library "xdelta3_dll.dll" from the _libraries folder used for?

Hello.
1. Perhaps, just replace old to new.
For example:
If I have xtool.exe with integrated libraries:
xtool.exe inject _libraries\fast-lzma2.dll
xtool.exe inject _libraries\preflate_dll.dll
xtool.exe inject _libraries\zlibwapi.dll

And I want to replace only one a new version fast-lzma2.dll, then:
xtool.exe inject _libraries\fast-lzma2.dll

Or probably, every time there is a new version of xtool, you need to create a new integration.

3. Yes.

[elit]

I have found something that seem like a bug, and I am not sure if its FA or xtool, but likely an xtool issue.

Say you precomp a file with reflate, but you have an xtool.ini config that contain all kind of codecs, like xmemcompress, quickbms based and so on. But you only used reflate.

Now to decode successfully, you have to have all those exe files + untouched xtool.ini otherwise xtool will throw general error! Even if you don't need any of those for decompression.
That just happened to me. I had to copy all those pointless exe's and exact same xtool.ini for it to work. If for example I edited out those unneeded codecs from xtool.ini and/or deleted crilayla.exe or xmemcompress.exe, then xtool won't decode my reflate pack!

EDIT: No wonder it fails:
xtool.png
Why it have to save all those things if it's not needed?

[Razor12911]

Originally Posted by elit

I have found something that seem like a bug, and I am not sure if its FA or xtool, but likely an xtool issue.

Say you precomp a file with reflate, but you have an xtool.ini config that contain all kind of codecs, like xmemcompress, quickbms based and so on. But you only used reflate.

Now to decode successfully, you have to have all those exe files + untouched xtool.ini otherwise xtool will throw general error! Even if you don't need any of those for decompression.
That just happened to me. I had to copy all those pointless exe's and exact same xtool.ini for it to work. If for example I edited out those unneeded codecs from xtool.ini and/or deleted crilayla.exe or xmemcompress.exe, then xtool won't decode my reflate pack!

EDIT: No wonder it fails:
Attachment 32214
Why it have to save all those things if it's not needed?

Xtool stores all possible methods because xtool itself doesn't know if are or will be used if all before even precompression begins.

[Razor12911]

Originally Posted by Cesar82

@Razor12911, thank you very much for the constant updates.

1) Is it possible to update the previously injected libraries?
2) XTool 0.3.21 has been removed from the main post... Will we have updates for the remaining plugins soon?
3) Is it necessary to include the "fast-lzma2.dll" library to use xtool's lzma2 internal method?
4) What xtool method is the library "xdelta3_dll.dll" from the _libraries folder used for?

1) yes
2) possibly, in the main post there is actually link to the older releases I just removed 0.3.21 to make people not ask what is the different between this version and the recent update.
3) yes
4) imperfect streams use xdelta for patching, this function initially came from a dll but I separated all dlls from the main executable and gave the options to the user to include such a feature.

[elit]

Greetings Razor.
Back on page 34 where I wrote about reflate problems, causing crc errors... This time I was able to catch in on console:

reflatebug.png

But the thing is, exact same run may pass successfully on second or more tries(and then maybe fail again, so random dice per run). I am starting to be cautious about HW issue possibility on my side, but then I had no such issues with zlib yet. Using -t8.

Looks like either thread race issue, or my HW memory. This is on xtool v0.5.3. I will investigate this further.

EDIT:
Latest v0.6.2 == same issue.
If xtool do not fail during compression(precomp), then it will always recompress successfully during unpacking - i.e. probably not HW issue after all.
Last stable version without this bug that I have is 0.3.21. Cannot replicate on it. I believe this one is stable.[Its not, later I found..] Unfortunately it miss a lot of streams that newer xtool-reflate can see.
EDIT2:
Forgot to add. Changing chunk size(or not using depth) can help sometimes, but on big enough data stream one chunk size may pass at one point and fail at different one(where again some other size could work and so on). Also, I started having these issues only recently which correspond with me upgrading to later xtool version than I was stuck with.
Also, I think this have way more likelihood of happening on files that do not contain(or very little) deflate chunks. When FA reported CRC fails it was pretty much files that happen to share same extension with those that do.
EDIT3:
Yup, not my HW. I just threw whole Halo Master Collection directory to xtool 0.3.21(without filter) and it completed successfully. Then I tried both v0.5.3 and v0.6.2 and it failed on address violation.
EDIT4:
I will further confirm it later here, but using -lm *may* work as a workaround for reflate. Not due to a memory shortage, but the way chunks/data and threads are handled. No, it didn't help. [ADD: neither did -t1]
EDIT5:
Preflate works without issues, so I had to settle with it. Final repack size is ~1gb bigger though. I do hope reflate get fixed in the future as I prefer it.

[Razor12911]

samples?

[elit]

Originally Posted by Razor12911

samples?

Unfortunately this wouldn't be possible, it need to be tested on whole Halo Master Collection directory. Because you are unlikely to trigger on small data. Even different arrangement of files(sorting) or chunk size can affect where or if the issue happen at all. Closest I was able to get for smaller data sample were 3 ipak files which is still ~1gb and less reliable to replicate. This could be a rare byte sequence that cause this. And if so I don't know if it's within reflate dll or xtool.
Best way to test is to run it on whole Halo MCC directory without any filter(all files), even better with FA gui instead cmd. That is the closest I know.

[L33THAK0R]

Is there a list of every valid codec/method that XTool supports? I'm talking about the ones outside of the codecs listed in the included documentation. It seems like a silly question but I havent kept that close an eye on XTool's development and with the dropping of GrittiBanzli support just thought I'd see if anyone has an up-to date list.

[Snake288]

Hello Razor12911
Dying Light 2 Stay Human compression XTool 2020 (Database you can provide support ?
Dying Light 2 Stay Human example Data
https://lifeboxtransfer.com/s/5d1043...d-ee4116099aea

[KaktoR]

Originally Posted by L33THAK0R

Is there a list of every valid codec/method that XTool supports? I'm talking about the ones outside of the codecs listed in the included documentation. It seems like a silly question but I havent kept that close an eye on XTool's development and with the dropping of GrittiBanzli support just thought I'd see if anyone has an up-to date list.

-mpng (have to be used with -d1 parameter and reflate/preflate afterwards)
-mflac
-mbrunsli
-mjojpeg
-mpackjpg

[KaktoR]

Originally Posted by Snake288

Hello Razor12911
Dying Light 2 Stay Human compression XTool 2020 (Database you can provide support ?
Dying Light 2 Stay Human example Data
https://lifeboxtransfer.com/s/5d1043...d-ee4116099aea

Did they change compression? I'm sure in some older version of this game they used just zlib or sort of if I remember correctly.