XTool 2020 (Main Project)

[dixen]

[re4lfs]
Encode=re4lfs.exe <filein>.lfs <fileout>.pack
Decode=re4lfs.exe <filein>.pack <fileout>.lfs

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat

[Masquerade]

Originally Posted by dixen

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat

Afaik no, because DELZORec requires non solid + 2 input files whereas FA can only deliver 1 input file.

[Razor12911]

Originally Posted by dixen

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat

no, it's not possible because xtool can't be made to give two inputs nor accept two outputs

[Razor12911]

Update available

Changes

- generate database feature fixed
- fixed external executable support issues
- fixed lz4f level setting bug

[Razor12911]

Update available

Changes

- updated oodle scanner
- updated external executable support
- updated configuration based plugin support to add depth information
- updated verbose mode

Notes

Given that very small number of people are able to make their own plugins and codecs to use in xtool, I've added an example that imports codecs from QuickBMS after you have generated your database and have no codec to use with it.

For compression:

[cpk,snappy,rfpk]
Encode=quickbms.exe -s "comtype <codec> ; clog <fileout> 0 <insize> <outsize>" "" <filein>
Decode=quickbms.exe -s "comtype <codec>_COMPRESS ; clog <fileout> 0 <insize> <outsize>" "" <filein>

For encryption:

[blowfish,xxtea]
Encode=quickbms.exe -s "open FDDE <fileres> 1 ; getdstring X_KEY <ressize> 1; encryption <codec> X_KEY "" 0 <ressize>; log <fileout> 0 <insize>" "" <filein>
Decode=quickbms.exe -s "open FDDE <fileres> 1 ; getdstring X_KEY <ressize> 1; encryption <codec> X_KEY "" 1 <ressize>; log <fileout> 0 <insize>" "" <filein>

You'll need to check http://aluigi.altervista.org/papers/quickbms.txt for the list of supported compressors built in so that you don't pick something like lz2k and expect it to work (quickbms itself has to be able to both compress and decompress). You can add more codecs like I have done here "cpk,snappy,rfpk", if you wanted to add lzss for example, it then becomes "cpk,snappy,rfpk,lzss" then from there you can use Bms2Xtl for scripts that have this comtype and possibly expect xtool to use quickbms with no real issues (hopefully).

Also note that if for whatever reason there is crc mismatch, xtool will still help you by applying xdelta automatically.

Also also note that this generate temps files meaning the process is slower because it has to both read and write to disk therefore, only use this approach if there is nothing else that you can to do to improve the situation.

[elit]

OMG ok with this I think you opened can for a whole lot of more codecs/games to be able to process.

[Razor12911]

Update available

Changes

- fixed issue with storing correct recompression information when stream patching is performed

Notes

This issue was brought up by Masquerade and L0v3craft when they were trying to precompress WRC 10 and upon decoding xtool would produce an error so thanks to them, it has been resolved however I'd like to add more useful information regarding this game and why even with xtool helping you with deltas will not allow you to process all streams based on how xtool was designed.

We all know that xtool patches streams if they cannot be restored correctly however it sets its own boundaries, by default the patch file cannot be 5% larger than the original size else the patch fails as highlighted here with some streams being missed. https://fileforums.com/showpost.php?...&postcount=248

This 5% can be changed by user via the command --diff=5p, with 5p being 5%, so to capture all these other streams, you'll have to increase this percentage.

CHUNK_57.PKG
using -mwrc10

Compressed 1 file, 99,435,664 => 189,185,421 bytes. Ratio 190.26%
Compression time: cpu 0.14 sec/real 3.71 sec = 4%. Speed 26.82 mB/s

using -mwrc10 --diff=20p

Compressed 1 file, 99,435,664 => 263,029,277 bytes. Ratio 264.52%
Compression time: cpu 0.09 sec/real 3.61 sec = 3%. Speed 27.53 mB/s

Verbose information displays

[0] Processing lz4f stream at 0000000000005476 (514 >> 893 >> 514) using l9:b4:d0 has failed
[0] - Patching stream at 0000000000005476 (514 >> 514) [26] has failed

Patching this 514 byte stream produced a 26 byte patch file, which is 5.1% and that is larger than the 5% limit so increasing this means you are allowing xtool to accept this stream.

--diff=20p

[0] Processing lz4f stream at 0000000000005476 (514 >> 893 >> 514) using l9:b4:d0 has failed
[0] - Patched stream at 0000000000005476 (514 >> 514) [26] successfully

[Masquerade]

LEGO® Star Wars™: The Skywalker Saga uses Kraken

XTool is created by Razor12911

Streams: 284071/284071
Time: 00:13:43 (00:48:35)
Memory: 512 MB (512 MB)

100.0%
Errorlevel=0

Compressed 1 file, 4,061,948,534 => 9,598,998,234 bytes. Ratio 236.32%
Compression time: cpu 4.22 sec/real 1032.25 sec = 0%. Speed 3.94 mB/s
All OK

[GamerfromPoland]

OMG it's a great tool! Thanks for it!
https://radaryonline.pl/gdzie-jest-burza/

[Razor12911]

Update available

Changes

- added IO functions (erase, replace)
- fixed external executable support bugs

Notes

Xtool has introduced IO functions, which should help you perform file and folder operations such as erase and replace (for now, more will be added). I actually wanted to add these functions a long time ago as they could be useful for repacking however I delayed them time after time because precompression needed more attention but as there have been no bug reports for the past 3 weeks I decided to start working on them.

To summarise, Erase is a feature that fills a given input with zeroes in case you wanted to repack certain data separately and Replace is a feature that replaces existing data within certain files with another.
Xtool will search for locations of the extracted streams and store these positions for when you use decode function to revert the changes (yes, the process is reversible)

Erase

xtool erase extracted_streams original_data [decode_data]
xtool decode decode_data extracted_streams original_data

Use cases for erase is the removal of language/videos files from archives in games after using another extraction tool, so rather than manually searching for positions and storing them, erase will make this process automated.

An example is after extracting video files from an archive and then wanting to remove credits video, the syntax would be

xtool erase credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak credits.xtl
xtool decode credits.xtl credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak

decode will fill the zeroes with the original data

Replace

xtool replace old_streams new_streams original_data [decode_data]
xtool decode decode_data extracted_streams original_data

The use cases for replace is possibly game file modification.

An example is having several modified files and the original files and you wanted to bulk replace these files, you'd have to prepare two folders with the same file structure and the syntax would be

xtool replace "textures\original\*" "textures\modified\*" "game\*" mod_tex.xtl
xtool decode mod_tex.xtl "textures\original\*" "game\*"

decode will fill the modified data sectors with the original to restore the file its original form.

PS

If you are using these features for a repack in which people would select languages or video credits for example, if the users did not select any of these to be installed then there would be missing files. Xtool's decode command would still work and will try to restore the original data using whatever data that was selected and available without problems.

[Razor12911]

Update available

Changes

- added IO functions (find, extract, patch)
- generate database feature and IO functions now can search for streams larger than chunk size

Notes

More IO functions introduced. Find simply helps you track positions of extracted files from a given input, while extract uses a generated decode file data input to extract streams in case if you wanted another person to extract their very own streams using your own findings and patch, well patch compares two inputs and generates a diff file which can allow you to patch an input to make it similar to the new data.

Find

xtool find [parameters] extracted_streams original_data [decode_data]

Find can be used to search for streams to then produce decode data which you can upload here on the forum in cases where you wanted to inform people what files should be extracted if they wanted for example to separate audio files per language from a game.

An example where I wanted to make a guide of how to separate english, german, italian, russian and spanish languages from the game would be something like:

xtool find "extracted\en\*" "game\*" en.xtl
xtool find "extracted\de\*" "game\*" de.xtl
xtool find "extracted\it\*" "game\*" it.xtl
xtool find "extracted\es\*" "game\*" es.xtl
xtool find "extracted\ru\*" "game\*" ru.xtl

These files are then uploaded for people if they wanted to extract the same data by themselves by using xtool via the "extract" function or for later use by yourself.

Extract

xtool extract decode_data original_data extracted_streams

Extract works mostly hand in hand with with the find function as its job is to use the decode_data to extract the data that was used for its generation.

A use case for it would be similar to the find example but from another user's perspective, so if someone uploaded decode data to use to extract my very own streams from their investigations, rather than them giving me position ranges for languages or the tools to use, we can just use their uploaded decode data to do it ourselves

xtool extract en.xtl "game\*" "extracted\en\*"
xtool extract de.xtl "game\*" "extracted\de\*"
xtool extract it.xtl "game\*" "extracted\it\*"
xtool extract es.xtl "game\*" "extracted\es\*"
xtool extract ru.xtl "game\*" "extracted\ru\*"

Then from here, we use these extracted files, to even use the erase function introduced in the version prior to blank out the sectors to prepare our own repack

xtool erase "extracted\*" "game\*" languages.xtl

which we can then restore after installation using

xtool decode languages.xtl "extracted\*" "game\*"

Patch

xtool patch [parameters] old_data new_data patch_data
xtool decode patch_data old_data

Patch's use case can be the generation of update patches in cases where you already made a repack but for whatever reason refuse to re-repack the game so instead choose to generate a patch file which can be used to update the old repack with newer files

xtool patch "Sims4\*" "Sims4_updated\*" sims4_wedding_stories.patch

Then after the installation of the original repack, you can add an additional archive that has sims4_wedding_stories.patch which will be used to update the game

xtool decode sims4_wedding_stories.patch "Sims4\*"

I noticed after compiling xtool.exe that xdelta is set to compress these patch files which means that they cannot be precompressed nor compressed with something better, however this compression in the next version will be removed.

[Razor12911]

Update available

Changes

- added IO functions (archive, execute)
- fixed issue in patch io function
- removed compression on patch diff files

Notes

More IO functions added. Archive behaves like -m0 but allows you to the archive to stdout and read from stdin when decoding (if you ever need that). Then there is execute which allows you execute several instances of another executable in parallel mode while all their inputs are fed from one source and all their output is fed to one destination.

Archive

xtool archive files1 files2... archive

There is nothing to add here as I have personal uses for this but the example would be

xtool archive game\* mygame.xtl
xtool decode mygame.xtl extracted\*

Execute

xtool execute [parameters] input output [exec_syntax]

Too lazy to write description but, here's an example

xtool.exe execute -c64mb -t8 UI.sb UI.bin bcm.exe -b64 [filein] [fileout]
xtool.exe decode -t100p UI.bin UI_dec.sb bcm.exe -d [filein] [fileout]

The left side of the syntax is to command xtool and after specifying input and output files, the right side begins and here you can write the command line that should be used to perform execution.

[filein], [fileout], [stdin], [stdout] can be used and denote what IO the program being executed uses.
[] is used to avoid conflicting with Freearc's <>

Freearc example would look like this

[External compressor:xbcm]
header    = 0
packcmd   = xtool.exe execute { -option} - - <stdin> <stdout> bcm.exe -b64 [filein] [fileout]
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout> bcm.exe -d [filein] [fileout]

and the method would be -mxbcm:c64mb:t75p

[Razor12911]

Update available

Changes

- added png stream preprocessor
- removed grittibanzli codec (since nobody uses it)

Notes

Xtool is able to process deflate/zlib streams and png images do contain these streams however, they are split up into several blocks which at times does prevent the program from being able to process them. The png stream preprocessor's job is to concatenate (rejoin all these blocks into a single stream) which can then be processed by xtool, so if you know your input contains these images then it's best to include -mpng into the method chain and use -d1 to first preprocess the streams then process them using zlib, reflate or preflate (preflate is the preferred method to use).

Results

without png preprocessor:

Compressed 1 file, 7,232,549 => 8,291,632 bytes. Ratio 114.64% -mreflate
Compressed 1 file, 7,232,549 => 7,232,655 bytes. Ratio 100.00% -mpreflate

with png preprocessor:

Compressed 1 file, 7,232,549 => 26,075,119 bytes. Ratio 360.52% -mpng+reflate -d1
Compressed 1 file, 7,232,549 => 25,289,529 bytes. Ratio 349.66% -mpng+preflate -d1

[Cesar82]

@Razor12911, can i use the parameters in that order in freearc commandline?

xtool:mpreflate:mpng:d1

[Razor12911]

yes but you'll have to change arc.ini from { -moption} to { -option}