FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   XTool 2020 (Database Maker) (https://fileforums.com/showthread.php?t=104109)

Razor12911 01-02-2021 02:23

XTool 2020 (Database Maker)
 
Database Maker/Generator is part of xtool and this post is dedicated to explaining how it works.


If xtool does not have a plugin designed to work for a certain input or game but if for example you managed to extract (not unpack) streams a certain file has then you order xtool to create a plugin in a form of a database file which can then be used within the program.

The syntax is as follows:
Code:

xtool generate [parameters] extracted_streams original_data database_output
Parameters:
  -m#  - codec to use for precompression
  -c#  - scanning range of precompressor [16mb]
  -t#  - number of working threads [50p]

the inputs of both extracted_streams and original_data can either be a file or a folder (wildcards/masks) are accepted.

Example:
Code:

xtool.exe generate -mxcompress "VehicleHRLiveries.bff_extract\*" "VehicleHRLiveries.bff" pcars.xtl

Razor12911 01-02-2021 02:24

QuickBMS to Xtool
 
1 Attachment(s)
bms2xtool is a separate project that helps generate database files from quickbms scripts.

Since this forum is either incredibly lazy or they can't code as there are no plugins for xtool submitted on the forum I thought of creating such a tool, QuickBMS has a lot of scripts made for a lot of games and has a community dedicated to making more of them which xtool doesn't. :o

The syntax is as follows:
Code:

bms2xtl script game_files database
the inputs of game_files can either be a file or a folder (wildcards/masks) are accepted.

Required files:
Code:

quickbms.exe or quickbms_4gb_files.exe
Get them from here

Source:
https://github.com/Razor12911/bms2xtool

Razor12911 01-02-2021 02:25

Log Converter
 
I'm spamming forum

Python-Boy 01-02-2021 07:18

Quote:

Originally Posted by Razor12911 (Post 490191)
I'm spamming forum

Then don't do that coz this is month of february not April.:D

KaktoR 01-02-2021 12:14

Quote:

Originally Posted by Razor12911 (Post 490191)
I'm spamming forum

Please, spam a bit more of your usefull stuff :D

Razor12911 14-06-2021 00:20

1 Attachment(s)
Dunia/Disrupt Engine Database maker

Instructions

Drag and drop game folder on make_db.bat, a database file (*.xtl) will be created

Supported games

Far Cry 3
Far Cry 4
Far Cry 5
Far Cry 6
Far Cry New Dawn
The Crew 2
Watch Dogs 1 (use bms2xtl)
Watch Dogs 2
Watch Dogs Legion

L33THAK0R 14-06-2021 02:28

Quote:

Originally Posted by Razor12911 (Post 492876)
Dunia/Disrupt Engine Database maker

Instructions

Drag and drop game folder on make_db.bat, a database file (*.xtl) will be created

Supported games

Far Cry 3
Far Cry 4
Far Cry 5 (needs testing)
Far Cry 6 (I'm waiting for you)
Far Cry New Dawn (needs testing)
The Crew 2
Watch Dogs 1 (wait for xtool v0.4+ update)
Watch Dogs 2 (wait for R2)
Watch Dogs Legion (wait for R2)

Is "Far Cry 2" able to work with this? As far as I know it is one of two titles to use Dunia 1 (the other being the Ubisoft licensed "Avatar" title).

Razor12911 14-06-2021 02:36

upload common.dat, common.fat and all other *.fat files as samples.

:( Sad8669 14-06-2021 03:05

Razor, do you need WDL samples?

I asked because i have the game files right now.

L33THAK0R 14-06-2021 04:17

Quote:

Originally Posted by Razor12911 (Post 492879)
upload common.dat, common.fat and all other *.fat files as samples.

Sample upload is structured as it appears in the title's directory, as some files have identical names. The second title mentioned "James Cameron's Avatar: The Game" does not have a sample included, as it does not seem to have any ".fat", or ".dat" files present, despite it using the same iteration of the Dunia engine as "Far Cry 2". It instead seems to house all of its required assets within ".pak" files.

KaktoR 14-06-2021 11:12

4 Attachment(s)
Here are some for testing

PS: db maker named them all as fc3.xtl (maybe it's a bug), so I have renamed them.

ffmla 15-06-2021 01:43

Quote:

Originally Posted by KaktoR (Post 492890)
Here are some for testing

PS: db maker named them all as fc3.xtl (maybe it's a bug), so I have renamed them.

some doubt arises.

did you change the fc3.xtl name in bat file?
Code:

for /R "%~1\" %%a IN (*.fat) do dunia2_db "%%a" fc3.xtl

KaktoR 15-06-2021 10:49

Quote:

Originally Posted by ffmla (Post 492897)
some doubt arises.

did you change the fc3.xtl name in bat file?
Code:

for /R "%~1\" %%a IN (*.fat) do dunia2_db "%%a" fc3.xtl

Good point. Answer is no :D

Razor12911 26-06-2021 00:19

1 Attachment(s)
Unreal Engine ucas Database maker

Instructions

Open batch file
Insert the name of game, enter the decryption key and then save
Drag and drop game folder on make_db.bat, a database file (*.xtl) along with key file (*.key) will be created

Use these with xtool.

Razor12911 26-06-2021 09:12

Unreal Engine ucas Database maker updated

Changes

- Fixed decryption issues

Notes

That's all I can do for now, fixed some issues during my spare time. I think there's still a few left.

Razor12911 06-10-2021 15:46

Dunia/Disrupt Engine Database maker updated

Changes

- Added support for Far Cry 6

Notes

No idea if this works on the whole game, I tested on 1 sample.

Far Cry 6\data_final\pc\common.dat
Code:

Tested 2 files, 1,287,326,620 => 684,093,252 bytes. Ratio 188.18%
Testing time: cpu 0.80 sec/real 11.15 sec = 7%. Speed 61.36 mB/s


KaktoR 08-10-2021 07:49

For FC6 use lz4 v1.8.2

fctber_disk.dat
Code:

Tested 2 files, 3,646,603,232 => 2,313,847,968 bytes. Ratio 157.60%
Testing time: cpu 0.86 sec/real 17.36 sec = 5%. Speed 133.32 mB/s


Razor12911 05-01-2022 16:38

Dunia/Disrupt Engine Database maker updated

Changes

- Added support for all* fat files in Watch Dogs Legion

Notes

I have no idea if all fat files are supported (I don't have game, only source is torrent files which my ISP does not appreciate so...)
This is perhaps why I stopped pirating games and started buying them

You'll need to wait for next xtool update for this to work, something is up with the main program causing it to crash.

KaktoR 07-01-2022 07:35

1 Attachment(s)
Watch Dogs Legion

Code:

common.dat: 32781 stream(s)
commonengine.dat: 22 stream(s)
patch.dat: 113756 stream(s)
patch_brazilian.dat: 7255 stream(s)
patch_english.dat: 82028 stream(s)
patch_french.dat: 9519 stream(s)
patch_german.dat: 8264 stream(s)
patch_italian.dat: 7126 stream(s)
patch_japanese.dat: 3600 stream(s)
patch_russian.dat: 6981 stream(s)
patch_spanish.dat: 4819 stream(s)
shadersobj.dat: 168500 stream(s)
london.dat: 103303 stream(s)
london_brazilian.dat: 84705 stream(s)
london_cache.dat: 27030 stream(s)
london_english.dat: 84690 stream(s)
london_french.dat: 83419 stream(s)
london_german.dat: 84721 stream(s)
london_hires.dat: 41591 stream(s)
london_italian.dat: 84720 stream(s)
london_japanese.dat: 84718 stream(s)
london_preload.dat: 43551 stream(s)
london_russian.dat: 85156 stream(s)
london_spanish.dat: 84667 stream(s)
london_ultra.dat: 12124 stream(s)
Database created successfully


Razor12911 17-01-2022 11:46

Information on main post added

Changes

- Added information showcasing database generation feature

Razor12911 29-01-2022 14:04

1 Attachment(s)
Information on QuickBMS to Xtool added

Changes

- Added information and attachment showcasing bms2xtool (bms script to xtl database file)

dixen 29-01-2022 23:55

I try it on LEGO I.J but


Quote:

d:\RePack_tools\bms2xtool_0.1>bms2xtl ttgames.bms "GAME.DAT" lego.xtl
Exception: File operation not supported
UPD.
LEGO Batman The Videogame - same error((

Masquerade 30-01-2022 01:11

Quote:

Originally Posted by dixen (Post 495600)
I try it on LEGO I.J but



UPD.
LEGO Batman The Videogame - same error((

Recompression of LZ2K is not supported. QBMS can only extract LZ2K, not compress.

dixen 30-01-2022 02:01

Quote:

Originally Posted by Masquerade (Post 495601)
Recompression of LZ2K is not supported. QBMS can only extract LZ2K, not compress.

Damn..I forgot about that

Razor12911 16-02-2022 22:40

1 Attachment(s)
QuickBMS to XTool updated

Changes

Fixed issues with some scripts not being parsed correctly (thanks Masquerade)
Added support for xtool v0.4+
Made overall progress to be visible from taskbar

Notes
You can find source code from here

Razor12911 21-02-2022 15:01

QuickBMS to XTool updated

Changes

- Added user input option (auto write to cmd window any common prompts)
- Fixed issue with string to binary conversion of keys
- Fixed issue with handling streams that are both encrypted and compressed
- Fixed issue with storing incorrect decryption keys

Notes

It can be sometimes difficult to properly detect streams when you use xtool directly, especially oodle streams if decompressed streams is unknown however if a quick bms script was made, you can take advantage of this and quickly generate a database to use in xtool. Same goes for unreal engine games where ue4d plugin seems to not work as intended, you can loaded up the unreal_tournament_4 script, enter the key and everything should be taken care of.

Some results on Project Cars 3

bdt+oo2reck:
Code:

Compressed 1 file, 538,214,104 => 1,028,286,891 bytes. Ratio 191.06%
Compression time: cpu 0.77 sec/real 160.61 sec = 0%. Speed 3.35 mB/s

bdt+xtool:kraken:
Code:

Compressed 1 file, 538,214,104 => 1,035,969,556 bytes. Ratio 192.48%
Compression time: cpu 0.77 sec/real 141.78 sec = 1%. Speed 3.80 mB/s

xtool:pcars3 using (nfsshift.bms script):
Code:

Compressed 1 file, 538,214,104 => 1,058,023,182 bytes. Ratio 196.58%
Compression time: cpu 0.69 sec/real 25.66 sec = 3%. Speed 20.98 mB/s

Not only you get better results but the process is also faster. :)

dixen 22-02-2022 01:44

1 Attachment(s)
Test database for Mortal Kombat 11

XTool+mk.xtl

Pack
Quote:

Compressed 2 files, 1,694,239,720 => 3,013,304,346 bytes. Ratio 177.86%
Compression time: cpu 1.53 sec/real 137.23 sec = 1%. Speed 12.35 mB/s
All OK
Unpack
Quote:

Tested 2 files, 3,013,304,346 => 1,694,239,720 bytes. Ratio 177.86%
Testing time: cpu 1.78 sec/real 120.46 sec = 1%. Speed 14.06 mB/s
All OK
*Requied oo2core_5_win64.dll from game folder

Xtoo(0.42)+mk11.xtl+srep+lzma2

Quote:

Compressed 3,786 files, 81,540,296,959 => 59,769,328,360 bytes. Ratio 73.30%
Compression time: cpu 838.47 sec/real 23004.64 sec = 4%. Speed 3.54 mB/s
All OK
**Requied ~300 gb for decompressiion and srep proccess

Razor12911 22-02-2022 08:03

QuickBMS to XTool updated

Changes

- Added safe mode feature

Notes

This issue is not related to bms2xtl but it's related to QuickBMS itself and was detected by KaktoR where quickbms will run out of memory when trying to process some streams especially ones where oodle is used, since quickbms internally uses old oodle library taken from Warframe, newer games seem to cause the old library to possibly have a memory leak.

Safe mode mitigates this issue (only use it if you have an issue when QuickBMS produces an error related to memory or something similar and it should be kept disabled under normal use)

Luminior 23-02-2022 03:13

Sword and Fairy 7
Method: bms2xtool v0.3 + xtool v0.4.2 + srep + lolz

Quote:

Compressed 16 files, 69,496,034,241 => 34,281,984,937 bytes. Ratio 49.33%
Compression time: cpu 1290.97 sec/real 41904.95 sec = 3%. Speed 1.66 mB/s
All OK
Notes: other libraries were not used.

Razor12911 20-03-2022 17:33

QuickBMS to XTool updated

Changes

- Fixed issue with file inputs smaller than 4MB

Razor12911 24-03-2022 12:05

Unreal Engine ucas Database maker updated

Changes

- Added support for new structure seen used in Ghostwire Tokyo

Notes

This still uses the old xtool 0.3.21 database structure, wait for another update if you plan on using it with 0.4+

kuyhaa 24-03-2022 19:03

Quote:

Originally Posted by Razor12911 (Post 496279)
Unreal Engine ucas Database maker updated

Changes

- Added support for new structure seen used in Ghostwire Tokyo

Notes

This still uses the old xtool 0.3.21 database structure, wait for another update if you plan on using it with 0.4+

Compressing 2 files, 850,906,556 bytes
Compressing Snowfall-WindowsNoEditor_0_P.pak
Compressing Snowfall-WindowsNoEditor_0_P.ucas
Compressed 2 files, 850,906,556 => 850,907,110 bytes. Ratio 100.00%
Compression time: cpu 6.33 sec/real 8.72 sec = 73%. Speed 97.60 mB/s
All OK

-mgwtokyo+zlib -d1

unreal gwtokyo.xtl and gwtokyo.key near xtool

have wrong ?

Razor12911 24-03-2022 19:21

kraken was used, not zlib

Masquerade 24-03-2022 23:13

Quote:

Originally Posted by Razor12911 (Post 496282)
kraken was used, not zlib

Code:

Compressing 3 files, 851,793,200 bytes
  Compressing Snowfall-WindowsNoEditor_0_P.pak
  Compressing Snowfall-WindowsNoEditor_0_P.ucas
  Compressing Snowfall-WindowsNoEditor_0_P.utoc
Compressed 3 files, 851,793,200 => 851,805,895 bytes. Ratio 100.00%
Compression time: cpu 0.84 sec/real 145.87 sec = 1%. Speed 5.84 mB/s
All OK

Code:

[External compressor: gwt]
header    = 0
packcmd  = xtool precomp -mgwtokyo+kraken -c256mb -t16 - - <stdin> <stdout>

xtool v3.2.1, oo2core_9_win64.dll

:confused::confused:

Razor12911 24-03-2022 23:14

Quote:

Originally Posted by Masquerade (Post 496284)
Code:

Compressing 3 files, 851,793,200 bytes
  Compressing Snowfall-WindowsNoEditor_0_P.pak
  Compressing Snowfall-WindowsNoEditor_0_P.ucas
  Compressing Snowfall-WindowsNoEditor_0_P.utoc
Compressed 3 files, 851,793,200 => 851,805,895 bytes. Ratio 100.00%
Compression time: cpu 0.84 sec/real 145.87 sec = 1%. Speed 5.84 mB/s
All OK

Code:

[External compressor: gwt]
header    = 0
packcmd  = xtool precomp -mgwtokyo+kraken -c256mb -t16 - - <stdin> <stdout>

xtool v3.2.1, oo2core_9_win64.dll

:confused::confused:

-d1 :(

dixen 24-03-2022 23:35

oo2core_7_win64.dll

*.pak
Quote:

Streams: 401/402/402
Time: 00:00:23 (00:03:24)
Memory: 392 MB (392 MB)

100%
Errorlevel=0
Compressed 1 file, 622,183,516 => 848,346,910 bytes. Ratio 136.35%
Compression time: cpu 0.55 sec/real 25.16 sec = 2%. Speed 24.73 mB/s
All OK
*.ucas
Quote:

Streams: 5550/5564
Time: 00:02:03 (00:08:04)
Memory: 776 MB (776 MB)

100%
Errorlevel=0
Compressed 1 file, 228,723,040 => 910,585,777 bytes. Ratio 398.12% 100%
Compression time: cpu 0.20 sec/real 124.58 sec = 0%. Speed 1.84 mB/s
All OK

kuyhaa 25-03-2022 01:31

Quote:

Originally Posted by dixen (Post 496286)
oo2core_7_win64.dll

*.pak


*.ucas

oo2core_7

Quote:

Compressing 228,723,040 bytes

xtool321.exe precomp -mgwtokyo+kraken -c256mb -d1

XTool is created by Razor12911

Streams: 38/5640
Time: 00:02:53 (00:27:21)
Memory: 128 MB (128 MB)

100%
confiused :(

dixen 25-03-2022 01:37

Quote:

Originally Posted by kuyhaa (Post 496287)
confiused :(

oo2core_7_win64.dll 569.0 Кб

kuyhaa 25-03-2022 01:55

Quote:

Originally Posted by dixen (Post 496288)
oo2core_7_win64.dll 569.0 Кб

solved

Razor12911 28-03-2022 10:41

Unreal Engine ucas Database maker updated

Changes

- Added support for newer xtool 0.4+


All times are GMT -7. The time now is 07:46.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com