FileForums - XTool 2020 (Database Maker)

Page 1 of 2

Show 40 post(s) from this thread on one page

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - XTool 2020 (Database Maker) (https://fileforums.com/showthread.php?t=104109)

Razor12911

01-02-2021 02:23

XTool 2020 (Database Maker)

Database Maker/Generator is part of xtool and this post is dedicated to explaining how it works.

If xtool does not have a plugin designed to work for a certain input or game but if for example you managed to extract (not unpack) streams a certain file has then you order xtool to create a plugin in a form of a database file which can then be used within the program.

The syntax is as follows:

Code:

xtool generate [parameters] extracted_streams original_data database_output

Parameters: 

  -m#  - codec to use for precompression

  -c#  - scanning range of precompressor [16mb]

  -t#  - number of working threads [50p]

the inputs of both extracted_streams and original_data can either be a file or a folder (wildcards/masks) are accepted.

Example:

Code:

xtool.exe generate -mxcompress "VehicleHRLiveries.bff_extract\*" "VehicleHRLiveries.bff" pcars.xtl

Razor12911

01-02-2021 02:24

QuickBMS to Xtool

1 Attachment(s)

bms2xtool is a separate project that helps generate database files from quickbms scripts.

Since this forum is either incredibly lazy or they can't code as there are no plugins for xtool submitted on the forum I thought of creating such a tool, QuickBMS has a lot of scripts made for a lot of games and has a community dedicated to making more of them which xtool doesn't. :o

The syntax is as follows:

Code:

bms2xtl script game_files database

the inputs of game_files can either be a file or a folder (wildcards/masks) are accepted.

Required files:

Code:

quickbms.exe or quickbms_4gb_files.exe

Get them from here

Source:
https://github.com/Razor12911/bms2xtool

Razor12911

01-02-2021 02:25

Log Converter

I'm spamming forum

Python-Boy

01-02-2021 07:18

Quote:

Originally Posted by Razor12911 (Post 490191)

I'm spamming forum

Then don't do that coz this is month of february not April.:D

KaktoR

01-02-2021 12:14

Quote:

Originally Posted by Razor12911 (Post 490191)

I'm spamming forum

Please, spam a bit more of your usefull stuff :D

Razor12911

14-06-2021 00:20

1 Attachment(s)

Dunia/Disrupt Engine Database maker

Instructions

Drag and drop game folder on make_db.bat, a database file (*.xtl) will be created

Supported games

Far Cry 3
Far Cry 4
Far Cry 5
Far Cry 6
Far Cry New Dawn
The Crew 2
Watch Dogs 1 (use bms2xtl)
Watch Dogs 2
Watch Dogs Legion

L33THAK0R

14-06-2021 02:28

Quote:

Originally Posted by Razor12911 (Post 492876)

Dunia/Disrupt Engine Database maker

Instructions

Drag and drop game folder on make_db.bat, a database file (*.xtl) will be created

Supported games

Far Cry 3
Far Cry 4
Far Cry 5 (needs testing)
Far Cry 6 (I'm waiting for you)
Far Cry New Dawn (needs testing)
The Crew 2
Watch Dogs 1 (wait for xtool v0.4+ update)
Watch Dogs 2 (wait for R2)
Watch Dogs Legion (wait for R2)

Is "Far Cry 2" able to work with this? As far as I know it is one of two titles to use Dunia 1 (the other being the Ubisoft licensed "Avatar" title).

Razor12911

14-06-2021 02:36

upload common.dat, common.fat and all other *.fat files as samples.

:( Sad8669

14-06-2021 03:05

Razor, do you need WDL samples?

I asked because i have the game files right now.

L33THAK0R

14-06-2021 04:17

Quote:

Originally Posted by Razor12911 (Post 492879)

upload common.dat, common.fat and all other *.fat files as samples.

Sample upload is structured as it appears in the title's directory, as some files have identical names. The second title mentioned "James Cameron's Avatar: The Game" does not have a sample included, as it does not seem to have any ".fat", or ".dat" files present, despite it using the same iteration of the Dunia engine as "Far Cry 2". It instead seems to house all of its required assets within ".pak" files.

KaktoR

14-06-2021 11:12

4 Attachment(s)

Here are some for testing

PS: db maker named them all as fc3.xtl (maybe it's a bug), so I have renamed them.

ffmla

15-06-2021 01:43

Quote:

Originally Posted by KaktoR (Post 492890)

Here are some for testing

PS: db maker named them all as fc3.xtl (maybe it's a bug), so I have renamed them.

some doubt arises.

did you change the fc3.xtl name in bat file?

Code:

for /R "%~1\" %%a IN (*.fat) do dunia2_db "%%a" fc3.xtl

KaktoR

15-06-2021 10:49

Quote:

Originally Posted by ffmla (Post 492897)

some doubt arises.

did you change the fc3.xtl name in bat file?

Code:

for /R "%~1\" %%a IN (*.fat) do dunia2_db "%%a" fc3.xtl

Good point. Answer is no :D

Razor12911

26-06-2021 00:19

1 Attachment(s)

Unreal Engine ucas Database maker

Instructions

Open batch file
Insert the name of game, enter the decryption key and then save
Drag and drop game folder on make_db.bat, a database file (*.xtl) along with key file (*.key) will be created

Use these with xtool.

Razor12911

26-06-2021 09:12

Unreal Engine ucas Database maker updated

Changes

- Fixed decryption issues

Notes

That's all I can do for now, fixed some issues during my spare time. I think there's still a few left.

Razor12911

06-10-2021 15:46

Dunia/Disrupt Engine Database maker updated

Changes

- Added support for Far Cry 6

Notes

No idea if this works on the whole game, I tested on 1 sample.

Far Cry 6\data_final\pc\common.dat

Code:

Tested 2 files, 1,287,326,620 => 684,093,252 bytes. Ratio 188.18%

Testing time: cpu 0.80 sec/real 11.15 sec = 7%. Speed 61.36 mB/s

KaktoR

08-10-2021 07:49

For FC6 use lz4 v1.8.2

fctber_disk.dat

Code:

Tested 2 files, 3,646,603,232 => 2,313,847,968 bytes. Ratio 157.60%

Testing time: cpu 0.86 sec/real 17.36 sec = 5%. Speed 133.32 mB/s

Razor12911

05-01-2022 16:38

Dunia/Disrupt Engine Database maker updated

Changes

- Added support for all* fat files in Watch Dogs Legion

Notes

I have no idea if all fat files are supported (I don't have game, only source is torrent files which my ISP does not appreciate so...)
This is perhaps why I stopped pirating games and started buying them

You'll need to wait for next xtool update for this to work, something is up with the main program causing it to crash.

KaktoR

07-01-2022 07:35

1 Attachment(s)

Watch Dogs Legion

Code:

common.dat: 32781 stream(s)

commonengine.dat: 22 stream(s)

patch.dat: 113756 stream(s)

patch_brazilian.dat: 7255 stream(s)

patch_english.dat: 82028 stream(s)

patch_french.dat: 9519 stream(s)

patch_german.dat: 8264 stream(s)

patch_italian.dat: 7126 stream(s)

patch_japanese.dat: 3600 stream(s)

patch_russian.dat: 6981 stream(s)

patch_spanish.dat: 4819 stream(s)

shadersobj.dat: 168500 stream(s)

london.dat: 103303 stream(s)

london_brazilian.dat: 84705 stream(s)

london_cache.dat: 27030 stream(s)

london_english.dat: 84690 stream(s)

london_french.dat: 83419 stream(s)

london_german.dat: 84721 stream(s)

london_hires.dat: 41591 stream(s)

london_italian.dat: 84720 stream(s)

london_japanese.dat: 84718 stream(s)

london_preload.dat: 43551 stream(s)

london_russian.dat: 85156 stream(s)

london_spanish.dat: 84667 stream(s)

london_ultra.dat: 12124 stream(s)

Database created successfully

Razor12911

17-01-2022 11:46

Information on main post added

Changes

- Added information showcasing database generation feature

Razor12911

29-01-2022 14:04

1 Attachment(s)

Information on QuickBMS to Xtool added

Changes

- Added information and attachment showcasing bms2xtool (bms script to xtl database file)

dixen

29-01-2022 23:55

I try it on LEGO I.J but

Quote:

d:\RePack_tools\bms2xtool_0.1>bms2xtl ttgames.bms "GAME.DAT" lego.xtl
Exception: File operation not supported

UPD.
LEGO Batman The Videogame - same error((

Masquerade

30-01-2022 01:11

Quote:

Originally Posted by dixen (Post 495600)

I try it on LEGO I.J but

UPD.
LEGO Batman The Videogame - same error((

Recompression of LZ2K is not supported. QBMS can only extract LZ2K, not compress.

dixen

30-01-2022 02:01

Quote:

Originally Posted by Masquerade (Post 495601)

Recompression of LZ2K is not supported. QBMS can only extract LZ2K, not compress.

Damn..I forgot about that

Razor12911

16-02-2022 22:40

1 Attachment(s)

QuickBMS to XTool updated

Changes

Fixed issues with some scripts not being parsed correctly (thanks Masquerade)
Added support for xtool v0.4+
Made overall progress to be visible from taskbar

Notes
You can find source code from here

Razor12911

21-02-2022 15:01

QuickBMS to XTool updated

Changes

- Added user input option (auto write to cmd window any common prompts)
- Fixed issue with string to binary conversion of keys
- Fixed issue with handling streams that are both encrypted and compressed
- Fixed issue with storing incorrect decryption keys

Notes

It can be sometimes difficult to properly detect streams when you use xtool directly, especially oodle streams if decompressed streams is unknown however if a quick bms script was made, you can take advantage of this and quickly generate a database to use in xtool. Same goes for unreal engine games where ue4d plugin seems to not work as intended, you can loaded up the unreal_tournament_4 script, enter the key and everything should be taken care of.

Some results on Project Cars 3

bdt+oo2reck:

Code:

Compressed 1 file, 538,214,104 => 1,028,286,891 bytes. Ratio 191.06%

Compression time: cpu 0.77 sec/real 160.61 sec = 0%. Speed 3.35 mB/s

bdt+xtool:kraken:

Code:

Compressed 1 file, 538,214,104 => 1,035,969,556 bytes. Ratio 192.48%

Compression time: cpu 0.77 sec/real 141.78 sec = 1%. Speed 3.80 mB/s

xtool:pcars3 using (nfsshift.bms script):

Code:

Compressed 1 file, 538,214,104 => 1,058,023,182 bytes. Ratio 196.58%

Compression time: cpu 0.69 sec/real 25.66 sec = 3%. Speed 20.98 mB/s

Not only you get better results but the process is also faster. :)

dixen

22-02-2022 01:44

1 Attachment(s)

Test database for Mortal Kombat 11

XTool+mk.xtl

Pack

Quote:

Compressed 2 files, 1,694,239,720 => 3,013,304,346 bytes. Ratio 177.86%
Compression time: cpu 1.53 sec/real 137.23 sec = 1%. Speed 12.35 mB/s
All OK

Unpack

Quote:

Tested 2 files, 3,013,304,346 => 1,694,239,720 bytes. Ratio 177.86%
Testing time: cpu 1.78 sec/real 120.46 sec = 1%. Speed 14.06 mB/s
All OK

*Requied oo2core_5_win64.dll from game folder

Xtoo(0.42)+mk11.xtl+srep+lzma2

Quote:

Compressed 3,786 files, 81,540,296,959 => 59,769,328,360 bytes. Ratio 73.30%
Compression time: cpu 838.47 sec/real 23004.64 sec = 4%. Speed 3.54 mB/s
All OK

**Requied ~300 gb for decompressiion and srep proccess

Razor12911

22-02-2022 08:03

QuickBMS to XTool updated

Changes

- Added safe mode feature

Notes

This issue is not related to bms2xtl but it's related to QuickBMS itself and was detected by KaktoR where quickbms will run out of memory when trying to process some streams especially ones where oodle is used, since quickbms internally uses old oodle library taken from Warframe, newer games seem to cause the old library to possibly have a memory leak.

Safe mode mitigates this issue (only use it if you have an issue when QuickBMS produces an error related to memory or something similar and it should be kept disabled under normal use)

Luminior

23-02-2022 03:13

Sword and Fairy 7
Method: bms2xtool v0.3 + xtool v0.4.2 + srep + lolz

Quote:

Compressed 16 files, 69,496,034,241 => 34,281,984,937 bytes. Ratio 49.33%
Compression time: cpu 1290.97 sec/real 41904.95 sec = 3%. Speed 1.66 mB/s
All OK

Notes: other libraries were not used.

Razor12911

20-03-2022 17:33

QuickBMS to XTool updated

Changes

- Fixed issue with file inputs smaller than 4MB

Razor12911

24-03-2022 12:05

Unreal Engine ucas Database maker updated

Changes

- Added support for new structure seen used in Ghostwire Tokyo

Notes

This still uses the old xtool 0.3.21 database structure, wait for another update if you plan on using it with 0.4+

kuyhaa

24-03-2022 19:03

Quote:

Originally Posted by Razor12911 (Post 496279)

Compressing 2 files, 850,906,556 bytes
Compressing Snowfall-WindowsNoEditor_0_P.pak
Compressing Snowfall-WindowsNoEditor_0_P.ucas
Compressed 2 files, 850,906,556 => 850,907,110 bytes. Ratio 100.00%
Compression time: cpu 6.33 sec/real 8.72 sec = 73%. Speed 97.60 mB/s
All OK

-mgwtokyo+zlib -d1

unreal gwtokyo.xtl and gwtokyo.key near xtool

have wrong ?

Razor12911

24-03-2022 19:21

kraken was used, not zlib

Masquerade

24-03-2022 23:13

Quote:

Originally Posted by Razor12911 (Post 496282)

kraken was used, not zlib

Code:

Compressing 3 files, 851,793,200 bytes

  Compressing Snowfall-WindowsNoEditor_0_P.pak

  Compressing Snowfall-WindowsNoEditor_0_P.ucas

  Compressing Snowfall-WindowsNoEditor_0_P.utoc

Compressed 3 files, 851,793,200 => 851,805,895 bytes. Ratio 100.00%

Compression time: cpu 0.84 sec/real 145.87 sec = 1%. Speed 5.84 mB/s

All OK

Code:

[External compressor: gwt]

header    = 0

packcmd   = xtool precomp -mgwtokyo+kraken -c256mb -t16 - - <stdin> <stdout>

xtool v3.2.1, oo2core_9_win64.dll

:confused::confused:

Razor12911

24-03-2022 23:14

Quote:

Originally Posted by Masquerade (Post 496284)

Code:

Compressing 3 files, 851,793,200 bytes

  Compressing Snowfall-WindowsNoEditor_0_P.pak

  Compressing Snowfall-WindowsNoEditor_0_P.ucas

  Compressing Snowfall-WindowsNoEditor_0_P.utoc

Compressed 3 files, 851,793,200 => 851,805,895 bytes. Ratio 100.00%

Compression time: cpu 0.84 sec/real 145.87 sec = 1%. Speed 5.84 mB/s

All OK

Code:

[External compressor: gwt]

header    = 0

packcmd   = xtool precomp -mgwtokyo+kraken -c256mb -t16 - - <stdin> <stdout>

xtool v3.2.1, oo2core_9_win64.dll

:confused::confused:

-d1 :(

dixen

24-03-2022 23:35

oo2core_7_win64.dll

*.pak

Quote:

Streams: 401/402/402
Time: 00:00:23 (00:03:24)
Memory: 392 MB (392 MB)

100%
Errorlevel=0
Compressed 1 file, 622,183,516 => 848,346,910 bytes. Ratio 136.35%
Compression time: cpu 0.55 sec/real 25.16 sec = 2%. Speed 24.73 mB/s
All OK

*.ucas

Quote:

Streams: 5550/5564
Time: 00:02:03 (00:08:04)
Memory: 776 MB (776 MB)

100%
Errorlevel=0
Compressed 1 file, 228,723,040 => 910,585,777 bytes. Ratio 398.12% 100%
Compression time: cpu 0.20 sec/real 124.58 sec = 0%. Speed 1.84 mB/s
All OK

kuyhaa

25-03-2022 01:31

Quote:

Originally Posted by dixen (Post 496286)

oo2core_7_win64.dll

*.pak

*.ucas

oo2core_7

Quote:

Compressing 228,723,040 bytes

xtool321.exe precomp -mgwtokyo+kraken -c256mb -d1

XTool is created by Razor12911

Streams: 38/5640
Time: 00:02:53 (00:27:21)
Memory: 128 MB (128 MB)

100%

confiused :(

dixen

25-03-2022 01:37

Quote:

Originally Posted by kuyhaa (Post 496287)

confiused :(

oo2core_7_win64.dll 569.0 Кб

kuyhaa

25-03-2022 01:55

Quote:

Originally Posted by dixen (Post 496288)

oo2core_7_win64.dll 569.0 Кб

solved

Razor12911

28-03-2022 10:41

Unreal Engine ucas Database maker updated

Changes

- Added support for newer xtool 0.4+

All times are GMT -7. The time now is 07:46.

Page 1 of 2

Show 40 post(s) from this thread on one page