FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   XTool 2020 (Main Project) (https://fileforums.com/showthread.php?t=102832)

FitGirl 12-08-2019 19:18

Quote:

Originally Posted by ZAZA4EVER (Post 482067)
oodle leviathan Stream Dont Work :o

They have custom chunks in there, no size in headers, so the decompressor should guess the right size, which is not a good idea :) Also it's slow af in Oodle 2.6.* .

Razor12911 12-08-2019 19:25

Quote:

Originally Posted by ZAZA4EVER (Post 482067)
oodle leviathan Stream Dont Work :o

Quote:

Originally Posted by Razor12911 (Post 482063)
Notes
Placeholder does not mean the codec is ready to be used universally, it's intended for plugins

Wolfenstein Youngblood ... chunkbase_9_pc.resources

- Could you upload a sample, I need to check something


Quote:

Originally Posted by FitGirl (Post 482068)
They have custom chunks in there, no size in headers, so the decompressor should guess the right size, which is not a good idea :) Also it's slow af in Oodle 2.6.* .

Last time I checked the headers did contain size, it was a while ago but I didn't add support because no game used this codec back then

Simorq 16-08-2019 07:49

Hi Razor12911

Assassin's Creed: Origins
Test File: DataPC_extra.forge / oodle4

Code:

Creating archive: data.arc using oodle
Compressed 1 file, 1,049,427,968 => 1,049,428,104 bytes. Ratio 100.00%   
Compression time: cpu 1.47 sec/real 4.11 sec = 36%. Speed 255.62 mB/s
All OK

__________________________________________________ _____________________________________________
DataPC_SharedGroup_02.forge Sample File
Oodle_DLL_x86/x64

Oodle precompressor (Side project)
Code:

Creating archive: data.arc using oo2recm
Compressed 1 file, 105,512,960 => 167,404,655 bytes. Ratio 158.66%       
Compression time: cpu 0.17 sec/real 15.59 sec = 1%. Speed 6.77 mB/s
All OK

XTool 1908_R3
Code:

Creating archive: data2.arc using oodle
Compressed 1 file, 105,512,960 => 105,513,032 bytes. Ratio 100.00%       
Compression time: cpu 0.09 sec/real 0.53 sec = 18%. Speed 199.34 mB/s
All OK


______________________________________

The problem was solved thanks to KaktoR.

doofoo24 16-08-2019 08:34

Quote:

Originally Posted by Razor12911 (Post 482063)
- Added LZ4 codec placeholders (lz4,lz4hc)
- Added LZO codec placeholders (lzo1c,lzo1x,lzo2a)
- Added ZSTD codec placeholders (zstd)
- Added Oodle codec placeholders (lzna,kraken,mermaid,selkie,leviathan)
- Added LZX codec placeholders (lzx)

"placeholders" :)

KaktoR 16-08-2019 08:50

@Simorq:

Oodle is not inside xtool 2019 yet (they are only "placeholders" for later use). Only zlib and unity lz4 are inside.

Razor12911 16-08-2019 15:13

Yes they are placeholders meant to be used by the plugins, they will get universal support eventually. But the plugins should give the best output since they are specific and not universal and as we know the universal tools I made (side projects) couldn't handle some games properly so I added placeholders to try and give the best precompression via plugins

@everyone, update was delayed. Because I noticed several issues in xtool thanks to Panker, FitGirl and of course you guys, plus I was busy making plugins for some games.

The changelog for next update

- Updated plugin support [DONE]
- Fixed plugin stream detection issues [DONE]
- Added delta/diff support for imperfect streams [0%]
- zlib codec renamed to deflate [DONE]
- Fixed several deflate related issues [DONE]
- Added png stream support [90%]
- Fixed future stream issues [DONE]
- Added low memory usage mode [0%]
- Added zstd universal stream support [DONE]
- Added lzo1x universal stream support (alpha) [DONE]

Plugins done so far:

haemimont, frostbite and anvil and updated unity plugin

K4miKaZe 16-08-2019 15:25

This is looking good.

Many thanks Razor12911 !!!

ansi 19-08-2019 02:48

When packing CS:GO vpk files, xtool 1908 R3 freezes when got to 5%.
Parameters: zlib,lz4:c32mb,t100p
However, there are no freezes on the xtool 1908 R2 version.

Andu21 21-08-2019 08:17

Remove lz4 from the parameters and try again.

Razor12911 21-08-2019 15:10

Update available

1908_R4
- Updated plugin support
- Fixed plugin stream detection issues
- Added delta/diff support for imperfect streams
- zlib codec renamed to deflate
- Fixed several deflate related issues
- Added png stream support
- Fixed future stream issues
- Added low memory usage mode (slower)

Notes

If this update does not fix most issues then my PC crashing has really ruined the development of this project. :(
If you take a look at the change log, it says I renamed zlib codec to deflate, well the reason for that is because zlib codec will be reintroduced later on but it will be mostly use by plugins that find zlib streams in games so as a normal user, use deflate

doofoo24 21-08-2019 15:19

is plugin still placeholder (lz4/lzo...) ?

Razor12911 21-08-2019 15:21

yep, lz4 will forever be a placeholder because I really have no idea how to detect stream, as for lzo well that is a placeholder for that version but next version, it will be there.

nik1967 22-08-2019 00:30

Hello, Razor12911!
What's this? :eek:
The file was downloaded several times.
https://i89.fastpic.ru/big/2019/0822...4f18ef55de.png

dixen 22-08-2019 01:27

Quote:

Originally Posted by nik1967 (Post 482269)
Hello, Razor12911!
What's this? :eek:
The file was downloaded several times.
https://i89.fastpic.ru/big/2019/0822...4f18ef55de.png

Same bug)

IgorKolesnik 22-08-2019 02:02

When downloading, it says that the archive is corrupted. (

IgorKolesnik 22-08-2019 02:27

Quote:

Originally Posted by doofoo24 (Post 482273)
use this

thank

Razor12911 22-08-2019 15:30

what is going here lol :D

Snake288 22-08-2019 18:37

Hello,
doofoo24,Razor12911

xtool_1908_R4.7z

The file has been downloaded several times.

file won't open

When downloading, it says that the archive is corrupted.

Razor12911 22-08-2019 19:01

Update available

1908_R5
- Added lzo1x universal stream detection (alpha)

Notes

I've ran a few tests on Far Cry 3 + Blood Dragon and seems like it is working, what you must know is this thing is slow but there is still room for improvement. For each stream detected, there are 2 trials taken and from those trials at least one of them is incorrect furthermore from those 2 trials, there is level detection from level 1-9 so you can imagine the speed deduction but there are ways to improve this somewhere down the line.

Maximum stream support is 4MB, any stream larger than this is not considered.

Streams compressed with lzo1x_999 are the only ones supported.

And don't use this on Unreal Engine games or games that have been compressed with something like lzopro because it will NOT work.

Grumpy 22-08-2019 19:02

@Razor

The same server problem may have corrupted this file as well ;)

Razor12911 22-08-2019 19:12

@Grumpy

hahaha.

@everyone

If you have issues downloading, here is external link
https://mega.nz/#F!03BkjKQA!WpTxyngxucC-ToBRK9xbkw

Grumpy 23-08-2019 02:11

@Razor

I have only just found this out, according to Empire the 4 affected Threads and Files were all posted/uploaded by 'mausschieber' ... so you may have to check your files/uploads afterall. Sorry :o

Razor12911 29-08-2019 15:19

Update available

1908_R6
- Added zlib codec placeholder (zlib)
- Added zstd universal stream detection
- Added kraken, mermaid, selkie universal stream detection
- Fixed plugin issues
- Improved lz4, lzo, zstd, oodle processing speeds
- Removed lzx placeholder (codec complicated)

Simorq 29-08-2019 16:06

1908_R6 32/64 bit kraken Test
 
http://uupload.ir/files/ubam_1.jpg

CPU Stuck...

Tested with: t1/t50p/t100p - c16m/c32m/c64m/c128m


Code:

File Test: ShadowOfWar (snowmtn_lighting.arch06 1.59 GB)

doofoo24 29-08-2019 16:59

yah multi threading stuck during encode :(
but t1 work normal...

Razor12911 29-08-2019 17:28

I only started working on this like today, was busy the whole week and I am completely not surprised about the errors considering I didn't really run any tests :D

While you guys are at it, try the zstd codec, that's also not tested.

Forgot to release notes:

Well I did tweak the detection a bit to detect more streams than oo2reck and its friends

IgorKolesnik 29-08-2019 23:41

1 Attachment(s)
when using the xtool R5 reflate version, this error pops up. deflate method

using parameter
packcmd = xtool.exe precomp:deflate:t1,c16mb - - <stdin> <stdout>

Razor12911 30-08-2019 06:34

Quote:

Originally Posted by IgorKolesnik (Post 482432)
when using the xtool R5 reflate version, this error pops up. deflate method

using parameter
packcmd = xtool.exe precomp:deflate:t1,c16mb - - <stdin> <stdout>

Someone also reported a similar issue however I need a sample to fix this sort of issue.

IgorKolesnik 30-08-2019 06:41

Quote:

Originally Posted by Razor12911 (Post 482434)
Someone also reported a similar issue however I need a sample to fix this sort of issue.

tested on Ancestors: The Humankind Odyssey

doofoo24 03-09-2019 21:23

tested (xtool_1908_R6) zstd on MEA xtool.exe remain stuck 0%...
tested (xtool_1908_R6/xtool_1908_R5/xtool_1908_R4) deflate on Spyro Reignited Trilogy pak files always error CRC during install...

KaktoR 10-09-2019 11:57

Razor, if you start working on oodle codec again in future, could you take a look on this sample please? It uses oodle5 dll but your oodle scan detects only [0] for unknown.

FitGirl 10-09-2019 14:12

Quote:

Originally Posted by KaktoR (Post 482565)
Razor, if you start working on oodle codec again in future, could you take a look on this sample please? It uses oodle5 dll but your oodle scan detects only [0] for unknown.

Standard TellTale encryption inside.

ozolt 31-10-2019 08:19

Hi Razor12911, can you please rename zstd codec for xtool precomp:{compressor} to something else? I'm asking because [External compressor:zstd] was common entry for standard zstd.exe (for me it works faster than CLS versions). Now we can't use both at the same time since xtool uses same identifier in arc.ini... Yes, I know I can create separate xtool 2019 entry in arc.ini just for that codec but I would rather avoid that if possible.
Regards

chichi119 03-11-2019 01:52

K*nh chúc bác th*nh công trong công việc............................................ ................!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

PSP17 06-11-2019 02:57

What combination need to be used for decompression red dead redemption 2 files ?

ShivShubh 06-11-2019 03:37

Quote:

Originally Posted by PSP17 (Post 483344)
What combination need to be used for decompression red dead redemption 2 files ?

oo2reck

FitGirl 06-11-2019 04:12

Quote:

Originally Posted by PSP17 (Post 483344)
What combination need to be used for decompression red dead redemption 2 files ?

Ash Shiv mentioned below, oo2reck. But keep in mind, that RPFs are partially encrypted, so not all streams will be found and recompressed. It's a new encryption scheme, not the one used in GTA5.

doofoo24 16-11-2019 07:00

@Razor12911 any further development on xtool or the project on halt :(

Simorq 20-01-2020 08:11

XTool_R5 32bit
 
Code:

[External compressor:deflate,reflate,unity,lzo1c,lzo1x,lzo2a]
header    = 0
packcmd  = "Razor12911\XTool_R5\XTool.exe" precomp:{compressor}:c32mb,lm,t50p - - <stdin> <stdout>
unpackcmd = "Razor12911\XTool_R5\XTool.exe" decode:t100p - - <stdin> <stdout>

Works well.

The lm option is very important and prevents XTool from getting stuck.
translate.google:p

Razor12911 20-01-2020 08:17

1 Attachment(s)
Quote:

Originally Posted by doofoo24 (Post 483501)
@Razor12911 any further development on xtool or the project on halt :(

Well I had issues handling oodle and every single time I try to fix bugs caused by it I always get errors from nowhere, been doing this for since September last year and I decided to give up on it a few days before new year.


All times are GMT -7. The time now is 20:54.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com