XTool 2020 (Main Project)

[Simorq]

#############################################
# reflate (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

XTool x86 Command Line

xtool.exe precomp:reflate:c16mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

Test 1 = OK
Test 2 = OK
Test 3 = OK
Test 4 = OK

XTool x86 Command Line

xtool.exe precomp:reflate:c32mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

Test 1

P: OK
D: OK (FC: no differences encountered)

Test 2

P: EAccessViolation: Access violation at address 5378614D in module 'xtool.exe'. Read of address 5378614D
XTool is created by Razor12911

D: EReadError: Stream read error


Test 3

P: Exception EAccessViolation in module xtool.exe at 000BCEC7.
Access violation at address 004BCEC7 in module 'xtool.exe'. Read of address 00611D34.
D: EReadError: Stream read error

reflate works well with c16m in x86.

[Simorq]

#############################################
# zlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

XTool x86 Command Line

zlib
xtool.exe precomp:zlib:c32mb and c16mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res


TEST 1
P: OK
D: CRC Error (0F2C892C: 09 D7)

TEST 2
P: Error (EReadError: Stream read error)

TEST 3
P: Error (EReadError: Stream read error)

TEST 4
P: EAccessViolation: Access violation at address 00000000 in module 'xtool.exe'. Read of address 00000000

D: EReadError: Stream read error

zlib is very unstable, once successful, failed several times

[Simorq]

#############################################
# rzlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

XTool x86 Command Line

rzlib
xtool.exe precomp:rzlib:c16mb/c32mb/c64mb/c128mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

TEST 1
P: Error (EReadError: Stream read error)

TEST 2
P: Error (EReadError: Stream read error)

TEST 3
P: Error (EReadError: Stream read error)

TEST 4
P: Error (EReadError: Stream read error)

TEST 5
P: Error (EReadError: Stream read error)

rzlib always fails.

[Razor12911]

@Simorq

Looks like I'll have to go through debugging road once more

@everyone

Update available

1908_R3
- Compatiblity broken (old history db and inputs cannot work in this version)
- Added plugin support
- Added LZ4 codec placeholders (lz4,lz4hc)
- Added LZO codec placeholders (lzo1c,lzo1x,lzo2a)
- Added ZSTD codec placeholders (zstd)
- Added Oodle codec placeholders (lzna,kraken,mermaid,selkie,leviathan)
- Added LZX codec placeholders (lzx)
- rzlib codec removed (use zlib)
- Added x86 memory limit

Notes
Placeholder does not mean the codec is ready to be used universally, it's intended for plugins which can be found here

[ZAZA4EVER]

xtool_1908_R3 X64 ZLIB CODEC
eFootball Pes2020 Demo ... dt00_4K_x64.cpk

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Test.bin
Compressed 1 file, 15,252,601 => 72,893,019 bytes. Ratio 477.91%
Compression time: cpu 0.05 sec/real 37.50 sec = 0%. Speed 0.41 mB/s
All OK

Zlib Codec is Stable .. i tested it Three Times ...
/////////////////////////////////////////////////////////////
xtool_1908_R3 X64 oodle Codec
Wolfenstein Youngblood ... chunkbase_9_pc.resources

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Test.bin
Compressed 1 file, 100,552,642 => 100,552,714 bytes. Ratio 100.00%
Compression time: cpu 0.33 sec/real 3.38 sec = 10%. Speed 29.79 mB/s
All OK

oodle leviathan Stream Dont Work

[FitGirl]

Originally Posted by ZAZA4EVER

oodle leviathan Stream Dont Work

They have custom chunks in there, no size in headers, so the decompressor should guess the right size, which is not a good idea Also it's slow af in Oodle 2.6.* .

[Razor12911]

Originally Posted by ZAZA4EVER

oodle leviathan Stream Dont Work

Originally Posted by Razor12911

Notes
Placeholder does not mean the codec is ready to be used universally, it's intended for plugins

Wolfenstein Youngblood ... chunkbase_9_pc.resources

- Could you upload a sample, I need to check something

Originally Posted by FitGirl

They have custom chunks in there, no size in headers, so the decompressor should guess the right size, which is not a good idea Also it's slow af in Oodle 2.6.* .

Last time I checked the headers did contain size, it was a while ago but I didn't add support because no game used this codec back then

[Simorq]

Hi Razor12911

Assassin's Creed: Origins
Test File: DataPC_extra.forge / oodle4

Creating archive: data.arc using oodle
Compressed 1 file, 1,049,427,968 => 1,049,428,104 bytes. Ratio 100.00%     
Compression time: cpu 1.47 sec/real 4.11 sec = 36%. Speed 255.62 mB/s
All OK

__________________________________________________ _____________________________________________
DataPC_SharedGroup_02.forge Sample File
Oodle_DLL_x86/x64

Oodle precompressor (Side project)

Creating archive: data.arc using oo2recm
Compressed 1 file, 105,512,960 => 167,404,655 bytes. Ratio 158.66%         
Compression time: cpu 0.17 sec/real 15.59 sec = 1%. Speed 6.77 mB/s
All OK

XTool 1908_R3

Creating archive: data2.arc using oodle
Compressed 1 file, 105,512,960 => 105,513,032 bytes. Ratio 100.00%         
Compression time: cpu 0.09 sec/real 0.53 sec = 18%. Speed 199.34 mB/s
All OK

______________________________________

The problem was solved thanks to KaktoR.

[doofoo24]

Originally Posted by Razor12911

- Added LZ4 codec placeholders (lz4,lz4hc)
- Added LZO codec placeholders (lzo1c,lzo1x,lzo2a)
- Added ZSTD codec placeholders (zstd)
- Added Oodle codec placeholders (lzna,kraken,mermaid,selkie,leviathan)
- Added LZX codec placeholders (lzx)

"placeholders"

[KaktoR]

@Simorq:

Oodle is not inside xtool 2019 yet (they are only "placeholders" for later use). Only zlib and unity lz4 are inside.

[Razor12911]

Yes they are placeholders meant to be used by the plugins, they will get universal support eventually. But the plugins should give the best output since they are specific and not universal and as we know the universal tools I made (side projects) couldn't handle some games properly so I added placeholders to try and give the best precompression via plugins

@everyone, update was delayed. Because I noticed several issues in xtool thanks to Panker, FitGirl and of course you guys, plus I was busy making plugins for some games.

The changelog for next update

- Updated plugin support [DONE]
- Fixed plugin stream detection issues [DONE]
- Added delta/diff support for imperfect streams [0%]
- zlib codec renamed to deflate [DONE]
- Fixed several deflate related issues [DONE]
- Added png stream support [90%]
- Fixed future stream issues [DONE]
- Added low memory usage mode [0%]
- Added zstd universal stream support [DONE]
- Added lzo1x universal stream support (alpha) [DONE]

Plugins done so far:

haemimont, frostbite and anvil and updated unity plugin

[K4miKaZe]

This is looking good.

Many thanks Razor12911 !!!

[ansi]

When packing CS:GO vpk files, xtool 1908 R3 freezes when got to 5%.
Parameters: zlib,lz4:c32mb,t100p
However, there are no freezes on the xtool 1908 R2 version.

[Andu21]

Remove lz4 from the parameters and try again.

[Razor12911]

Update available

1908_R4
- Updated plugin support
- Fixed plugin stream detection issues
- Added delta/diff support for imperfect streams
- zlib codec renamed to deflate
- Fixed several deflate related issues
- Added png stream support
- Fixed future stream issues
- Added low memory usage mode (slower)

Notes

If this update does not fix most issues then my PC crashing has really ruined the development of this project.
If you take a look at the change log, it says I renamed zlib codec to deflate, well the reason for that is because zlib codec will be reintroduced later on but it will be mostly use by plugins that find zlib streams in games so as a normal user, use deflate