XTool 2020 (Main Project)

[elit]

And preflate is worse than zlib hmmm..:

ff13pref.png

[Razor12911]

Update available

Changes

- added wav stream detector
- added flac codec
- added jpg stream detector
- added packjpg, brunsli, jojpeg codec
- added feature that allows input to be a directory
- added feature to extract detected streams
- updated database feature
- updated deduplication feature
- IO function decode updated

Notes

I have added wav audio compression as an alternative of msc for people who has issues with it or don't want to use Freearc's tta codec as it processes wav files individually. One thing to note, it's not as good as tta, tak or even frog as it is based on flac codec (It was the only open source codec I could work with...).

Also added jpg image compression codecs, you can pick between packjpg, brunsli or jojpeg. packjpg seems to have a memory leak, brunsli is fast but cannot deal with large jpg images and jojpeg can be used if you're after ratio as it is paq based (seems to be buggy so stick to packjpg or brunsli for now)

A new parameter is added --extract=[path], this will extract all detected streams to a directory... if you're interested in the streams themselves.

database and deduplication features have been updated and can now be used at all times without worrying about crc collisions.

Special thanks to KaktoR for running several tests and Shelwien for providing compiled libraries for brunsli and jojpeg.

@elit
These issues will be investigated in time.

[dixen]

Find little uncritical bug

Compressing 77,790,338 bytes with xtool.exe precomp -mFLAC -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 1/1
Time: 00:00:02 (00:00:02)
Memory: 276 MB (276 MB)

100%
Errorlevel=0
Compressed 1 file, 77,790,338 => 53,378,764 bytes. Ratio 68.62%
Compression time: cpu 0.08 sec/real 2.78 sec = 3%. Speed 28.01 mB/s
All OK

Compressing 77,790,338 bytes with xtool.exe precomp -mFLAC --verbose -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 0000000001FFFFFF (33554432)
EAccessViolation: Access violation at address 0000000000409A77 in module 'xtool.exe'. Read of address 00007FF4FE6BFBA4

Errorlevel=1

ERROR: general (de)compression error in xtool

Anyway, thank Razor for update and new functions)

One question. Why TTA but not TAK?

Original.wav 74.1 mb
ENCTTA.tta 50.3 mb
ENCTAK.tak 48.7 mb

[Masquerade]

Originally Posted by dixen

One question. Why TTA but not TAK?

TAK isn't Open Source.

[elit]

Interesting, I should note that compression difference between different wav codecs is minimal. Also for me in the past, I found wavpack the best, It handle more formats(WAV/BWF/RF64, Sony Wave64, Apple CAF, Philips DSDIFF, and Sony DSF (all with no size limitations)), more modes(8, 16, 24, and 32-bit integer PCM; 32-bit float PCM; DSD audio; mono, stereo, and multichannel; sampling rates from 6 to 192 kHz (and non-standard rates)), RIFF chunks, is open source, does support STDIO which for me work unlike flac, which for some reason during decompression kept running past original size inflating to infinity(when used in FA with stdio mode).

However, flac is great too because is non-symetric so decompression is fast(although wavpack also seem to support asym. cmp.).

I wonder how this work, if xtool is able to use it on chunks when wav files are inside big game packs, that would be indeed very useful. In fact, I don't know what would be the point beyond that as we could then just use standalone codecs?

[Prince4]

Could someone please remind me what's the difference between the 0.3.21 and 0.6.0 versions?

[Razor12911]

Originally Posted by dixen

Find little uncritical bug

your chunk size is too small for that input

Originally Posted by Prince4

Could someone please remind me what's the difference between the 0.3.21 and 0.6.0 versions?

0.3.21 is an old version that supported old plugins and it is kept there until all plugins are updated to support 0.4+ versions, use 0.6.0 if you are not using old plugins.

Originally Posted by elit

Interesting, I should note that compression difference between different wav codecs is minimal.

https://wiki.hydrogenaud.io/index.ph...ess_comparison

I picked codecs based on the information provided here, it's not that I haven't tried wavpack. I did but it requires additional code as it generates lossy wav and a patch file in case you wanted to restore the original pcm data whereas flac does not. In addition, it doesn't seem to be thread safe which means it cannot work in xtool environment (maybe I'm mistaking it with tta).

Originally Posted by elit

I wonder how this work, if xtool is able to use it on chunks when wav files are inside big game packs, that would be indeed very useful. In fact, I don't know what would be the point beyond that as we could then just use standalone codecs?

That's pretty much how it is intended to work, wav files are scanned within files and then fed to codecs.

[elit]

Originally Posted by Razor12911

I did but it requires additional code as it generates lossy wav and a patch file in case you wanted to restore the original pcm data whereas flac does not. In addition, it doesn't seem to be thread safe which means it cannot work in xtool environment (maybe I'm mistaking it with tta).

That is optional feature with extra switches, by default it work same as flac - no lossy wav. As for thread safety, I believe its single threaded app so could be used chunk per thread but I don't know for sure.
I am also perfectly ok with flac as well so don't worry about it, I just though this one may have been better due to more formats supported.
https://www.wavpack.com

[Razor12911]

Update available

Changes

- added fast lzma2 compression for portable mode
- fixed issues with wav stream detection
- fixed minor issue with stream deduplication feature

Notes

I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Example

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -

[CR2032]

Originally Posted by Razor12911

Update available

Changes

- added fast lzma2 compression for portable mode
- fixed issues with wav stream detection
- fixed minor issue with stream deduplication feature

Notes

I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Example

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -

Thank You So Much For Your Hard Work, Razor12911!

[L33THAK0R]

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

[:( Sad8669]

As far as i know, it was added to increase the portability of xtool.

Notes:
I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

[Cesar82]

Originally Posted by Razor12911

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -

XTool is going independent, abandoning its helpers, lol.
Maybe in the future it will get its own final compression with high compression gain and higher performance than lolz... It would be great!

[Masquerade]

Originally Posted by L33THAK0R

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

No, it's just a final compresison option. A bit like how precomp has built in LZMA compressor

@Razor thanks for the continued updates!

[CR2032]

Originally Posted by L33THAK0R

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

Hello.
No. It's just a creative and really a gifted step, not more.