FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   XTool 2020 (Main Project) (https://fileforums.com/showthread.php?t=102832)

elit 04-07-2022 06:20

1 Attachment(s)
And preflate is worse than zlib hmmm..:

Attachment 32017

Razor12911 07-07-2022 02:53

Update available

Changes

- added wav stream detector
- added flac codec
- added jpg stream detector
- added packjpg, brunsli, jojpeg codec
- added feature that allows input to be a directory
- added feature to extract detected streams
- updated database feature
- updated deduplication feature
- IO function decode updated

Notes

I have added wav audio compression as an alternative of msc for people who has issues with it or don't want to use Freearc's tta codec as it processes wav files individually. One thing to note, it's not as good as tta, tak or even frog as it is based on flac codec (It was the only open source codec I could work with...).

Also added jpg image compression codecs, you can pick between packjpg, brunsli or jojpeg. packjpg seems to have a memory leak, brunsli is fast but cannot deal with large jpg images and jojpeg can be used if you're after ratio as it is paq based (seems to be buggy so stick to packjpg or brunsli for now)

A new parameter is added --extract=[path], this will extract all detected streams to a directory... if you're interested in the streams themselves.

database and deduplication features have been updated and can now be used at all times without worrying about crc collisions.

Special thanks to KaktoR for running several tests and Shelwien for providing compiled libraries for brunsli and jojpeg.

@elit
These issues will be investigated in time.

dixen 07-07-2022 03:52

Find little uncritical bug

Quote:

Compressing 77,790,338 bytes with xtool.exe precomp -mFLAC -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 1/1
Time: 00:00:02 (00:00:02)
Memory: 276 MB (276 MB)

100%
Errorlevel=0
Compressed 1 file, 77,790,338 => 53,378,764 bytes. Ratio 68.62%
Compression time: cpu 0.08 sec/real 2.78 sec = 3%. Speed 28.01 mB/s
All OK

Quote:

Compressing 77,790,338 bytes with xtool.exe precomp -mFLAC --verbose -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 0000000001FFFFFF (33554432)
EAccessViolation: Access violation at address 0000000000409A77 in module 'xtool.exe'. Read of address 00007FF4FE6BFBA4

Errorlevel=1

ERROR: general (de)compression error in xtool
Anyway, thank Razor for update and new functions)

One question. Why TTA but not TAK?

Quote:

Original.wav 74.1 mb
ENCTTA.tta 50.3 mb
ENCTAK.tak 48.7 mb

Masquerade 07-07-2022 04:40

Quote:

Originally Posted by dixen (Post 497488)
One question. Why TTA but not TAK?

TAK isn't Open Source.

elit 07-07-2022 04:55

Interesting, I should note that compression difference between different wav codecs is minimal. Also for me in the past, I found wavpack the best, It handle more formats(WAV/BWF/RF64, Sony Wave64, Apple CAF, Philips DSDIFF, and Sony DSF (all with no size limitations)), more modes(8, 16, 24, and 32-bit integer PCM; 32-bit float PCM; DSD audio; mono, stereo, and multichannel; sampling rates from 6 to 192 kHz (and non-standard rates)), RIFF chunks, is open source, does support STDIO which for me work unlike flac, which for some reason during decompression kept running past original size inflating to infinity(when used in FA with stdio mode).

However, flac is great too because is non-symetric so decompression is fast(although wavpack also seem to support asym. cmp.).

I wonder how this work, if xtool is able to use it on chunks when wav files are inside big game packs, that would be indeed very useful. In fact, I don't know what would be the point beyond that as we could then just use standalone codecs?

Prince4 07-07-2022 05:01

Could someone please remind me what's the difference between the 0.3.21 and 0.6.0 versions?

Razor12911 07-07-2022 06:27

Quote:

Originally Posted by dixen (Post 497488)
Find little uncritical bug

your chunk size is too small for that input
Quote:

Originally Posted by Prince4 (Post 497491)
Could someone please remind me what's the difference between the 0.3.21 and 0.6.0 versions?

0.3.21 is an old version that supported old plugins and it is kept there until all plugins are updated to support 0.4+ versions, use 0.6.0 if you are not using old plugins.

Quote:

Originally Posted by elit (Post 497490)
Interesting, I should note that compression difference between different wav codecs is minimal.

https://wiki.hydrogenaud.io/index.ph...ess_comparison

I picked codecs based on the information provided here, it's not that I haven't tried wavpack. I did but it requires additional code as it generates lossy wav and a patch file in case you wanted to restore the original pcm data whereas flac does not. In addition, it doesn't seem to be thread safe which means it cannot work in xtool environment (maybe I'm mistaking it with tta).

Quote:

Originally Posted by elit (Post 497490)
I wonder how this work, if xtool is able to use it on chunks when wav files are inside big game packs, that would be indeed very useful. In fact, I don't know what would be the point beyond that as we could then just use standalone codecs?

That's pretty much how it is intended to work, wav files are scanned within files and then fed to codecs.

elit 07-07-2022 10:43

Quote:

Originally Posted by Razor12911 (Post 497494)
I did but it requires additional code as it generates lossy wav and a patch file in case you wanted to restore the original pcm data whereas flac does not. In addition, it doesn't seem to be thread safe which means it cannot work in xtool environment (maybe I'm mistaking it with tta).

That is optional feature with extra switches, by default it work same as flac - no lossy wav. As for thread safety, I believe its single threaded app so could be used chunk per thread but I don't know for sure.
I am also perfectly ok with flac as well so don't worry about it, I just though this one may have been better due to more formats supported.
https://www.wavpack.com

Razor12911 12-07-2022 12:43

Update available

Changes

- added fast lzma2 compression for portable mode
- fixed issues with wav stream detection
- fixed minor issue with stream deduplication feature

Notes

I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Example
Code:

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -

CR2032 13-07-2022 07:30

Quote:

Originally Posted by Razor12911 (Post 497597)
Update available

Changes

- added fast lzma2 compression for portable mode
- fixed issues with wav stream detection
- fixed minor issue with stream deduplication feature

Notes

I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Example
Code:

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -

Thank You So Much For Your Hard Work, Razor12911!

L33THAK0R 13-07-2022 08:01

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

:( Sad8669 13-07-2022 09:06

As far as i know, it was added to increase the portability of xtool.

Code:

Notes:
I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.


Cesar82 13-07-2022 12:29

Quote:

Originally Posted by Razor12911 (Post 497597)
Code:

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -

XTool is going independent, abandoning its helpers, lol.
Maybe in the future it will get its own final compression with high compression gain and higher performance than lolz... It would be great!

Masquerade 13-07-2022 13:14

Quote:

Originally Posted by L33THAK0R (Post 497604)
Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

No, it's just a final compresison option. A bit like how precomp has built in LZMA compressor :)

@Razor thanks for the continued updates!

CR2032 14-07-2022 15:45

Quote:

Originally Posted by L33THAK0R (Post 497604)
Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

Hello.
No. It's just a creative and really a gifted step, not more.:)


All times are GMT -7. The time now is 02:51.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com