LZ_Scan

[elit]

LZ_Scan is a small scripted utility that consist of several tools and is primarily based on Razor's xtool. It is a reliable alternative to GFS(Game File Scan). Output size difference in this tool will be exactly what you get when using xtool on game data with same settings. It is also more stable than GFS. I posted it once somewhere, now I decided to update script and post it in its own thread. Supported formats are same as: zlib/deflate, lz4, lzo, zstd, crilayla as well as unreal and unity engines.

Usage(for 'other codecs'):
Run .bat file, type one of available codecs(z-xxx is based on ztool, x-xxx is based on xtool), enter extra parameters or skip with ENTER, enter file or directory path to scan, wait for finish, compare size differences.
Remember you can use file masks to scan only specific files, eg: "C:\gamefolder\*.pak"

EDIT2:
New update. Include uelr(Unreal decmp tool) and AFR(Anvil forge decmp tool). These however write temporarily to disk within lz_scan directory(therefore consider where to place it). AFR only need input file as output is written into NULL, because it display all necessary info on its own. Uelr need both input and output tmp file for lz_scan to be able to measure size differences. Temp files are automatically deleted.

EDIT3:
Updated to include latest version of xtool(v0.4.4). This version contain additional batch scripts you can use to test lz4, zstd and oodle on all levels, methods and libraries included.
For example:

test_lz4.bat file.ext

In case of oodle, all methods supported by xtool(kraken, selkie, mermaid, lzna, hydra and leviathan) will be tested on all included libraries.
In case of zstd, all libs will be tested as well.
In case of lz4, again all libs will be tested in both standard and lz4hc mode.

Additional libraries can be copied and added to batch script. Included ones are all that were available on github in compiled dll form(for both lz4 and zstd).
In fact, with these codecs this is now the recommended method to find out compression, default .bat is mainly for zlib/deflate/crilayla or specific engines.

EDIT4:
New and rewamped version! Lot of changes. This version make it easier to figure out compression with support of deep scan based on xtool's 'verbose' command and default scanner. Contain ALL available zstd and lz4 libraries(I compiled missing ones myself) from github to this day! Readme.txt will keep track of version and info, I decided to reset this to v1.0 despite previously releasing higher number for many reasons.

EDIT5:
New version 1.1, fixes x-lzo in cli script and add 'deepscan' script for it.

lz_scan_v1.1.7z

[elit]

Replace start.bat with this one, its better.

[Removed = obsolete.]

[elit]

You start with "start.bat". Then:
procedure.png
- First you type one of the available methods, like I did "pzlib:high" for this game file as example.
- Then path to file or directory as shown(use quotes for paths with spaces, also tab completition works just like in shell here so you dont have to type everything).
- Then you wait and when done you compare 2 size differences as shown in picture example. If second number is bigger it mean inflation worked for this. If same or too close you either got wrong method, game is not compressed or is not supported by ztool.
- You can also see overall time it took in seconds as below.
- Finally you can start all over again or quit.

[ZakirAhmad]

Can we mofify it for afr, uelr etc.

[elit]

Originally Posted by ZakirAhmad

Can we mofify it for afr, uelr etc.

I want to include them as well, when I have more time - in next version. This will also make it more universal not just for ztool.

[ZakirAhmad]

Originally Posted by elit

I want to include them as well, when I have more time - in next version. This will also make it more universal not just for ztool.

this would be great step.

[elit]

Unfortunately I found that neither uelr nor afr support stdio/pipelining, which without further coding is impossible to implement AFAIK.
EDIT: I am working on it already, it will be done but will use disk writes.

[elit]

New version released. Include uelr, AFR and script enhancements. Don't say I never loved you .

[ZakirAhmad]

Good job bro.

[elit]

Fixed non recursing subdirectories on mask. New file upped.
EDIT: There is still problem with parsing, will update soon.
EDIT2: Done. Hopefully now for good.
EDIT3: Turned out 7z already have cleaner solution with -r0 option. Upped again.

[KaktoR]

It would be good if you could add output.txt which lists all files contain streams.

[elit]

Originally Posted by KaktoR

It would be good if you could add output.txt which lists all files contain streams.

You dont need this tool for that, you can simply "for /r %i in ("*.*") do (ztool ... "%i" 1>> file.log) or something like that

To know which file have stream individually I would have to process them one by one and compare output. That would be too much for a simple shell script that use tar to process all in one stream. Also slower.

You can use file mask to check files with specific extension only, or check single file as that mean all others with same extension will be processable.

[elit]

New version 2.0 in the first post. Added x-tool 0.12 support. I have not tested oodle and crilayla and although it should work, let me know if it doesn't.

[IgorKolesnik]

can I check the file immediately for all possible encryption? or each method separately only?

[elit]

Originally Posted by IgorKolesnik

can I check the file immediately for all possible encryption? or each method separately only?

This tool scan for compression, not encryption. If you mean several types of compression at once, no. You select single compression and see if it inflate. You can do 2 things though:
1, test single smaller file known to be likely compressed first to save time
2, run several scripts at once in parallel, each doing its own specific cmp test