Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #466  
Old 10-03-2022, 19:19
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 244
Thanks: 176
Thanked 306 Times in 106 Posts
elit is on a distinguished road
Razor I think you mentioned in the past that -mzlib is able to detect deflate? I recall mentioning back then that it did not work for me. So here I am attaching set of png images that I just accidentally tested and did not work. Both -mreflate as well as previous ztool:high does work.

pngs.zip
Reply With Quote
Sponsored Links
  #467  
Old 10-03-2022, 23:48
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,636
Thanks: 2,013
Thanked 10,343 Times in 2,202 Posts
Razor12911 is on a distinguished road
-mzlib can detect deflate streams, but similarly to my previous post regarding leviathan, leviathan streams are detected however detection and being able to process them are two different things.

Code:
XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 000000000000A046 (41031)
[0] Actual zlib stream found at 0000000000000053 (40928 >> 87680)

[0] Processing streams on block 0000000000000000 to 000000000000A046 (41031)
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l11,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l12,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l13,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l14,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l15,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l16,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l17,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l18,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l19,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l21,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l22,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l23,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l24,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l25,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l26,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l27,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l28,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l29,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l31,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l32,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l33,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l34,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l35,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l36,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l37,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l38,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l39,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l41,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l42,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l43,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l44,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l45,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l46,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l47,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l48,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l49,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l51,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l52,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l53,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l54,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l55,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l56,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l57,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l58,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l59,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l61,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l62,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l63,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l64,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l65,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l66,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l67,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l68,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l69,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l71,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l72,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l73,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l74,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l75,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 3072) using l76,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l77,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l78,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l79,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l81,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l82,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l83,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l84,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l85,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l86,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l87,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l88,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l89,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l91,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l92,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l93,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l94,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l95,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l96,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l97,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l98,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l99,w15 has failed

Streams: 0/1
Time: 00:00:00 (00:00:00)
Memory: 128 MB (128 MB)
I tried to precompress one of the png files and as you can see, the method -mzlib was able to detect the deflate stream but then it failed to process it.
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
Gehrman (11-03-2022)
  #468  
Old 11-03-2022, 01:00
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,583
Thanks: 942
Thanked 5,847 Times in 2,159 Posts
KaktoR is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
Update available

Changes

- removed leviathan codec restriction

Notes

Xtool can "actually" detect leviathan streams however the only way it can process them is if they are divisible by the block size used by the new oodle compressors which is 262144, if a stream decompressed size cannot be divided by this value leaving no remainder then no way of predicting the stream size (at least that I know of) is possible hence why plugins are the only ones that were allowed to use the leviathan codec.

Code:
-mleviathan
If you are not satisfied with the results produced then find a way to make a plugin for proper support for that particular game.
It's better then nothing I guess, especially if you're not familiar with creating plugins, so thanks for the update

Code:
Streams: 354/415
Time: 00:01:18 (00:01:17)
Memory: 423 MB (423 MB)

64,1mb >> 352mb
__________________
Haters gonna hate
Reply With Quote
  #469  
Old 11-03-2022, 02:50
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 746
Thanks: 168
Thanked 789 Times in 393 Posts
Masquerade is on a distinguished road
Thanks Razor, this will be good for when the generator is working.
Reply With Quote
The Following User Says Thank You to Masquerade For This Useful Post:
Gehrman (11-03-2022)
  #470  
Old 11-03-2022, 05:18
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 244
Thanks: 176
Thanked 306 Times in 106 Posts
elit is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
-mzlib can detect deflate streams, but similarly to my previous post regarding leviathan, leviathan streams are detected however detection and being able to process them are two different things.
Thanks, so on more technical ground what is to be expected going forward regarding this? Specifically why is old reflate able to work without issues and mzlib not, is it matter of time/rework or is this a implementation limitation that is to be expected from mzlib, as compare to reflate and will not change in future?

Last edited by elit; 11-03-2022 at 06:06.
Reply With Quote
  #471  
Old 16-03-2022, 20:31
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 244
Thanks: 176
Thanked 306 Times in 106 Posts
elit is on a distinguished road
I cannot inflate zst(zstd) file from Paper Mario Origami King with xtool. With zstd.exe it does unpack just fine:

zstdproblem.png

I was so desperate that I actually downloaded and compiled ALL available zstd versions from github! I am attaching those dll's, zstd.exe and a .zst file itself:
zstdproblem.7z

PS(could it be due to window size - 48k ?)
Reply With Quote
  #472  
Old 16-03-2022, 20:38
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,636
Thanks: 2,013
Thanked 10,343 Times in 2,202 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Gehrman View Post
Sorry for late response but here's an example.

in the generate folder, you'll get the example of how database would be created.

Code:
xtool.exe generate -mre4lfs "game\*" "game\*" lfs.xtl
You'll notice that the unpacked data and the original data is the same, that's because this tool accepts game files as they are and doesn't need to extract additional streams.

Note: Chunk size is should be the same size as the largest stream/file

then in precompress folder is where you need to transfer the generated database file from generate folder to precompress folder (lfs.xtl)

Look in xtool.ini to see how the program should be configured based on the information you provided.

Code:
[re4lfs]
Encode=re4lfs.exe <filein>.lfs <fileout>.pack
Decode=re4lfs.exe <filein>.pack <fileout>.lfs
Results of the samples you provided
Code:
Compressed 5 files, 2,853,066 => 6,511,766 bytes. Ratio 228.24%
Compression time: cpu 0.02 sec/real 1.06 sec = 1%. Speed 2.70 mB/s
Attached Files
File Type: 7z example.7z (3.23 MB, 38 views)
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
Gehrman (17-03-2022)
  #473  
Old 16-03-2022, 21:11
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,636
Thanks: 2,013
Thanked 10,343 Times in 2,202 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by elit View Post
Thanks, so on more technical ground what is to be expected going forward regarding this? Specifically why is old reflate able to work without issues and mzlib not, is it matter of time/rework or is this a implementation limitation that is to be expected from mzlib, as compare to reflate and will not change in future?
there are several implementations of the deflate algorithm, it's very flexible. Think of using lzma to compress something, obviously there are compression levels that it comes with 1-9. -mzlib implementation is based around this idea that a user who uses this would have used a common setting such as compression level so for precompression it will decompress the data and then try to recompress it using these common levels, but of course lzma has methods that people use for other purposes like for stronger compression such as "lzma:ultra:a1:mfbt4:d158m: fb273:mc1000000000:lc8" mzlib cannot process such, similarly to deflate streams in png images as the way deflate set in such a way that it's good compressing pixels of an image but also make it quicker for loading/viewing. Reflate and preflate have no problem processing these as they used an advanced method for precompressing them.

You might say that mzlib is therefore useless, actually it still has its uses. mzlib is faster than both reflate and preflate because of its simple implementation and incurring no overhead furthermore mzlib produces better results because there is no additional header information file produced hence why I've preached several times to mix both zlib with reflate or preflate, if zlib fails to process the stream it just passes the stream to reflate/preflate for it to process it.

mzlib:
Compressed 1 file, 86,347,072 => 444,743,922 bytes. Ratio 515.07%
Compression time: cpu 0.16 sec/real 13.37 sec = 1%. Speed 6.46 mB/s

mreflate,l9:
Compressed 1 file, 86,347,072 => 445,179,589 bytes. Ratio 515.57%
Compression time: cpu 0.14 sec/real 14.33 sec = 1%. Speed 6.03 mB/s

mpreflate:
Compressed 1 file, 86,347,072 => 366,023,129 bytes. Ratio 423.90%
Compression time: cpu 0.14 sec/real 10.25 sec = 1%. Speed 8.42 mB/s

mzlib+reflate,l9:
Compressed 1 file, 86,347,072 => 444,941,384 bytes. Ratio 515.29%
Compression time: cpu 0.06 sec/real 11.74 sec = 1%. Speed 7.35 mB/s

mzlib+preflate:
Compressed 1 file, 86,347,072 => 444,744,743 bytes. Ratio 515.07%
Compression time: cpu 0.11 sec/real 11.23 sec = 1%. Speed 7.69 mB/s

As you can see, zlib perform faster when alone but better when you hook it up with reflate/preflate and preflate fails to process some streams. I've given people options, all up to them how they use xtool.

Quote:
Originally Posted by elit View Post
I cannot inflate zst(zstd) file from Paper Mario Origami King with xtool. With zstd.exe it does unpack just fine:

Attachment 31395

I was so desperate that I actually downloaded and compiled ALL available zstd versions from github! I am attaching those dll's, zstd.exe and a .zst file itself:
Attachment 31394

PS(could it be due to window size - 48k ?)
use --verbose mode to see which levels are producing the closest result to the original stream.
Code:
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 17101) using l1 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 16535) using l2 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 16351) using l3 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15784) using l4 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15711) using l5 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15493) using l6 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15462) using l7 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15462) using l8 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15444) using l9 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15442) using l10 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15438) using l11 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15437) using l12 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15428) using l13 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14817) using l14 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14733) using l15 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14689) using l16 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l17 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l18 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l19 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l20 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l21 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l22 has failed
level 17 is close but level 19 is commonly used, then set -mzstd,l19
Code:
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l19 has failed
[0] - Patched stream at 0000000000000000 (14692 >> 14688) [28] successfully
Code:
Compressed 1 file, 14,692 => 48,239 bytes. Ratio 328.34%
Compression time: cpu 0.00 sec/real 0.55 sec = 0%. Speed 0.03 mB/s
Used libzstd130.dll

Last edited by Razor12911; 16-03-2022 at 21:24.
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
Gehrman (17-03-2022), ScOOt3r (17-03-2022)
  #474  
Old 17-03-2022, 03:25
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 244
Thanks: 176
Thanked 306 Times in 106 Posts
elit is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
level 17 is close but level 19 is commonly used, then set -mzstd,l19
Thank you Razor! But wait, why it did not tried that level automatically? Is xtool trying only some levels here as well? Like up to 9, maybe due to speed?

If that is the case, may I suggest that it try to do all of them - up to certain mb of data(user defined). For instance first 100-200mb all levels and if nothing found drop to only speedier levels as is now etc.?
Reply With Quote
  #475  
Old 17-03-2022, 04:24
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,636
Thanks: 2,013
Thanked 10,343 Times in 2,202 Posts
Razor12911 is on a distinguished road
Dude it tried all levels, look at the log via verbose (level 1-22) and none of them produced crc perfect stream and you can even see that with level 19 the original stream was 14692 which was decompressed to 48079 and recompressed to 14688.

Xtool applies patches using xdelta only if you have specified what level was used else for zstd as an example which has 22 levels, each stream would have to be recompressed 22 times and 22 patches will exist and then how would xtool decide what level was used, by looking at which one produced the smaller diff file? because if you look at the log again level 17, 18, 19, 20, 21 and 22, yep 6 different levels produced the same result so how would xtool decide what the best level is? Even if you do it for the first 100-200mb, you may even find that zstd compression on some data even taps out at level 10 and any level higher than this will produce the same result. Xtool will still be confused af as to which level is the best which is why I left it to the user.

Xtool in such a scenario will just trust the user specified level and only help out if any of the streams fail to be properly processed by applying xdelta otherwise it will not apply any patches.

Edit: Verbose mode was added just so that the user can see why xtool fails and if there are patterns, like for example this stream "14692 >> 48079 >> 14688", with 4 bytes difference, I assume that all the streams in this game will show a pattern of 4 bytes also missing, which can mean when the game was compressed the developers turned on checksum mode for zstd, meaning a hash sized 4 byte was added for all the streams just to verify if the data is not corrupt in anyway and xtool uses zstd in default mode meaning checksum option is disabled hence why they produce 4 bytes less.

Last edited by Razor12911; 17-03-2022 at 04:30.
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
elit (17-03-2022), Gehrman (17-03-2022), ScOOt3r (17-03-2022)
  #476  
Old 17-03-2022, 05:27
Gehrman's Avatar
Gehrman Gehrman is offline
Registered User
 
Join Date: Jan 2020
Location: Save Palestine
Posts: 36
Thanks: 725
Thanked 41 Times in 16 Posts
Gehrman is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
Sorry for late response but here's an example.

in the generate folder, you'll get the example of how database would be created.

Code:
xtool.exe generate -mre4lfs "game\*" "game\*" lfs.xtl
You'll notice that the unpacked data and the original data is the same, that's because this tool accepts game files as they are and doesn't need to extract additional streams.

Note: Chunk size is should be the same size as the largest stream/file

then in precompress folder is where you need to transfer the generated database file from generate folder to precompress folder (lfs.xtl)

Look in xtool.ini to see how the program should be configured based on the information you provided.

Code:
[re4lfs]
Encode=re4lfs.exe <filein>.lfs <fileout>.pack
Decode=re4lfs.exe <filein>.pack <fileout>.lfs
Results of the samples you provided
Code:
Compressed 5 files, 2,853,066 => 6,511,766 bytes. Ratio 228.24%
Compression time: cpu 0.02 sec/real 1.06 sec = 1%. Speed 2.70 mB/s
Thank Razor12911
But strangely, it does not work with files larger than 15.9 MB.
this sample file
Attached Files
File Type: 7z 0e11c100.pack.lfs.7z (16.02 MB, 14 views)
Reply With Quote
  #477  
Old 17-03-2022, 05:45
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,636
Thanks: 2,013
Thanked 10,343 Times in 2,202 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
Note: Chunk size is should be the same size as the largest stream/file
generate -c##
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
Gehrman (17-03-2022)
  #478  
Old 17-03-2022, 06:51
Gehrman's Avatar
Gehrman Gehrman is offline
Registered User
 
Join Date: Jan 2020
Location: Save Palestine
Posts: 36
Thanks: 725
Thanked 41 Times in 16 Posts
Gehrman is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
generate -c##
good work
Code:
@echo off
xtool.exe generate -mre4lfs -c512mb "game\*" "game\*" lfs.xtl 
pause
Thank you for this wonderful tool
Reply With Quote
  #479  
Old 17-03-2022, 06:59
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 244
Thanks: 176
Thanked 306 Times in 106 Posts
elit is on a distinguished road
@Razor now I understand, thank you! Its clear that internal knowledge of xtool is still important. All make sense now although it also means my "lz scan" cannot be a simple fire & forget proof for all corner cases(well it could but I feel it would be too much to go with each level separately within the script + all the lib versions etc., maybe compiled binary with proper output report would do). It is fine though, as long as users are aware of this.

-------------------------------------------------------------------------------------------------------------------------------

Unrelated to above, I want to give one advice to all users regarding zstd(possibly lz4 as well). For best results it may be important to try all lib versions even if you found one working. During my switch games repacking I found for example that nearby versions and sometimes far versions may get better result(or pickup again) and by results not only I mean size but also speed. ZSTD have a habit to slow down to crawl(~1mb/s) when used on wrong lib version or can have as much as 2-3x speed difference on nearby versions where all of them can inflate very close to!

I dont have data to show you anymore, but just to imagine it was something like this. Lets say we have a random file of 8Mb, then:
Code:
libzstd130.dll > 10000k > 6min
libzstd131.dll > 13000k > 5min
libzstd132.dll > 16000k > 3min
libzstd133.dll > 16400k > 1min
libzstd134.dll > 16500k > 20s
libzstd135.dll > 16550k > 15s
libzstd136.dll > 8000k > 4min (back to original size)
...                           (here all following versions did nothing)
libzstd144.dll > 16200k > 1min
libzstd145.dll > 16620k > 13s
libzstd146.dll > 8700k > 4min
This is just example, not exact but yes something like that actually happened.

Last edited by elit; 17-03-2022 at 09:55.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
Razor12911 (17-03-2022)
  #480  
Old 17-03-2022, 18:24
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 244
Thanks: 176
Thanked 306 Times in 106 Posts
elit is on a distinguished road
Maybe I found a way to pinpoint zstd level. After you scan with 'verbose' you can see there are several levels same. However, using ':l17' to ':l22'(without 'verbose') separately further reveal which level is real. Only with l19 it shows this:
foundzstdlevel.png
All other levels remain '0/1' even if they show same repacked size in '--verbose' scan. After this it's only matter of figuring out lib version. In this case versions working were 114, 120, 130 and 131. Probably on bigger files time would start showing difference, but I have seen these giving same result before(but also not always) so once they match, any of them should be safe.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
ScOOt3r (18-03-2022)
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 15:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 18:16



All times are GMT -7. The time now is 08:33.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com