Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #31  
Old 09-01-2021, 22:19
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,465
Thanks: 1,899
Thanked 9,487 Times in 2,065 Posts
Razor12911 is on a distinguished road
You do realize that almost each time lz4 is updated on github it compresses data a bit differently? This "bit differently" is why xtool from the beginning allows users to change the library however they want so they pick whichever one gives the best result. When it comes to lz4, there is no universal dll for all games so if you want to add support, you must tell the user to always add the libraries that come with plugins to the DiskSpan_GUI folder.

The lz4 library I ship in xtool is v1.8.3 which is the commonly used library, some games use an older and some use a newer but nonetheless if the streams cannot be restored correctly, xdelta is applied.

If the user uses the incorrect liblz4.dll, xtool tries to use xdelta on the streams and if the user chose the wrong one or does not use the one that comes with a plugin, you end up with negative ratio. Where using a precompressor just gives worse result.

This is seen here, although I am not sure how this situation in particular happened...

Edit: The same is true with zstd, each update causes xtool to use xdelta to try to salvage some streams. Zlib would also be using the same method of xdelta but other libraries such as reflate and preflate exist that handle streams that zlib can't and when it comes to the situation of lz4, there is no such tool.
__________________
Reduce pollution, recycle projects

Last edited by Razor12911; 09-01-2021 at 22:24.
Reply With Quote
The Following 4 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (09-01-2021), Ele (10-01-2021), KaktoR (10-01-2021), parnexcodes (10-01-2021)
Sponsored Links
  #32  
Old 10-01-2021, 10:11
Cesar82's Avatar
Cesar82 Cesar82 is online now
Registered User
 
Join Date: May 2011
Location: Brazil
Posts: 606
Thanks: 761
Thanked 1,037 Times in 363 Posts
Cesar82 is on a distinguished road
@Razor12911, could you tell me if the FIFA plugin (EA_Frostbite_Engine) is not yet functional?
It only has a link to FIFA21, but the example was a little confusing and has several filesand a larger XTool (Maybe it was rewritten especially for this game).
I didn't understand where this part of the method came from? -mfrostbite3+zstd.dict:l19
If the plugin is ready, it could post an example with the other previous fifa using XTool 0.31?
How is it determined which section [Stream#] will be used for each game? In frostbite3.ini has 3 commented sections that are for FIFA games
If you want, let me know that I remove my posts from this topic of yours so that I don't have irrelevant content.

I tried to do some tests here, but it seems that XTool has an AI. Even with a lack of plugin libraries, XTool compresses too (no errors were shown), and the extraction was done normally. (It seems that for lack of DLL files, XTool changes the method to avoid errors).
You're the guy, congratulations on the great job.
Reply With Quote
  #33  
Old 10-01-2021, 12:16
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,465
Thanks: 1,899
Thanked 9,487 Times in 2,065 Posts
Razor12911 is on a distinguished road
FIFA is a special case like Far Cry Primal, Watch Dogs 2/Legion, they encrypted the game and used two algorithms, kraken/leviathan + zstd, the zstd used has dictionary. So even before we begin. You have to use FIFA 19 Cas Decryption Tool and then use xtool (I think for FIFA 19 and FIFA 20). As for the method "zstd.dict:l19", it's more or less the same as libdunia situation where I added a temporary codec internally and then remove it, however the xtool with the temporary codec can still be found here

I added these methods because of the scene, I thought they have made a breakthrough in bypassing the latest iteration of Denuvo so I added several methods quickly but as you can see right now, these games are still uncracked so I remove the methods and then slowly put them back via plugin support now that I have made xtool accept them.

Quote:
Originally Posted by Cesar82 View Post
@Razor12911I tried to do some tests here, but it seems that XTool has an AI. Even with a lack of plugin libraries, XTool compresses too (no errors were shown), and the extraction was done normally. (It seems that for lack of DLL files, XTool changes the method to avoid errors).
You're the guy, congratulations on the great job.
Of course checks are made to see if a library can be initialized or not. I'm not sure if you checked the source code included in Unreal Engine plugin

Capture2.PNG

Here the plugin checks if a user has given a key and necessary parameters and if nothing is given then using -mue4 then the plugin will be unavailable.

If for example you are just trying to use frostbite3.ini, it has 5 different streams to be detected, if you have liblz4.dll, xtool will search for all 5 different streams but when it tries to add the streams that were found and finds that some libraries are missing, these streams are discarded so that xtool continues working and not raise an error.

Edit:

Code:
[Stream1]
// DAInquisition
Name=zlib
Codec=zlib:l98
BigEndian=1
Signature=0x0270
Structure=Dict(1),DSize(3),Signature(2),CSize(2),ZLibHdr(2),Stream
StreamOffset=0
CompressedSize=CSize - 6
DecompressedSize=DSize
Condition1=DSize > CSize
Condition2=CSize > 64
Condition3=CSize <= 262144
Condition4=Dict=0
Condition5=ZLibHdr = 0x78DA

[Stream2]
// BF4-BFV, NFS2015-NFSHeat, MECatalyst
Name=lz4
Codec=lz4
BigEndian=1
Signature=0x0970
Structure=Dict(1),DSize(3),Signature(2),CSize(2),Stream
StreamOffset=0
CompressedSize=CSize
DecompressedSize=DSize
Condition1=DSize > CSize
Condition2=CSize > 64
Condition3=CSize <= 262144
Condition4=Dict = 0

[Stream3]
// MEAndromeda 
Name=zstd
Codec=zstd:l19
BigEndian=1
Signature=0x0F70
Structure=Dict(1),DSize(3),Signature(2),CSize(2),ZStdHdr(4),Stream
StreamOffset=-4
CompressedSize=CSize
DecompressedSize=DSize
Condition1=DSize > CSize
Condition2=CSize > 64
Condition3=CSize <= 262144
Condition4=Dict = 0
Condition5=ZStdHdr = 0x28B52FFD

[Stream4]
// FIFA 18-20
Name=kraken
Codec=kraken:l8:t128
BigEndian=1
Signature=0x1970
Structure=Dict(1),DSize(3),Signature(2),CSize(2),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize
Condition1=DSize > CSize
Condition2=CSize > 64
Condition3=CSize <= 262144
Condition4=Dict = 0
Condition5=OodleHdr = 0x8C06

[Stream5]
// FIFA 21
Name=leviathan
Codec=leviathan:l8:t128
BigEndian=1
Signature=0x1970
Structure=Dict(1),DSize(3),Signature(2),CSize(2),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize
Condition1=DSize > CSize
Condition2=CSize > 64
Condition3=CSize <= 262144
Condition4=Dict = 0
Condition5=OodleHdr = 0x8C0C
Alternatively, when using -mfrostbite3 method, you can use it this way -mfrostbite3:zstd, xtool will check what Streams are called "zstd" as they have names, Stream3 is the one that is called zstd, so it will only search using information from Stream3. Same as -mfrostbite3:zstd:zlib, where Stream1 and Stream3 are the ones that are searched. However if no parameter is given, -mfrostbite3, all 5 streams are searched.
__________________
Reduce pollution, recycle projects

Last edited by Razor12911; 10-01-2021 at 12:34.
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (10-01-2021), KaktoR (10-01-2021)
  #34  
Old 10-01-2021, 12:40
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,465
Thanks: 1,899
Thanked 9,487 Times in 2,065 Posts
Razor12911 is on a distinguished road
To properly explain how xtool behaves. The plugins I post here should be considered "external" plugins, there are also plugins in xtool called "internal" plugins. These are just common compressors used in games, zlib, lz4, lzo and etc. Since some internal plugins have issues detecting streams by themselves, lz4 as an example. What normally happens is, the external plugins I post here give information to the internal plugins. Telling them exactly how the streams are found and where to look for them. So if you use a plugin like frostbite3 and don't add zlibwapi.dll, liblz4.dll and etc. Basically no libraries whatsoever. When the external plugins give information to the internal ones. They will refuse to accept streams or any of the information as they didn't initialize because of missing libraries therefore no error is given however the ratio is affected. You can try it now yourself, use any of the plugins without the required libraries, xtool will just give a result of 100% ratio and no error. Encoding is never a problem, the problem starts when you are decoding, if you used something when encoding but it's missing when decoding. XTool will give you an error telling you a certain method was not found.

Edit:

This is what I told this person here, to run xtool without stdio so that I can see the error message that xtool gives.

Here's an example, I used zlib method when encoding and then I removed zlibwapi.dll and then tried to decode.
Attached Images
File Type: png Capture2.PNG (13.0 KB, 103 views)
__________________
Reduce pollution, recycle projects

Last edited by Razor12911; 10-01-2021 at 12:46.
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (10-01-2021), ffmla (10-01-2021), KaktoR (10-01-2021)
  #35  
Old 10-01-2021, 13:38
Cesar82's Avatar
Cesar82 Cesar82 is online now
Registered User
 
Join Date: May 2011
Location: Brazil
Posts: 606
Thanks: 761
Thanked 1,037 Times in 363 Posts
Cesar82 is on a distinguished road
@Razor12911, now I understand about the sections [Stream#], the method must be used the value obtained in the key Name=.
The link to the temporary codec is not accessible.
I haven't looked at the Unreal Engine source code (I'll see).
Quote:
run xtool without stdio.
I don't know if only I don't understand. You could explain that.
Check your inbox.
Thanks!

Last edited by Cesar82; 10-01-2021 at 13:47.
Reply With Quote
  #36  
Old 13-01-2021, 11:31
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,465
Thanks: 1,899
Thanked 9,487 Times in 2,065 Posts
Razor12911 is on a distinguished road
Plugins Updated

Changes

- Updated The Crew 2 to support latest update
- Updated Far Cry Primal, Watch Dogs 2 and Watch Dogs Legion
__________________
Reduce pollution, recycle projects
Reply With Quote
The Following 4 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (13-01-2021), DiCaPrIo (13-01-2021), ffmla (14-01-2021), Masquerade (13-01-2021)
  #37  
Old 13-01-2021, 17:17
patricktech patricktech is offline
Registered User
 
Join Date: Jan 2021
Location: Indonesia
Posts: 1
Thanks: 0
Thanked 0 Times in 0 Posts
patricktech is on a distinguished road
Nice
Reply With Quote
  #38  
Old 13-01-2021, 21:04
HunterEmrah HunterEmrah is offline
Registered User
 
Join Date: Jan 2013
Location: Canada
Posts: 46
Thanks: 26
Thanked 15 Times in 14 Posts
HunterEmrah is on a distinguished road
Quote:
Originally Posted by Snake288 View Post
Cyberpunk 2077 about compression please, I want to get help.
i have the same problem when compressing cyberpunk 2077 with cb2077 plugin, it gives error in random progress percent (1.2%, 1.5%, 1.7%)

Code:
F:\Games\Cyberpunk 2077\pack\pak>arc a -s; -dsgerpn -lc- -di -hppassword -cfg"C:\Program Files (x86)\FreeArc\bin\xtool2020\arc.ini" -wF:\Temp -mxtool+srep+lzma2 ..\data1.bin * -r
FreeArc 0.67 (March 15 2014) Creating archive: ..\data1.bin using xtool+srep+lzma2
Memory for compression 0b, decompression 0b, cache 1mb
Compressing 16 files, 57,137,917,952 bytes. Processed   1.7%
ERROR: write error (disk full?) in compression algorithm xtool
arc.ini
Code:
[External compressor:xtool]
header    = 0
packcmd   = XTool.exe precomp -mcp2077 -c128mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = XTool.exe decode -t100p - - <stdin> <stdout>

[External compressor:srep]
header = 0
packcmd   = srep -m3f -a1 {options} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = srep -d -mem2048mb - - <stdin> <stdout>

[External compressor:lzma2]
header = 0
packcmd = 7z_fa a -txz -an -mcrc=0 -mx9 -m0=lzma2:a=1:d=350m:mf=bt4:fb=273:mc=4294967295:lc=4:lp=0:pb=2 -mmt=3 -si -so <stdin> <stdout>
unpackcmd = 7z_fa x -txz -an -y -si -so <stdin> <stdout>
There is no problem when compressing files separately (tried only these files: basegame_4_animation.archive, lang_en_voice.archive, lang_ru_voice.archive), but I get this error when I compress all ".archive" files at the same time (except what I mentioned before).

also tried without <stdin> <stdout> but it's stuck on %100 after arc.exe compressing files in to one archive.

Last edited by HunterEmrah; 13-01-2021 at 21:08.
Reply With Quote
  #39  
Old 13-01-2021, 22:44
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,465
Thanks: 1,899
Thanked 9,487 Times in 2,065 Posts
Razor12911 is on a distinguished road
Code:
[External compressor:xtool]
header    = 0
packcmd   = XTool.exe precomp -mcp2077 -c128mb -t100p --dbase $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = XTool.exe decode -t100p - - <stdin> <stdout>
Try using these settings and what are your system specs?

Also if there are errors that xtool reports, I want to see the error message.
__________________
Reduce pollution, recycle projects

Last edited by Razor12911; 13-01-2021 at 22:53.
Reply With Quote
  #40  
Old 14-01-2021, 00:30
hwang4 hwang4 is offline
Registered User
 
Join Date: Sep 2007
Location: Algeria
Posts: 65
Thanks: 30
Thanked 9 Times in 3 Posts
hwang4 is on a distinguished road
HunterEmrah
"i have the same problem when compressing cyberpunk 2077 with cb2077 plugin, it gives error in random progress percent (1.2%, 1.5%, 1.7%)"


Decreasing "thread usage" may help ! (t90p worked for me)!
Reply With Quote
  #41  
Old 14-01-2021, 10:58
HunterEmrah HunterEmrah is offline
Registered User
 
Join Date: Jan 2013
Location: Canada
Posts: 46
Thanks: 26
Thanked 15 Times in 14 Posts
HunterEmrah is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
what are your system specs?
i5-10500 6c12t, 16 GB DDR4 3200MHz, HDD. ok will try later and share result. thanks!
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
XTool 2020 (Main Project) Razor12911 Conversion Tutorials 287 Today 09:18
XTool - Successor of ZTool Razor12911 Conversion Tutorials 570 10-12-2020 23:21
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 07:26
XTool 2019 (Plugins) Razor12911 Conversion Tutorials 50 16-05-2020 07:14



All times are GMT -7. The time now is 11:36.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com