Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #331  
Old 23-05-2021, 03:03
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,003
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
Update available

Changes

- added depth info functions
- added support for oodle 2.9.0+ functions
- fixed data patching bug
- updated oodle codec
- updated command line parser

Changes between 0.3.9 to 0.3.11

Code:
  ES_R11 (0.3.11)
  - fixed x86 build bugs
  - fixed config multi-threading bug
  - fixed resource management bug
  - fixed deduplication bug

  ES_R10 (0.3.10)
  - minor bug fixes
  - added diff tolerance parameter (--diff=)
  - fixed plugin database bug
  - fixed lz4 codec bug
  - updated oodle codec
  - updated library structure
  - added resource management
  - added direct use encryption codecs
  - added embedded deduplication feature (--dedup) [makes temps during encoding]
@Cesar82

I updated the command line parser

Code:
[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp { -moption} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>
If you use "-mxtool:zlib:ue4,m1,k0x115E..." via Freearc, this is passed to xtool as "-mzlib+ue4:m1:k0x115E..." which should allow you to use it in your project.

Last edited by Razor12911; 23-05-2021 at 03:53.
Reply With Quote
The Following 7 Users Say Thank You to Razor12911 For This Useful Post:
:( Sad8669 (23-05-2021), Cesar82 (23-05-2021), ffmla (24-05-2021), Flzzy (23-05-2021), Grumpy (24-05-2021), sajmon83 (04-06-2021), vint56 (26-05-2021)
Sponsored Links
  #332  
Old 23-05-2021, 03:17
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,003
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
@Everyone

The oodle precompressor in xtool is now more aggressive (means slower), I'll add aggression parameter to control how much time it should spend finding more and more streams however this only applies to kraken, mermaid, selkie and hydra. Leviathan is still problematic and you'll have to rely on plugins for data compressed using this codec.

Here are a few benchmarks that show what I am talking about:

0.99 GB (1,066,131,456 bytes) > 1.52 GB (1,638,079,260 bytes) [took 2 minutes, 9 seconds] (xtool 0.3.9)
0.99 GB (1,066,131,456 bytes) > 1.84 GB (1,977,644,853 bytes) [took 4 minutes, 56 seconds] (xtool 0.3.12)
0.99 GB (1,066,131,456 bytes) > 1.53 GB (1,649,308,007 bytes) [took 8 minutes, 52 seconds] (oo2reck)

As you can see, version 0.3.12 is now slower but it found more streams than 0.3.9 or the oodle precompressor side project.

This is a sample from Cyberpunk 2077 and since the game does have a plugin these are the results if you had use the plugin

0.99 GB (1,066,131,456 bytes) > 2.26 GB (2,432,780,012 bytes) [took 41 seconds] (plugin)

So what do these results mean? Well the newer version of xtool traded speed for more ratio in an attempt to beat the side project while still being faster. Also if a plugin for a specific game is created, it shows that the plugins will constantly be superior to the universal precompressor as it's not only faster but produces better results.

In conclusion, try to run tests with other games to see if the new xtool produces better results than oo2rec and if a plugin for a game exists then it's best to just use it.

Last edited by Razor12911; 23-05-2021 at 03:20.
Reply With Quote
The Following 4 Users Say Thank You to Razor12911 For This Useful Post:
:( Sad8669 (23-05-2021), Flzzy (23-05-2021), Grumpy (24-05-2021), pratikpatel8982 (23-05-2021)
  #333  
Old 23-05-2021, 03:24
:( Sad8669's Avatar
:( Sad8669 :( Sad8669 is offline
Registered User
 
Join Date: Oct 2020
Location: Triangulum Galaxy, Messier 33
Posts: 189
Thanks: 81
Thanked 146 Times in 88 Posts
:( Sad8669 is on a distinguished road
I am now even more fired up for FIFA.

Thanks for these updates.
__________________
The X who used to CODE, memories are bizzare thing indeed.
Reply With Quote
  #334  
Old 23-05-2021, 03:27
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,003
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
For FIFA and all these other frostbite games, 0.3.13 update is needed
Reply With Quote
  #335  
Old 23-05-2021, 03:33
:( Sad8669's Avatar
:( Sad8669 :( Sad8669 is offline
Registered User
 
Join Date: Oct 2020
Location: Triangulum Galaxy, Messier 33
Posts: 189
Thanks: 81
Thanked 146 Times in 88 Posts
:( Sad8669 is on a distinguished road
Crash Bandicoot 4 Comparison - For zlib streams

0.3.12
Code:
Compressed 1 file, 19,665,315 => 91,532,563 bytes. Ratio 465.45%
Compression time: cpu 0.05 sec/real 2.77 sec = 2%. Speed 7.11 mB/s
All OK
0.3.9
Code:
Compressed 1 file, 19,665,315 => 98,920,152 bytes. Ratio 503.02%
Compression time: cpu 0.06 sec/real 2.80 sec = 2%. Speed 7.03 mB/s
All OK

Those FIFA's after 19 are such a pain...
__________________
The X who used to CODE, memories are bizzare thing indeed.
Reply With Quote
  #336  
Old 23-05-2021, 03:38
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,003
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by :( Sad8669 View Post
Crash Bandicoot 4 Comparison - For zlib streams

0.3.12
Code:
Compressed 1 file, 19,665,315 => 91,532,563 bytes. Ratio 465.45%
Compression time: cpu 0.05 sec/real 2.77 sec = 2%. Speed 7.11 mB/s
All OK
0.3.9
Code:
Compressed 1 file, 19,665,315 => 98,920,152 bytes. Ratio 503.02%
Compression time: cpu 0.06 sec/real 2.80 sec = 2%. Speed 7.03 mB/s
All OK

Those FIFA's after 19 are such a pain...
must be deduplication, check command line

FIFA after 19 isn't a problem, I don't have samples for these games so I am limited by the little samples that I have.

Last edited by Razor12911; 23-05-2021 at 03:41.
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
Grumpy (24-05-2021)
  #337  
Old 23-05-2021, 04:05
Cesar82's Avatar
Cesar82 Cesar82 is online now
Registered User
 
Join Date: May 2011
Location: Brazil
Posts: 772
Thanks: 1,287
Thanked 1,639 Times in 558 Posts
Cesar82 is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post

@Cesar82
I updated the command line parser

Code:
[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp { -moption} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>
If you use "-mxtool:zlib+ue4,m1,k0x115E..." via Freearc, this is passed to xtool as "-mxtool:zlib+ue4:m1:k0x115E..." which should allow you to use it in your project.
@Razor12911, I don't know if another method besides ue4 has parameters. But supposing yes, how would I use a combination of 2 methods in which each one contains parameters?
Assuming that zlib had a parameter -m3 for example (just to specify the debt):

For example, assuming that zlib had a parameter -m3 (just to specify the debt) would it look like this?
Code:
"-mxtool:zlib,m3+ue4,m1,k0x115E..."
Should the arc.ini file be used in this way only { -moption}?
to use zlib+ue4 (Borderlands 3) it would not be necessary to send the -d1 parameter to XTool.
How can I pass or not the -d1 if I use the command like the arc.ini information reported above?
Reply With Quote
  #338  
Old 23-05-2021, 04:12
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,003
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
Code:
[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp { -moption} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>
Code:
-mxtool:zlib,m3:ue4,m1,k0x115E...
If you plan to add -d1 option via command line, you then have to modify the arc.ini to something like

Code:
[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp { -option} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>
Code:
-mxtool:d1:mzlib,m3:mue4,m1,k0x115E
which equates to -d1 -mzlib,m3 -mue4,m1,k0x115E
then -d1 -mzlib+ue4:m1:k0x115E when it reaches xtool
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (23-05-2021), Grumpy (24-05-2021)
  #339  
Old 23-05-2021, 05:06
:( Sad8669's Avatar
:( Sad8669 :( Sad8669 is offline
Registered User
 
Join Date: Oct 2020
Location: Triangulum Galaxy, Messier 33
Posts: 189
Thanks: 81
Thanked 146 Times in 88 Posts
:( Sad8669 is on a distinguished road
Might hook you up with some samples, once i get to testing it. If you would like.

Yes it was the dedup.

Without dedup :
Code:
Compressed 1 file, 19,665,315 => 98,946,941 bytes. Ratio 503.15%
Compression time: cpu 0.00 sec/real 4.12 sec = 0%. Speed 4.77 mB/s
All OK
__________________
The X who used to CODE, memories are bizzare thing indeed.
Reply With Quote
  #340  
Old 23-05-2021, 06:53
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,552
Thanks: 934
Thanked 5,796 Times in 2,138 Posts
KaktoR is on a distinguished road
Bethesda plugin does not work with xtool 0.3.12

Code:
0.3.9
Compressed 1 file, 373,427,426 => 532,225,912 bytes. Ratio 142.52%
Compression time: cpu 0.25 sec/real 4.74 sec = 5%. Speed 78.73 mB/s
All OK

0.3.12
Compressed 1 file, 373,427,426 => 373,427,610 bytes. Ratio 100.00%
Compression time: cpu 0.41 sec/real 2.22 sec = 18%. Speed 168.52 mB/s
All OK
Yes I have replaced bsa.dll with the new one.
__________________
Haters gonna hate
Reply With Quote
The Following User Says Thank You to KaktoR For This Useful Post:
Razor12911 (24-05-2021)
  #341  
Old 24-05-2021, 04:12
21h2 21h2 is offline
Registered User
 
Join Date: May 2021
Location: Earth
Posts: 2
Thanks: 0
Thanked 0 Times in 0 Posts
21h2 is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
XTool is program made specifically repackaging games by providing a full suite of useful features such as data precompression, archiving, encryption and etc.

With that being said, nothing restricts it from being used on everyday files such as documents, pictures and media but with few limitations.

Read the documentation to find out how it works and how to use it.

Link for older releases.
i still dont understand how i can zip thinks with it
Reply With Quote
  #342  
Old 24-05-2021, 14:08
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,003
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
Update available

Changes

- updated lz4 codec
- updated library structure
- updated depth info functions
- updated depth feature
Reply With Quote
The Following 13 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (24-05-2021), ffmla (24-05-2021), Flzzy (24-05-2021), Grumpy (24-05-2021), L0v3craft (24-05-2021), mausschieber (25-05-2021), Mortal Lord (24-05-2021), Perykles (25-05-2021), pratikpatel8982 (25-05-2021), PsYcHo_RaGE (05-06-2021), sajmon83 (25-05-2021), shazzla (24-05-2021), vint56 (26-05-2021)
  #343  
Old 28-05-2021, 20:23
kast1450 kast1450 is offline
Registered User
 
Join Date: May 2021
Location: USA
Posts: 4
Thanks: 1
Thanked 0 Times in 0 Posts
kast1450 is on a distinguished road
XTool LZ4:
8:20:52 PM - Overall input size: 6.54 GB
8:20:52 PM - Overall output size: 3.91 GB
8:20:52 PM - Overall conversion time: 00:00:52

XTool 2020 LZ4:
8:22:33 PM - Overall input size: 6.54 GB
8:22:33 PM - Overall output size: 6.54 GB
8:22:33 PM - Overall conversion time: 00:01:02

Is this normal or a bug? I'm using the same settings with the same dataset yet one LZ4 gives good compression and the other gives no compression.
Reply With Quote
  #344  
Old 03-06-2021, 07:07
ScOOt3r ScOOt3r is offline
Registered User
 
Join Date: Jun 2019
Location: Canada
Posts: 29
Thanks: 185
Thanked 14 Times in 13 Posts
ScOOt3r is on a distinguished road
If i wanted to use Oodle with xtool
would this be correct with the latest verison of xtool

[External compressor:xtool]
header = 0
packcmd = xtool\xtool.exe precomp -moodle -c256mb -t22 --dbase --dedup - - <stdin> <stdout>

cause it doesnt work for me... get a disc full error

thanks
ScOOt3r
Reply With Quote
  #345  
Old 03-06-2021, 07:08
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,003
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by kast1450 View Post
XTool LZ4:
8:20:52 PM - Overall input size: 6.54 GB
8:20:52 PM - Overall output size: 3.91 GB
8:20:52 PM - Overall conversion time: 00:00:52

XTool 2020 LZ4:
8:22:33 PM - Overall input size: 6.54 GB
8:22:33 PM - Overall output size: 6.54 GB
8:22:33 PM - Overall conversion time: 00:01:02

Is this normal or a bug? I'm using the same settings with the same dataset yet one LZ4 gives good compression and the other gives no compression.
xtool hasn't had a working lz4 codec.

Quote:
Originally Posted by ScOOt3r View Post
If i wanted to use Oodle with xtool
would this be correct with the latest verison of xtool

[External compressor:xtool]
header = 0
packcmd = xtool\xtool.exe precomp -moodle -c256mb -t22 --dbase --dedup - - <stdin> <stdout>

cause it doesnt work for me... get a disc full error

thanks
ScOOt3r
there is no codec with the name oodle however there is kraken, mermaid and selkie.

Edit: the parameter you used will require about 6GB ram if no streams are found and it will require even more if they are found

(256mb x 22 threads) + 256mb = 5,888MB ram usage and if the average inflation ratio is 200%, you're looking at 12GB+ ram usage by xtool. I'm letting you know before hand that the parameter you have input will make xtool use a lot of memory.

Last edited by Razor12911; 03-06-2021 at 07:12.
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (03-06-2021), sajmon83 (04-06-2021)
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 15:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 18:16



All times are GMT -7. The time now is 08:47.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com