Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #136  
Old 22-06-2018, 02:47
Mini Mini is offline
Registered User
 
Join Date: Nov 2013
Location: russia
Posts: 119
Thanks: 166
Thanked 76 Times in 43 Posts
Mini is on a distinguished road
usa_world.dat
Overall size: 304237021 >> 824626021

works!
Reply With Quote
Sponsored Links
  #137  
Old 22-06-2018, 14:32
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,943 Times in 2,140 Posts
Razor12911 is on a distinguished road
I figured out what they used with the spare break I had, they used the Hydra codec if I'm not mistaken. I didn't add Hydra support in xtool but I will in future.
Reply With Quote
The Following 7 Users Say Thank You to Razor12911 For This Useful Post:
COPyCAT (15-07-2018), EzzEldin16 (28-06-2018), ffmla (23-06-2018), Jiva newstone (23-06-2018), Mini (22-06-2018), shazzla (28-06-2018), ShivShubh (24-11-2019)
  #138  
Old 23-06-2018, 22:26
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,943 Times in 2,140 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Mini View Post
usa_world.dat
Overall size: 304237021 >> 824626021

works!
Could you rerun test, I want to see if correct and improved information is found. I played around with the scanner a bit.

@everyone
Participate if possible, try testing this with several games that use oodle, leave out Project CARS 2 out for now since that game is encrypted.
Attached Files
File Type: 7z odet2.7z (614.3 KB, 76 views)
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
Shateiel (12-07-2018), ShivShubh (24-11-2019), Simorq (24-06-2018)
  #139  
Old 23-06-2018, 23:56
Mini Mini is offline
Registered User
 
Join Date: Nov 2013
Location: russia
Posts: 119
Thanks: 166
Thanked 76 Times in 43 Posts
Mini is on a distinguished road
usa_world.dat
Overall size: 314783065 >> 844382444
Reply With Quote
  #140  
Old 28-06-2018, 04:35
Edison007's Avatar
Edison007 Edison007 is offline
Registered User
 
Join Date: Mar 2014
Location: EU
Posts: 20
Thanks: 0
Thanked 40 Times in 13 Posts
Edison007 is on a distinguished road
Quote:
Originally Posted by Mini View Post
The Crew 2 - Open Beta
xtool not working

files https://www64.zippyshare.com/v/dbhy7vY9/file.html
I had time, and i added support for this files.
Also I added parsing FAT from watch_dogs, but without (de)compression yet.
It takes time to deal with the xcompress-library.
Attached Files
File Type: 7z DERec_005.7z (357.7 KB, 85 views)
Reply With Quote
The Following 8 Users Say Thank You to Edison007 For This Useful Post:
COPyCAT (15-07-2018), dixen (29-06-2018), doofoo24 (28-06-2018), ffmla (28-06-2018), Jiva newstone (28-06-2018), Mini (28-06-2018), ShivShubh (28-06-2018), Simorq (28-06-2018)
  #141  
Old 11-07-2018, 20:03
ZakirAhmad ZakirAhmad is offline
Registered User
 
Join Date: Sep 2015
Location: Kmr
Posts: 290
Thanks: 56
Thanked 54 Times in 37 Posts
ZakirAhmad is on a distinguished road
@Razor: is xtool going to b updated.
Reply With Quote
  #142  
Old 20-05-2020, 14:46
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,943 Times in 2,140 Posts
Razor12911 is on a distinguished road
Hi guys, long time

I haven't coded in a while so you can imagine:
96806729_3811479972226955_3580484423860092928_n.jpg

I came up with an idea before I went hiatus which was deduplication of streams in xtool. Basically xtool permutes or disregards then removes repeated streams to improve performance. It's the same stuff that srep does only, xtool does it before srep so it improves IO and lessens memory usage.

What I want you guys to do is check out if this thing improves ratio if any and if decompression memory of srep goes down significantly, example. For examples like GTAV and lastly if there are no issues, I did rework the source a bit to make it stable.

My test input is No Man's Sky with the Beyond update (seems like devs have done a good job making sure they don't repeat game resources, wasn't the results I was not expecting...)

Code:
xtool 2019:
Compressed 104 files, 9,610,947,737 => 22,079,473,953 bytes. Ratio 229.73%
Compression time: cpu 12.20 sec/real 559.10 sec = 2%. Speed 17.19 mB/s

srep393b:
100%: 22,079,475,924 -> 14,306,151,312: 64.79%.  Cpu 123 mb/s (171.078 sec), real 126 mb/s (167.671 sec) = 102%.  Remains 00:00
Decompression memory is 699 mb.  6,413,440 matches = 102,615,040 bytes = 0.72% of file

lzturbo:
Compressed 1 file, 14,306,163,456 => 7,573,816,911 bytes. Ratio 52.94%
Compression time: cpu 16.19 sec/real 255.92 sec = 6%. Speed 55.90 mB/s
Code:
xtool 2020:
Compressed 104 files, 9,610,947,737 => 18,987,442,809 bytes. Ratio 197.56%
Compression time: cpu 10.94 sec/real 483.97 sec = 2%. Speed 19.86 mB/s

srep393b:
100%: 18,987,444,781 -> 14,276,551,933: 75.19%.  Cpu 113 mb/s (160.000 sec), real 114 mb/s (158.234 sec) = 101%.  Remains 00:00
Decompression memory is 436 mb.  4,696,348 matches = 75,141,568 bytes = 0.53% of file

lzturbo:
Compressed 1 file, 14,276,560,829 => 7,565,632,525 bytes. Ratio 52.99%
Compression time: cpu 16.86 sec/real 250.04 sec = 7%. Speed 57.10 mB/s
What do these results mean? and why is xtool 2020 producing less output compared to 2019 but somehow after using srep, the output is better?

Well as usual, I always have ideas to try to improve the xtool project for even more speed. I reviewed the project from all possible angles and found that optimisations of the program are nearing its limits meaning I cannot make the program any faster.

However then came an idea to steal some of the work srep does and make xtool involved, this resulted in the idea of making xtool detect and permute repeated streams, as a result. 2020 xtool most of the time will produce less output but when you compare it to 2019's version after using srep, the output is roughly the same or better.

You might ask, what is the point of taking some of srep's work and making it for xtool. Well this is because games over the years get endless updates, example is No Man's Sky and GTAV so there is a high chance that there are several repeated streams and as a result, if xtool doesn't take the job, it will be a job for xtool to deal with when installing a game because srep will require more decompression memory and xtool will process more streams resulting in extended installation times.

Note: I was expecting srep to produce the same result from both 2019 and 2020, because the main idea here is to improve installation times.

TL;DR

Xtool 2020 will produce less output compared to xtool 2019 because it takes some of the work of srep to reduce memory usage when installing a game while also improving speed significantly due to less streams being processed and less IO. (Theoretically at least)

Overall test results (compression):
8.95 GB (original) > 20.5 GB (xtool 2019 / 9 mins 19 secs) > 13.3 GB (srep m3f / 699mb dec memory) > 7.05 GB (lzturbo 32 b128)
8.95 GB (original) > 17.6 GB (xtool 2020 / 8 mins 4 secs) > 13.2 GB (srep m3f / 436mb dec memory) > 7.04 GB (lzturbo 32 b128)

I used "lzturbo 32" to speed up the final compression results (I'm a bit busy), but you will get the idea when you run tests yourselves.

Decompression mode is unavailable, I still have to write the code for it.

If you don't have GTAV or any other game which has live updates or a game with a lot of repetitions, you can just make a copy of the same input then give it to xtool to see what happens.

These optimisations are making me feel like Intel right now... 14nm+++++++++++++++++++++++
Attached Files
File Type: 7z xtool_dedup_test.7z (2.22 MB, 100 views)

Last edited by Razor12911; 20-05-2020 at 14:52.
Reply With Quote
The Following 13 Users Say Thank You to Razor12911 For This Useful Post:
78372 (23-05-2020), BLACKFIRE69 (23-05-2020), Cesar82 (20-05-2020), devil777 (20-05-2020), DiCaPrIo (21-05-2020), dixen (20-05-2020), doofoo24 (20-05-2020), ffmla (20-05-2020), Gehrman (21-05-2020), giothothan (31-07-2020), Gupta (20-05-2020), L0v3craft (21-05-2020), rambohazard (22-05-2020)
  #143  
Old 20-05-2020, 17:25
Cesar82's Avatar
Cesar82 Cesar82 is offline
Registered User
 
Join Date: May 2011
Location: Brazil
Posts: 728
Thanks: 1,167
Thanked 1,529 Times in 527 Posts
Cesar82 is on a distinguished road
@Razor12911
Thanks for the great job.
After completing this if you have time (also if possible), do a functional calback for your cls-diskspan library.
There are also some bugs with this library, using compressions formed by 3 methods.
Thank you!
Reply With Quote
The Following User Says Thank You to Cesar82 For This Useful Post:
Gehrman (22-05-2020)
  #144  
Old 20-05-2020, 18:48
doofoo24 doofoo24 is offline
Registered User
 
Join Date: Nov 2016
Location: canada
Posts: 408
Thanks: 138
Thanked 456 Times in 227 Posts
doofoo24 is on a distinguished road
so i guess only game with zlib for now maybe game like mad max will see improvement last time i compressed the game it was with xtool+srep:m2f+lolz memory usage was way less than srep:m3f but the final game size was 300-400mb more but game install was much more faster...
if i remember correctly m3f dec mem was around 17gb but with m2f around 3-4gb...
Reply With Quote
  #145  
Old 20-05-2020, 23:40
devil777's Avatar
devil777 devil777 is offline
Registered User
 
Join Date: May 2019
Location: Let.it.be
Posts: 106
Thanks: 280
Thanked 65 Times in 48 Posts
devil777 is on a distinguished road
Smile

To be... honest... desiring a Xtool version... which will work like Pzlib v3 4ever


Thanks razor for all the works
Reply With Quote
  #146  
Old 21-05-2020, 23:53
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,943 Times in 2,140 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Cesar82 View Post
@Razor12911
Thanks for the great job.
After completing this if you have time (also if possible), do a functional calback for your cls-diskspan library.
There are also some bugs with this library, using compressions formed by 3 methods.
Thank you!
Noted
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (22-05-2020), Gehrman (22-05-2020)
  #147  
Old 23-05-2020, 06:17
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,943 Times in 2,140 Posts
Razor12911 is on a distinguished road
Ok I decided to get Need For Speed The Run because I remembered that the game required a lot of decompression memory for srep to best demonstrate what I want to do with this deduplication idea.

I was surprised by the results

Xtool 2019:
Code:
Compressed 120 files, 14,503,034,446 => 29,120,915,487 bytes. Ratio 200.79%
Compression time: cpu 17.25 sec/real 807.42 sec = 2%. Speed 17.96 mB/s

100%: 29,120,917,667 -> 8,068,516,819: 27.71%.  Cpu 223 mb/s (124.672 sec), real 225 mb/s (123.527 sec) = 101%.  Remains 00:00
Decompression memory is 5660 mb.  2,922,729 matches = 46,763,664 bytes = 0.58% of file

Compressed 1 file, 8,068,559,139 => 4,315,048,356 bytes. Ratio 53.48%
Compression time: cpu 10.02 sec/real 172.58 sec = 6%. Speed 46.75 mB/s
Xtool 2020:
Code:
Compressed 120 files, 14,503,034,446 => 12,121,343,984 bytes. Ratio 83.58%
Compression time: cpu 16.13 sec/real 368.86 sec = 4%. Speed 39.32 mB/s

100%: 12,121,346,164 -> 8,010,462,480: 66.09%.  Cpu 141 mb/s (81.719 sec), real 145 mb/s (79.890 sec) = 102%.  Remains 00:00
Decompression memory is 759 mb.  2,105,722 matches = 33,691,552 bytes = 0.42% of file

Compressed 1 file, 8,010,469,968 => 4,296,282,176 bytes. Ratio 53.63%
Compression time: cpu 9.88 sec/real 159.45 sec = 6%. Speed 50.24 mB/s

Overall test results (compression):
13.5 GB (original) > 27.1 GB (xtool 2019 / 13 mins 27 secs) > 7.51 GB (srep m3f / 5660mb dec memory) > 4.02 GB (lzturbo 32 b128)
13.5 GB (original) > 11.2 GB (xtool 2020 / 6 mins 9 secs) > 7.45 GB (srep m3f / 759mb dec memory) > 4.00 GB (lzturbo 32 b128)

Not only is xtool 2020 faster but it produced a slightly better result while reducing srep's decompression memory. That 13.5 GB > 11.2 GB is not a mistake, xtool did precompress the data but there were too many repeated streams in the input and it removed a lot of them which resulted in negative ratio.

Notes: Xtool does incur additional decoding memory but it will definitely not be 5.6GB, it will far less. Maybe 2GB but if I apply proper memory management, I can reduce it to 1GB saving an additional 3GB of ram for lolz or something else.
Reply With Quote
The Following 8 Users Say Thank You to Razor12911 For This Useful Post:
78372 (23-05-2020), devil777 (23-05-2020), DiCaPrIo (23-05-2020), dixen (23-05-2020), ffmla (23-05-2020), Gehrman (23-05-2020), Masquerade (23-05-2020), shazzla (23-05-2020)
  #148  
Old 23-05-2020, 08:13
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 648
Thanks: 772
Thanked 926 Times in 307 Posts
78372 is on a distinguished road
Code:
FreeArc 0.67 (March 15 2014) creating archive: data19.arc
Compressed 1 file, 197,759,488 => 941,714,574 bytes. Ratio 476.19%
Decompression memory is 116 mb.  146,130 matches = 2,338,080 bytes = 0.71% of file
Code:
FreeArc 0.67 (March 15 2014) creating archive: data20.arc
Compressed 1 file, 197,759,488 => 734,656,804 bytes. Ratio 371.49%
Decompression memory is 43 mb.  136,880 matches = 2,190,080 bytes = 0.66% of file
Good work
Attached Images
File Type: png test.png (20.9 KB, 110 views)
__________________
NOT AVAILABLE
Reply With Quote
The Following 4 Users Say Thank You to 78372 For This Useful Post:
BLACKFIRE69 (23-05-2020), ffmla (23-05-2020), Gehrman (23-05-2020), Razor12911 (23-05-2020)
  #149  
Old 23-05-2020, 17:36
Cesar82's Avatar
Cesar82 Cesar82 is offline
Registered User
 
Join Date: May 2011
Location: Brazil
Posts: 728
Thanks: 1,167
Thanked 1,529 Times in 527 Posts
Cesar82 is on a distinguished road
@Razor12911
Good work! Great results!

I don't understand practically anything about compression and flow methods, etc., but CIU has XTool (old Xtool, not 2019) and ZTool, and as far as I know some of the flows (I think that's it) are not supported by Xtool.
I would like to know if in this version when it is finished it will support all flows that old ZTool and Xtool support, or it will always be necessary to keep XTool + ZTool in CIU if I use XTool 2020.
It would be great to be able to remove some of the compressors from CIU.
Thank you!
Reply With Quote
The Following User Says Thank You to Cesar82 For This Useful Post:
Gehrman (24-05-2020)
  #150  
Old 23-05-2020, 20:54
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 188
Thanks: 16
Thanked 225 Times in 121 Posts
FitGirl is on a distinguished road
Thanks for returning to the project, deduplication is a very useful feature.
I have an idea which will reduce the required RAM for dedup. You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen, cause reads will be rare and the RAM won't be used that much. 1-2 GB is a pretty big amount even for machines with 8 GB. And for users with 4 GB installation will be almost impossible, considering srep and lolz/lzma. Even with page file. So reduction/control over used RAM is a must, I think.

I'd recommend you Halo Reach for testing dedup, it has tons of duplicate streams of a different size.

Last edited by FitGirl; 23-05-2020 at 20:56.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump



All times are GMT -7. The time now is 05:24.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com