Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #151  
Old 23-05-2020, 23:40
Gupta Gupta is offline
Registered User
 
Join Date: Aug 2016
Location: India
Posts: 396
Thanks: 139
Thanked 701 Times in 228 Posts
Gupta is on a distinguished road
Quote:
You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen,
maybe he can introduce the second phase in compression, then he can store forward reference count for a stream that should ideally decrease requirements memory size beyond window size and that should increase compression too.

HDDs are very slow, I recently upgraded to nvme based storage and I can feel the speed.
__________________
XD

Last edited by Gupta; 23-05-2020 at 23:46.
Reply With Quote
Sponsored Links
  #152  
Old 28-05-2020, 12:48
panker1992's Avatar
panker1992 panker1992 is offline
Registered User
 
Join Date: Oct 2015
Location: Always Somewhere
Posts: 515
Thanks: 112
Thanked 739 Times in 289 Posts
panker1992 is on a distinguished road
Dedup

Quote:
Originally Posted by FitGirl View Post
Thanks for returning to the project, deduplication is a very useful feature.
I have an idea which will reduce the required RAM for dedup. You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen, cause reads will be rare and the RAM won't be used that much. 1-2 GB is a pretty big amount even for machines with 8 GB. And for users with 4 GB installation will be almost impossible, considering srep and lolz/lzma. Even with page file. So reduction/control over used RAM is a must, I think.

I'd recommend you Halo Reach for testing dedup, it has tons of duplicate streams of a different size.
there is also a sorting match feature that can reduce ram needed and that is as follows.

srep does a very good job finding matches that are located far away!
that in order to happen makes a dictionary!
IF you sort the files you feed srep you can actually reduce ram needed and its speed

Sorting preprocession can speedup the process and cost less ram !! and remove IO overhead because NO temps
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows
My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45.
Reply With Quote
  #153  
Old 29-05-2020, 02:23
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,943 Times in 2,140 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by FitGirl View Post
Thanks for returning to the project, deduplication is a very useful feature.
I have an idea which will reduce the required RAM for dedup. You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen, cause reads will be rare and the RAM won't be used that much. 1-2 GB is a pretty big amount even for machines with 8 GB. And for users with 4 GB installation will be almost impossible, considering srep and lolz/lzma. Even with page file. So reduction/control over used RAM is a must, I think.

I'd recommend you Halo Reach for testing dedup, it has tons of duplicate streams of a different size.
Quote:
Originally Posted by Gupta View Post
maybe he can introduce the second phase in compression, then he can store forward reference count for a stream that should ideally decrease requirements memory size beyond window size and that should increase compression too.


HDDs are very slow, I recently upgraded to nvme based storage and I can feel the speed.
Quote:
Originally Posted by panker1992 View Post
there is also a sorting match feature that can reduce ram needed and that is as follows.

srep does a very good job finding matches that are located far away!
that in order to happen makes a dictionary!
IF you sort the files you feed srep you can actually reduce ram needed and its speed

Sorting preprocession can speedup the process and cost less ram !! and remove IO overhead because NO temps

Believe me I have several ideas of how to reduce memory usage before even relying on virtual memory. Optimisation is my middle name.
Reply With Quote
The Following 8 Users Say Thank You to Razor12911 For This Useful Post:
78372 (29-05-2020), BLACKFIRE69 (29-05-2020), devil777 (30-05-2020), DiCaPrIo (29-05-2020), ffmla (29-05-2020), Gupta (29-05-2020), L0v3craft (29-05-2020), shazzla (29-05-2020)
  #154  
Old 29-05-2020, 10:13
bunti_o4u's Avatar
bunti_o4u bunti_o4u is offline
Registered User
 
Join Date: Aug 2017
Location: India
Posts: 157
Thanks: 29
Thanked 157 Times in 60 Posts
bunti_o4u is on a distinguished road
Quote:
Originally Posted by Edison007 View Post
I had time, and i added support for this files.
Also I added parsing FAT from watch_dogs, but without (de)compression yet.
It takes time to deal with the xcompress-library.
does it support stdio?

if you add stdio, it would be great..
Reply With Quote
  #155  
Old 29-05-2020, 13:30
panker1992's Avatar
panker1992 panker1992 is offline
Registered User
 
Join Date: Oct 2015
Location: Always Somewhere
Posts: 515
Thanks: 112
Thanked 739 Times in 289 Posts
panker1992 is on a distinguished road
xcompress if pure windows 10 compression, and i think it supports it by default.
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows
My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45.
Reply With Quote
  #156  
Old 10-07-2020, 17:06
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,943 Times in 2,140 Posts
Razor12911 is on a distinguished road
Can you guys test this. I added preflate as an alternative of reflate in case if you are having issues with crc errors when you use reflate or prolonged precompression times such as this:
Quote:
Originally Posted by dixen View Post
Dishonored: Death of the Outsider

*.resources - 17 gb > 25.1 gb for 23 minutes on 4 threads

For example. ZTool or Xtool (v0.12) - 30% for 4 hours

Thanks BLACKFIRE69

test unpack
I'm no expert when it comes to C++ and I did my best to compile a library for xtool to use so it may have bugs.

I ran a benchmark and here are the results

benchmark results on "game1.resources" from "Dishonored Death of the Outsiders"
xtool_2019:
Code:
8.30 GB >> ?? >> ?? >> ??
precompression time using 4 threads = it's better to watch paint dry (takes hours)
xtool_2020 (WIP):
Code:
8.30 GB >> 11.3 GB >> 8.74 GB >> 6.24 GB
precompression time using 2 threads = 8 minutes, 48 seconds
precompression time using 4 threads = 5 minutes, 7 seconds
PrecompX:
Code:
8.30 GB >> 11.1 GB >> 8.79 GB >> 6.28 GB
precompression time using 4 threads = 8 minutes, 29 seconds
Attached Files
File Type: 7z xtool_20_preflate_test.7z (2.18 MB, 60 views)
Reply With Quote
The Following 9 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (11-07-2020), DiCaPrIo (11-07-2020), dixen (12-07-2020), ffmla (12-07-2020), Gehrman (17-07-2020), L0v3craft (11-07-2020), Masquerade (11-07-2020), PsYcHo_RaGE (11-07-2020), shazzla (11-07-2020)
  #157  
Old 12-07-2020, 14:13
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 255
Thanks: 304
Thanked 292 Times in 135 Posts
dixen is on a distinguished road
Dishonored Death of the Outsiders

game1.resources

Quote:
Compressed 1 file, 8,920,763,754 => 11,774,785,195 bytes. Ratio 131.99%
Compression time: cpu 11.31 sec/real 385.94 sec = 3%. Speed 23.11 mB/s
All OK
When unpack function?)
Reply With Quote
  #158  
Old 13-07-2020, 22:01
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,943 Times in 2,140 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by dixen View Post
Dishonored Death of the Outsiders

game1.resources



When unpack function?)
Soon, maybe this week. Still busy writing the main code while looking for more ways to speed up processing
Attached Images
File Type: png 99.PNG (131.5 KB, 154 views)
Reply With Quote
The Following 7 Users Say Thank You to Razor12911 For This Useful Post:
DiCaPrIo (14-07-2020), dixen (14-07-2020), doofoo24 (15-07-2020), ffmla (14-07-2020), L0v3craft (14-07-2020), lolaya (14-07-2020), shazzla (13-07-2020)
  #159  
Old 14-07-2020, 07:34
lolaya's Avatar
lolaya lolaya is offline
Registered User
 
Join Date: Jul 2011
Location: still life
Posts: 127
Thanks: 142
Thanked 34 Times in 25 Posts
lolaya is on a distinguished road
what is ztool and xtool and lolz?
Reply With Quote
  #160  
Old 14-07-2020, 09:42
Grumpy's Avatar
Grumpy Grumpy is offline
Moderator
 
Join Date: Jun 2004
Location: Australia
Posts: 5,607
Thanks: 1,099
Thanked 1,677 Times in 724 Posts
Grumpy is on a distinguished road
Quote:
Originally Posted by lolaya View Post
what is ztool and xtool and lolz?
That information is already available on these forums, search and you will find, you can not expect for others to always hold your hand and spoon feed you all the time.
__________________
Can't find a Game Conversion? Check the 'Conversion INDEX'
Reply With Quote
  #161  
Old 14-07-2020, 09:48
lolaya's Avatar
lolaya lolaya is offline
Registered User
 
Join Date: Jul 2011
Location: still life
Posts: 127
Thanks: 142
Thanked 34 Times in 25 Posts
lolaya is on a distinguished road
[email protected]

more posts here I can't find it easy
Reply With Quote
  #162  
Old 14-07-2020, 20:58
BLACKFIRE69's Avatar
BLACKFIRE69 BLACKFIRE69 is offline
Registered User
 
Join Date: Mar 2019
Location: In the Hell
Posts: 308
Thanks: 253
Thanked 871 Times in 210 Posts
BLACKFIRE69 is on a distinguished road
Quote:
Originally Posted by lolaya View Post
[email protected]

more posts here I can't find it easy
that's the why studying hard....
Reply With Quote
  #163  
Old 15-07-2020, 07:22
doofoo24 doofoo24 is offline
Registered User
 
Join Date: Nov 2016
Location: canada
Posts: 408
Thanks: 138
Thanked 456 Times in 227 Posts
doofoo24 is on a distinguished road
can't wait to test the new xtool
Reply With Quote
  #164  
Old 18-07-2020, 04:34
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,557
Thanks: 1,943
Thanked 9,943 Times in 2,140 Posts
Razor12911 is on a distinguished road
Here's an early working version of xtool. I'm still busy adding preflate, reflate and other things so currently only the zlib function works which means this will not work on Dishonored 2 or DOOM but it will work on most titles.

Drag and drop game files that are zlib compressed on the bat file

Things to test:

* Stability
* Can you cancel installation when xtool was used in setup
* If you have 16 threads or more, make xtool use 100% of your cpu to see if it can handle it
* if you have 32GB+ ram, set high chunk size, something like -c1gb and see if xtool was able to allocate the necessary memory
* If you have potato pc, don't worry I got a job for you check if there are no problems if you are using 1 thread both in encoding and decoding

Whatever you do, don't place preflate_dll.dll near the exe, not yet. else it will fail, the preflate code hasn't been added yet

Notes:

The file check must report
FC: no differences encountered
if the test went successfully, if it failed kindly upload the file it failed on.

This xtool uses slightly more memory than the 2019 version, I opted for stability than less memory usage

xtool now accepts maths expressions in command line, if you wanted it to use all cores minus 1, you can just write -t100p-1 or -t100-2, totally up to you

Next release/test focuses on:

+ preflate
+ reflate (reflate actually isn't replaced by preflate, it sometimes gives better output than preflate so I'll keep it just in case you are after best results)
+ depth setting
Attached Files
File Type: 7z xtool_zlib_test.7z (904.1 KB, 47 views)
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
ffmla (18-07-2020), PsYcHo_RaGE (18-07-2020), shazzla (18-07-2020)
  #165  
Old 18-07-2020, 06:14
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 539
Thanks: 115
Thanked 412 Times in 244 Posts
Masquerade is on a distinguished road
Hello Razor, here is my test (I did make a post before but I had to delete it since I used a reflate files by mistake):

My spec: Ryzen 7 2700 (8c/16t) + 16GB RAM.

Bat file editied for 128mb chunk size.

Here's is XTool using all 16 of my threads:



Testing on pak file from Astroneer:

Start size: 2.13GB

End Size: 3.75GB

No differences encountered

Not entirely sure how I'd go about testing it in a setup...
Reply With Quote
The Following 3 Users Say Thank You to Masquerade For This Useful Post:
dixen (18-07-2020), ffmla (18-07-2020), Razor12911 (18-07-2020)
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump



All times are GMT -7. The time now is 04:06.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com