Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #166  
Old 01-09-2020, 13:56
panker1992's Avatar
panker1992 panker1992 is offline
Registered User
 
Join Date: Oct 2015
Location: Always Somewhere
Posts: 527
Thanks: 113
Thanked 752 Times in 294 Posts
panker1992 is on a distinguished road
a brief Test was done


the files tested are Resident Evil 0 HD Remake PS3 all ARC files in ARC folder

tests was done with dedup activated.
Attached Images
File Type: png Screenshot_2020-09-01_23-49-13.png (351.4 KB, 502 views)
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows
My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45.
Reply With Quote
The Following User Says Thank You to panker1992 For This Useful Post:
Razor12911 (04-09-2020)
Sponsored Links
  #167  
Old 04-09-2020, 15:03
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,002
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
Update available

Changes

- added reflate forced verification
- updated deflate scanner
- fixed depthing issues
- fixed low memory mode issues
- fixed hanging issues when encoding
Reply With Quote
The Following 15 Users Say Thank You to Razor12911 For This Useful Post:
78372 (05-09-2020), Balaji007 (07-09-2020), COPyCAT (20-10-2020), DiCaPrIo (05-09-2020), dixen (05-09-2020), elit (08-09-2020), ffmla (05-09-2020), Gehrman (07-09-2020), kenzo34 (04-09-2020), L0v3craft (06-09-2020), Mortal Lord (05-09-2020), Pantsi (07-09-2020), PsYcHo_RaGE (06-09-2020), shazzla (04-09-2020), ZAZA4EVER (04-09-2020)
  #168  
Old 08-09-2020, 15:09
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,002
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
some benchmarks of the next version with oodle support

Oodle precompressor (Side project)
Code:
Compressed 1 file, 105,360,271 => 351,678,627 bytes. Ratio 333.79%
Compression time: cpu 0.13 sec/real 41.13 sec = 0%. Speed 2.56 mB/s
Xtool 2020
Code:
Compressed 1 file, 105,360,271 => 351,298,255 bytes. Ratio 333.43%
Compression time: cpu 0.08 sec/real 7.42 sec = 1%. Speed 14.20 mB/s
If you had problems with the oo2rec series or want features to be added, let me know.
Reply With Quote
The Following 11 Users Say Thank You to Razor12911 For This Useful Post:
78372 (08-09-2020), DiCaPrIo (09-09-2020), dixen (08-09-2020), ffmla (18-09-2020), Gehrman (08-09-2020), KaktoR (09-09-2020), L0v3craft (09-09-2020), Mortal Lord (09-09-2020), PsYcHo_RaGE (10-09-2020), shazzla (08-09-2020), ZAZA4EVER (10-09-2020)
  #169  
Old 09-09-2020, 12:54
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,551
Thanks: 932
Thanked 5,795 Times in 2,137 Posts
KaktoR is on a distinguished road
Can't wait to test and compare
__________________
Haters gonna hate
Reply With Quote
  #170  
Old 12-09-2020, 15:23
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,002
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
Update available

Changes

- added kraken codec
- fixed depthing issues

Notes

This doesn't detect kraken streams as the side project because I opted for speed but this will change in future once I figure out how to properly deal with the codec

Kraken comes with a level option which is used like this "-mkraken:l4" this is only if you know what level was used to speed up the process but if you don't know what level, you can just use kraken plainly like this "-mkraken" and xtool will try all possible levels until it gets the right one. If two levels were used then the method should look like this "-mkraken:l4,l5"

the deduplication applies to all codecs xtool has so if a game was compressed using kraken and it has many repeated streams, that should give you more speed.

the oodle in xtool doesn't require you to rename the dll therefore if "oo2core_9_win64.dll" ever gets released, xtool should be able to support it.

Last edited by Razor12911; 12-09-2020 at 15:26.
Reply With Quote
The Following 7 Users Say Thank You to Razor12911 For This Useful Post:
elit (17-09-2020), Gehrman (13-09-2020), L0v3craft (13-09-2020), Masquerade (13-09-2020), oo7 (07-12-2020), ScOOt3r (03-06-2021), shazzla (14-09-2020)
  #171  
Old 12-09-2020, 16:37
panker1992's Avatar
panker1992 panker1992 is offline
Registered User
 
Join Date: Oct 2015
Location: Always Somewhere
Posts: 527
Thanks: 113
Thanked 752 Times in 294 Posts
panker1992 is on a distinguished road
Well name a game that you need to be tested upon.

I think i will try sekiro first wish me luck
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows
My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45.
Reply With Quote
  #172  
Old 13-09-2020, 02:11
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 286
Thanks: 350
Thanked 327 Times in 148 Posts
dixen is on a distinguished road
DOOM Eternal

oo2reck

pack

Quote:
Compressed 2 files, 180,206,653 => 249,645,165 bytes. Ratio 138.53%
Compression time: cpu 0.28 sec/real 293.17 sec = 0%. Speed 0.61 mB/s
All OK
unpack

Quote:
Extracted 2 files, 249,645,165 => 180,206,653 bytes. Ratio 138.53%
Extraction time: cpu 0.27 sec/real 4.86 sec = 5%. Speed 37.10 mB/s
All OK
XTOOL 2009 r2

pack

Quote:
Compressed 2 files, 180,206,653 => 224,968,010 bytes. Ratio 124.84%
Compression time: cpu 0.38 sec/real 22.60 sec = 2%. Speed 9.07 mB/s
All OK
unpack

Quote:
Extracted 2 files, 224,968,010 => 180,206,653 bytes. Ratio 124.84%
Extraction time: cpu 0.47 sec/real 4.21 sec = 11%. Speed 49.24 mB/s
All OK
Compress with 7z archivator

oo2reck = 99.5 mb
xtool r2 = 99.8 mb

Compress with srep+lolz

oo2reck = 91.1 mb
xtool r2 = 93.9 mb


PS. Tested on my 2nd weak PC with FX-4100

Last edited by dixen; 14-09-2020 at 04:31.
Reply With Quote
The Following 4 Users Say Thank You to dixen For This Useful Post:
ffmla (17-09-2020), Gehrman (13-09-2020), L0v3craft (13-09-2020), Razor12911 (13-09-2020)
  #173  
Old 14-09-2020, 21:21
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,002
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
Update available

Changes

- documentation added

Notes

This release is the same as the one before, I just added a documentation so you know how to properly use this tool.

Last edited by Razor12911; 14-09-2020 at 21:33.
Reply With Quote
The Following 9 Users Say Thank You to Razor12911 For This Useful Post:
78372 (14-09-2020), elit (17-09-2020), ffmla (17-09-2020), Gehrman (15-09-2020), Grumpy (15-09-2020), L0v3craft (15-09-2020), Mortal Lord (15-09-2020), PsYcHo_RaGE (15-09-2020), shazzla (15-09-2020)
  #174  
Old 15-09-2020, 11:09
shazzla shazzla is offline
Registered User
 
Join Date: Nov 2010
Location: Hunnia
Posts: 221
Thanks: 408
Thanked 86 Times in 63 Posts
shazzla is on a distinguished road
@Razor12911 :
Very good job !

Just an idea : It would be nice to see how much data was "saved" during deduplication.
Maybe you add this feature later...
Thank you very much !
Reply With Quote
The Following User Says Thank You to shazzla For This Useful Post:
Razor12911 (16-09-2020)
  #175  
Old 16-09-2020, 13:53
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 286
Thanks: 350
Thanked 327 Times in 148 Posts
dixen is on a distinguished road
A little info

I try decompress all *.bff files on Project CARS 2 with XTOOL & oo2reck (both - with BDT)...and..

XTOOL R2 - 45 gb > 67 gb (for 30 min, but with many CRC errors on unpack)
oo2reck - 45 gb > 95 gb (for 3 hour. No error)
Reply With Quote
The Following User Says Thank You to dixen For This Useful Post:
Razor12911 (16-09-2020)
  #176  
Old 17-09-2020, 08:20
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Greetings Razor, I got few questions if you don't mind.

Regarding stream database, which hash type do you use? I am guessing that size of hash in bits will dictate speed benefit vs collisions # to data size. I think =>256bit should be good enough including for very big data(100+gb) or a lot of streams, regardless of occasional collisions and therefore should be enabled for speed benefits(if that's what you use)?

Thank you for documentation, this was needed. Please consider adding one to command line though. Help file works but gives script errors every time I change page(trying to access internet). I click ignore and it works but its a hassle. Command line help is also more practical, like was in ztool. It should also display version info at very least for user to know what (s)he have(well by (s)he I really mean all the time he + 1x FitGirl ).

How reliable is latest xtool now compared to ztool? I read some comments in the past where for instance it supposedly inflated only on individual files and not "tarred" ones etc.. Is it at least as reliable as ztool now? PS(I only use zlib/deflate in ztool anyway). I am interested in latest version, I know some stick to older versions of xtool.

I may be wrong but wasn't crilayla in previous xtool versions and is not anymore?

Isn't anything oodle gamepack-format dependent? I recall for API it needed to know both compressed and decompressed size in it's function parameters, meaning it won't be forever compatible if you stop updating the tool(unlike zlib/deflate)(and version differences aside)?

How many bytes are needed to store dedup data in separate file per single stream?

At least for zlib/deflate, is xtool endian neutral? Can it inflate both big and small endian byte streams?


Thank you
Reply With Quote
  #177  
Old 17-09-2020, 22:05
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,002
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
I have no idea what hash the dictionary uses but likely CRC-32 since it is 32-bit. But I don't think it's that much of an issue, I have yet to find a collision and I have run several tests, once I'm satisfied. I'll make the stream database a default feature.

I have not come across errors myself in the chm help file, I can always compile a pdf help file if there are issues. Perhaps the next release will come in different formats.

The reason the version information is not displayed is because this project is still at alpha stages, as you can see the tests dixen runs, there are still errors but eventually I'll commit to the idea of proper version history along with the program telling users what version it is. This is pretty much why the current version classification is written 2009_R3, where 20 is the year, 09 is the month and R3 means it's the 3rd release on that month, you just need to look at the exe dates to know what version you are using.

Xtool is reliable compared to ztool and should perform better, at least when using the zlib codec.

Crilayla was excluded from xtool because there is no room for slow tools in project plus xtool is now only available in 64-bit and the only crilayla dll around is 32-bit which is incompatible.

oodle is universal like zlib but there are issues with detecting the decompressed size which results in several streams being left behind and longer precompression times. I plan on improving this eventually, I will find a way to fix this.

About 4 bytes per repeated stream to store in dedup data.

endianness isn't something that considered when handling zlib/deflate streams. That's like saying, will a music player be able to play an mp3 player if it encoded using big endian, the standard is the same across all platforms.
Reply With Quote
The Following 5 Users Say Thank You to Razor12911 For This Useful Post:
78372 (18-09-2020), elit (18-09-2020), Harsh ojha (18-09-2020), Mortal Lord (18-09-2020), PsYcHo_RaGE (17-09-2020)
  #178  
Old 18-09-2020, 14:02
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Thanks. About that help error:
errhlp.png

About CRC, you may need to test that on big enough data (300gb+) and containing TONS of very small chunks(64-256k or ~128k) for it to be robust enough for all scenarios. Crc32 may reach iteration limit(collisions start way before that). Good idea is to compare srep m3 vs m5, m3 use VMAC(which is either 64bit or 128bit, dunno which use srep but very likely 128bit) and m5 is re-read bit perfect, so following attachment could help you get some hints regarding collision vs data:
srep-readme.zip

Thanks for that reliability reassuring, from now on I start use it exclusively instead ztool. Crilayla ditching is a disappointment though as this one is very common format in Japanese games and could also help with compressing console roms. You mentioned low speed as a reason but if I recall from past oodle was even slower?

~4 bytes in dedup per stream only?! So then you don't store distance, only single 32bit hash and nothing else me think. You compare with each newly found chunk if there is a crc match right?

About endians, ok but you still do have to search for a byte sequences in order to find something no? For example when I wrote bms script to decompress oodle chunks from xcom2 I searched for a oodle clues, in my case:
Code:
"\xC1\x83\x2A\x9E"
Then it matter if its that or in reverse. I dunno maybe zlib have same header ID order in all endians, but then what about deflate detection? That have no header, just a bytes test..
Reply With Quote
  #179  
Old 18-09-2020, 14:34
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 189
Thanks: 16
Thanked 227 Times in 122 Posts
FitGirl is on a distinguished road
I would never use crc32. In my repacking experience I've met three counts of FULL files crc32 match while having absolutely different content and sizes. Please use better/newer algos, otherwise there will be guaranteed collisions meaning corrupted data.
Reply With Quote
  #180  
Old 19-09-2020, 03:27
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Quote:
Originally Posted by FitGirl View Post
I would never use crc32. In my repacking experience I've met three counts of FULL files crc32 match while having absolutely different content and sizes. Please use better/newer algos, otherwise there will be guaranteed collisions meaning corrupted data.
Very different content and same hash.. that's quite something. This does depend on number of files, their size and especially polynomial of that crc though. Crc32's are multiple versions and polynomial of it is extremely important. I think there were some shitty variants that had low quality and only 2 that were really solid - one use Intel in their CPU's. With those crc32 should be perfectly fine and reliable but still only up to certain counts and data sizes - hence my worry since this tool is used for huge data and games are known to have both size and really big number of files inside their packs = tons of chunks.

As for collisions, he still cannot 100% rely on *any* hash so he have to verify by content before applying dedup regardless, otherwise its too risky. That mean occasional rare collision should not be a big deal to overall size, but also only if chunks are small. If you collide on multiple chunks of 10+mb or even 100+mb then you may get few hundred mb's worse compression.

If minimum chunk size is >= couple of kb's, few extra bytes of hash size should be negligible. I would suggest crc64 or sha128(or even better VMAC that srep use).
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 15:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 18:16



All times are GMT -7. The time now is 01:19.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com