Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #1  
Old 12-05-2016, 01:35
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,669
Thanks: 2,058
Thanked 10,572 Times in 2,227 Posts
Razor12911 is on a distinguished road
Fast Brute (test)

Like the title says, it's a test.

What needs testing here is comparison with what reflate does and/or what precomp does.

I noticed a bug after compiling that when you compress the output of brute, you get a very small output
Well it's a false output so it's just a waste of time if you want to see how much you could get if this
program as working. I was going to ask someone to try this program on Mad Max and maybe get size which
is less than 4GB in less than a day, unlike precomp (brute), this is faster though it achieves 0.2mb/s
on data without zlibs (tested on audio files) and can push up to 40mb/s or more on an input with
a lot of zlib streams (Well it is 0.2mb/s on my 9 year old Core 2 Extreme QX9650).

I already know that it doesn't work just as yet which is why I said just compare final size
with another zlib recompressor. (I edited zlib source to improve speed and that is what happened)
I get false data or repeated data making lzma compress the whole thing to a couple of kilos
and get 1% ratio which is false but working it though to be able to get correct data and make this work.


What to test in comparison with reflate:
The output size, since some people say "reflate" is the only recompressor that can detect "invalid" streams
and others can't, well you could try this program. Again, if an stream is invalid, its invalid, there is
nothing further you can do, but as I said detection and restoration are the only factors that separate these
zlib recompressors. (You know yourself)

You can also test on GTAV, I got expansion on one of the rpf files meaning it does work on GTAV as well.


What to test in comparison with precomp:
The output size and speed.

Files that you test on:
Make sure they do have zlib streams else you'll get 0.2mb/s. (brute scanning speed)

If you got Mad Max, run the program on the game, don't apply srep or lzma and those other things,
if you know what files reflate doesn't work on but precomp (brute) works, try running this.

A test on a benchmarking file that I always use to test everything is the UI.sb file and here are the results:
Input : 82.3MB
// Bigger is better
Brute (Test) : 421MB (Without verification)
Precomp + Reflate : 421MB (Evidence that reflate isn't really brute, it had to work with precomp to get that size)
Precomp (Brute) : 421MB (Probably, I had to cancel it, it takes long)
Precomp (Intense) : 373MB
Reflate (Gozarck) : 361MB
pZlib : 348MB

In terms of speed while working on a file (UI.sb):
Input : 82.3MB
Slowest to fastest:
pZlib : 348MB (6.39 seconds)
Brute (Test) : 421MB (47.70 seconds) //Quite slow but working on it.

Work of Razor12911 and Panker1992
Attached Images
File Type: png brute.PNG (43.7 KB, 474 views)
File Type: png brute2.PNG (16.5 KB, 472 views)
File Type: png brute3.PNG (17.3 KB, 469 views)
Attached Files
File Type: 7z brute_test2.7z (1.14 MB, 356 views)

Last edited by Razor12911; 12-05-2016 at 22:11.
Reply With Quote
The Following 14 Users Say Thank You to Razor12911 For This Useful Post:
1234567890123 (12-05-2016), Carldric Clement (12-05-2016), chum (22-05-2016), EzzEldin16 (05-10-2017), Jiva newstone (21-12-2017), kenzo34 (12-05-2016), Mini (12-05-2016), nasir ahmad (08-06-2016), panker1992 (12-05-2016), RamiroCruzo (12-05-2016), Sergey3695 (12-05-2016), Simorq (12-05-2016), Stor31 (12-05-2016), vint56 (12-05-2016)
Sponsored Links
  #2  
Old 12-05-2016, 02:12
Simorq's Avatar
Simorq Simorq is offline
Registered User
 
Join Date: Mar 2014
Location: Iran
Posts: 642
Thanks: 3,602
Thanked 1,296 Times in 464 Posts
Simorq is on a distinguished road
Talking

GTA V x64b.rpf
Input : 142 MB
Brute (Test) : 308 MB (2 min)
Brute+Srep+lzma:max: 55 MB (Monster Compressed )

Reflate u3 : 312 MB (10 min)
x64b.rpf (191MB) x64b.ref (359MB) x64b.srep (253MB) x64b.arc (162MB)

Brute (Test)


Reflate


Good

Last edited by Simorq; 12-05-2016 at 02:27.
Reply With Quote
The Following 2 Users Say Thank You to Simorq For This Useful Post:
1234567890123 (12-05-2016), Razor12911 (12-05-2016)
  #3  
Old 12-05-2016, 03:46
1234567890123 1234567890123 is offline
Registered User
 
Join Date: Aug 2014
Location: ankara
Posts: 92
Thanks: 124
Thanked 37 Times in 19 Posts
1234567890123 is on a distinguished road
simorg it's very good but which lzma code did you use on reflate u3? and which lzma method is the best lzma:max or another long method (using it in masked-ms).and what is the best compressor for after reflate? lzma nanozip fast paq8 (fp8) zpaq..
Reply With Quote
The Following User Says Thank You to 1234567890123 For This Useful Post:
Simorq (12-05-2016)
  #4  
Old 12-05-2016, 04:53
Mini Mini is offline
Registered User
 
Join Date: Nov 2013
Location: -
Posts: 125
Thanks: 166
Thanked 76 Times in 43 Posts
Mini is on a distinguished road
Mad Max

game24.arc - 812 mb

data.arc - 1.45 gb

time ~ 5 min
Reply With Quote
The Following 2 Users Say Thank You to Mini For This Useful Post:
Razor12911 (12-05-2016), Simorq (12-05-2016)
  #5  
Old 12-05-2016, 08:44
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,669
Thanks: 2,058
Thanked 10,572 Times in 2,227 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by Simorq View Post
GTA V x64b.rpf
Input : 142 MB
Brute (Test) : 308 MB (2 min)
Brute+Srep+lzma:max: 55 MB (Monster Compressed )

Reflate u3 : 312 MB (10 min)
x64b.rpf (191MB) x64b.ref (359MB) x64b.srep (253MB) x64b.arc (162MB)

Brute (Test)


Reflate


Good
Have to work on that 4mb difference
Like I said. The result you'll be getting are false after applying srep and lzma, bug caused after I played a little with zlib source so basically, the output size is the only thing that tells us it could work at some point.

Last edited by Razor12911; 12-05-2016 at 08:49.
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
Simorq (12-05-2016)
  #6  
Old 12-05-2016, 09:47
panker1992's Avatar
panker1992 panker1992 is offline
Registered User
 
Join Date: Oct 2015
Location: Always Somewhere
Posts: 534
Thanks: 113
Thanked 756 Times in 296 Posts
panker1992 is on a distinguished road
unfortunately i give exams, and i havent had the time to test myself

hope you guys give some feedback, and help along the line

make sure to test and provide any feedback or glitches this soft makes ... keep an eye on ram and IO...

thanks Zee
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows
My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45.
Reply With Quote
  #7  
Old 12-05-2016, 13:55
1234567890123 1234567890123 is offline
Registered User
 
Join Date: Aug 2014
Location: ankara
Posts: 92
Thanks: 124
Thanked 37 Times in 19 Posts
1234567890123 is on a distinguished road
Is it usable for decompression? i'm getting wclose: invalid argument (Bad file descriptor)
ERROR: CRC failed error.But veery good file size in (coh2 files) 2gb to 3gb.then srep lzma 146mb! i think something is wrong in brute.exe.with reflate 2gb to 3gb but then srep lzma 1100mb?

Last edited by 1234567890123; 12-05-2016 at 13:58.
Reply With Quote
  #8  
Old 12-05-2016, 14:08
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,669
Thanks: 2,058
Thanked 10,572 Times in 2,227 Posts
Razor12911 is on a distinguished road
I wrote up there
Quote:
don't apply srep or lzma and those other things
also

Quote:
I already know that it doesn't work just as yet which is why I said just compare final size
with another zlib recompressor. (I edited zlib source to improve speed and that is what happened)
I get false data or repeated data making lzma compress the whole thing to a couple of kilos
and get 1% ratio which is false but working it though to be able to get correct data and make this work.
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
Amsal (14-05-2016), Simorq (12-05-2016)
  #9  
Old 12-05-2016, 20:23
ChronoCross's Avatar
ChronoCross ChronoCross is offline
Registered User
 
Join Date: Sep 2014
Location: Acacia Dragons
Posts: 254
Thanks: 145
Thanked 200 Times in 115 Posts
ChronoCross is on a distinguished road
Really impresive reasult razor. I was made some test and the results are very close with reflate but brute test with 1 core is more faster than reflate and the results are very close. Also some files that precomp doesn't detect in faster mode is detected by brute and reflate too. Very nice. I request the decode! For more test.
Reply With Quote
  #10  
Old 12-05-2016, 21:35
Carldric Clement's Avatar
Carldric Clement Carldric Clement is offline
Registered User
 
Join Date: Aug 2014
Location: Toboh, Sabah, Malaysia
Posts: 558
Thanks: 567
Thanked 588 Times in 213 Posts
Carldric Clement is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
Like the title says, it's a test.

What needs testing here is comparison with what reflate does and/or what precomp does.

I noticed a bug after compiling that when you compress the output of brute, you get a very small output
Well it's a false output so it's just a waste of time if you want to see how much you could get if this
program as working. I was going to ask someone to try this program on Mad Max and maybe get size which
is less than 4GB in less than a day, unlike precomp (brute), this is faster though it achieves 0.2mb/s
on data without zlibs (tested on audio files) and can push up to 40mb/s or more on an input with
a lot of zlib streams (Well it is 0.2mb/s on my 9 year old Core 2 Extreme QX9650).

I already know that it doesn't work just as yet which is why I said just compare final size
with another zlib recompressor. (I edited zlib source to improve speed and that is what happened)
I get false data or repeated data making lzma compress the whole thing to a couple of kilos
and get 1% ratio which is false but working it though to be able to get correct data and make this work.


What to test in comparison with reflate:
The output size, since some people say "reflate" is the only recompressor that can detect "invalid" streams
and others can't, well you could try this program. Again, if an stream is invalid, its invalid, there is
nothing further you can do, but as I said detection and restoration are the only factors that separate these
zlib recompressors. (You know yourself)

You can also test on GTAV, I got expansion on one of the rpf files meaning it does work on GTAV as well.


What to test in comparison with precomp:
The output size and speed.

Files that you test on:
Make sure they do have zlib streams else you'll get 0.2mb/s. (brute scanning speed)

If you got Mad Max, run the program on the game, don't apply srep or lzma and those other things,
if you know what files reflate doesn't work on but precomp (brute) works, try running this.

A test on a benchmarking file that I always use to test everything is the UI.sb file and here are the results:
Input : 82.3MB
// Bigger is better
Brute (Test) : 421MB (Without verification)
Precomp + Reflate : 421MB (Evidence that reflate isn't really brute, it had to work with precomp to get that size)
Precomp (Brute) : 421MB (Probably, I had to cancel it, it takes long)
Precomp (Intense) : 373MB
Reflate (Gozarck) : 361MB
pZlib : 348MB

In terms of speed while working on a file (UI.sb):
Input : 82.3MB
Slowest to fastest:
pZlib : 348MB (6.39 seconds)
Brute (Test) : 421MB (47.70 seconds) //Quite slow but working on it.

Work of Razor12911 and Panker1992
Razor. if you build some 'cls_brute.dll' will work with ISDone for extract files.
Reply With Quote
  #11  
Old 12-05-2016, 22:13
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,669
Thanks: 2,058
Thanked 10,572 Times in 2,227 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by ChronoCross View Post
Really impresive reasult razor. I was made some test and the results are very close with reflate but brute test with 1 core is more faster than reflate and the results are very close. Also some files that precomp doesn't detect in faster mode is detected by brute and reflate too. Very nice. I request the decode! For more test.
Told ya, detection and restoration.

I still have to sort out restoration before I can make decode.
Reply With Quote
  #12  
Old 12-05-2016, 22:19
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,669
Thanks: 2,058
Thanked 10,572 Times in 2,227 Posts
Razor12911 is on a distinguished road
Test 2 uploaded.

You can now apply srep and lzma to get theoretical output size, fixed the so called bug.

Plus, I added Multi threading, I just want to see how much speed can I get out of this thing before I even start with restoration. Multi threading starts working if you got an input more than 64MB, It's 1 Thread = 64MB, watch RAM usage and CPU Usage, it uses the number of threads you have minus 1. So if you got an i7, it will use 7 threads which is 7 x 64MB =448MB, so expect +-1536MB RAM usage, but if you got crazy inputs like that game called Dirt Rally, then you're going to need about 6GB RAM.

Quote:
Originally Posted by Mini View Post
Mad Max

game24.arc - 812 mb

data.arc - 1.45 gb

time ~ 5 min
You can go again, I want to see what will happen to that 5 minutes you got. (Hopefully it is going to be less)

@Everyone
I'm taking a knee, Panker is busy with exams, but you can keep testing.
Attached Images
File Type: png brute5.PNG (43.1 KB, 430 views)

Last edited by Razor12911; 12-05-2016 at 22:26.
Reply With Quote
The Following 4 Users Say Thank You to Razor12911 For This Useful Post:
Lucas65 (13-05-2016), Mini (13-05-2016), RamiroCruzo (12-05-2016), Simorq (13-05-2016)
  #13  
Old 13-05-2016, 01:17
Mini Mini is offline
Registered User
 
Join Date: Nov 2013
Location: -
Posts: 125
Thanks: 166
Thanked 76 Times in 43 Posts
Mini is on a distinguished road
multi-th log
Quote:
Compressed 1 file, 852,172,800 => 1,610,249,238 bytes. Ratio 188.96%
Compression time: cpu 1.75 sec/real 393.73 sec = 0%. Speed 2.16 mB/s
data.arc (1 th) - 1.45 gb
DATA.ARC (multi-th) - 1.49 GB

time (1 th) - ~10 min
time (multi) - ~6 min
Reply With Quote
The Following 2 Users Say Thank You to Mini For This Useful Post:
Amsal (14-05-2016), Razor12911 (13-05-2016)
  #14  
Old 13-05-2016, 03:59
Simorq's Avatar
Simorq Simorq is offline
Registered User
 
Join Date: Mar 2014
Location: Iran
Posts: 642
Thanks: 3,602
Thanked 1,296 Times in 464 Posts
Simorq is on a distinguished road
GTA V x64b.rpf
Input : 142 MB
Brute (Test) : 309 MB (2.5 min)
Reflate Bilawal (Refl8v100): 312 MB (3 min)
Brute+Srep+lzma:max: 115 MB (3.5 min)
Reflate+Srep+lzma:max: 116 MB (4 min)

Brute (Test)
Code:
Compressed 1 file, 149,448,704 => 324,310,946 bytes. Ratio 217.00%
Compression time: cpu 0.17 sec/real 221.05 sec = 0%. Speed 0.68 mB/s
Brute+Srep+lzma
Code:
Compressed 1 file, 149,448,704 => 121,086,757 bytes. Ratio 81.02%
Compression time: cpu 130.48 sec/real 259.64 sec = 50%. Speed 0.58 mB/s
GOOD

Last edited by Simorq; 13-05-2016 at 04:06.
Reply With Quote
The Following 3 Users Say Thank You to Simorq For This Useful Post:
Amsal (13-05-2016), Razor12911 (13-05-2016), Sergey3695 (13-05-2016)
  #15  
Old 13-05-2016, 07:33
Lucas65 Lucas65 is offline
Registered User
 
Join Date: Nov 2015
Location: Italy
Posts: 59
Thanks: 110
Thanked 23 Times in 15 Posts
Lucas65 is on a distinguished road
Thanks for very good job. I tested first brute with this results:

On file GTAV (x64a.rpf - 46.4mb)
Only brute
Compressed 1 file, 48,699,392 => 154,259,682 bytes. Ratio 316.76%
Compression time: cpu 0.06 sec/real 33.15 sec = 0%. Speed 1.47 mB/s
only Reflate
Compressed 1 file, 48,699,392 => 155,496,915 bytes. Ratio 319.30%
Compression time: cpu 0.05 sec/real 266.69 sec = 0%. Speed 0.18 mB/s

On Fifa 15 file (data_graphic1.big - 17.9 mb)
Only precomp brute
Compressed 1 file, 18,855,627 => 58,149,544 bytes. Ratio 308.39%
Compression time: cpu 0.02 sec/real 167.31 sec = 0%. Speed 0.11 mB/s
Only brute
Compressed 1 file, 18,855,627 => 58,142,118 bytes. Ratio 308.35%
Compression time: cpu 0.00 sec/real 1.05 sec = 0%. Speed 18.04 mB/s
Only reflate
Compressed 1 file, 18,855,627 => 59,147,086 bytes. Ratio 313.68%
Compression time: cpu 0.02 sec/real 234.76 sec = 0%. Speed 0.08 mB/s

I tried brute2. Thank again Razor!
Reply With Quote
The Following 4 Users Say Thank You to Lucas65 For This Useful Post:
Amsal (13-05-2016), Mini (13-05-2016), Razor12911 (13-05-2016), Simorq (14-05-2016)
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Reflate test Razor12911 Conversion Tutorials 8 09-09-2015 01:35
Test Drive Unlimited 2 Beta Dopefish PC Games - Protection List 5 06-02-2011 05:46
Borderlands v1.20 [MULTI5] No-DVD/Fixed EXE crash when fast travel to Jakobs Cove mikehd PC Games 24 10-04-2010 10:32
Brute Force MrCaine01 XBox Games 17 21-06-2003 17:18
PS2 Tray Ejects/Closes Too Fast!!! cobraxt PS2 Games 2 16-06-2002 15:21



All times are GMT -7. The time now is 22:15.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com