Fast Brute (test)

[Razor12911]

Like the title says, it's a test.

What needs testing here is comparison with what reflate does and/or what precomp does.

I noticed a bug after compiling that when you compress the output of brute, you get a very small output
Well it's a false output so it's just a waste of time if you want to see how much you could get if this
program as working. I was going to ask someone to try this program on Mad Max and maybe get size which
is less than 4GB in less than a day, unlike precomp (brute), this is faster though it achieves 0.2mb/s
on data without zlibs (tested on audio files) and can push up to 40mb/s or more on an input with
a lot of zlib streams (Well it is 0.2mb/s on my 9 year old Core 2 Extreme QX9650).

I already know that it doesn't work just as yet which is why I said just compare final size
with another zlib recompressor. (I edited zlib source to improve speed and that is what happened)
I get false data or repeated data making lzma compress the whole thing to a couple of kilos
and get 1% ratio which is false but working it though to be able to get correct data and make this work.


What to test in comparison with reflate:
The output size, since some people say "reflate" is the only recompressor that can detect "invalid" streams
and others can't, well you could try this program. Again, if an stream is invalid, its invalid, there is
nothing further you can do, but as I said detection and restoration are the only factors that separate these
zlib recompressors. (You know yourself)

You can also test on GTAV, I got expansion on one of the rpf files meaning it does work on GTAV as well.


What to test in comparison with precomp:
The output size and speed.

Files that you test on:
Make sure they do have zlib streams else you'll get 0.2mb/s. (brute scanning speed)

If you got Mad Max, run the program on the game, don't apply srep or lzma and those other things,
if you know what files reflate doesn't work on but precomp (brute) works, try running this.

A test on a benchmarking file that I always use to test everything is the UI.sb file and here are the results:
Input : 82.3MB
// Bigger is better
Brute (Test) : 421MB (Without verification)
Precomp + Reflate : 421MB (Evidence that reflate isn't really brute, it had to work with precomp to get that size)
Precomp (Brute) : 421MB (Probably, I had to cancel it, it takes long)
Precomp (Intense) : 373MB
Reflate (Gozarck) : 361MB
pZlib : 348MB

In terms of speed while working on a file (UI.sb):
Input : 82.3MB
Slowest to fastest:
pZlib : 348MB (6.39 seconds)
Brute (Test) : 421MB (47.70 seconds) //Quite slow but working on it.

Work of Razor12911 and Panker1992

[Simorq]

GTA V x64b.rpf
Input : 142 MB
Brute (Test) : 308 MB (2 min)
Brute+Srep+lzma:max: 55 MB (Monster Compressed )

Reflate u3 : 312 MB (10 min)
x64b.rpf (191MB) x64b.ref (359MB) x64b.srep (253MB) x64b.arc (162MB)

Brute (Test)


Reflate


Good

[1234567890123]

simorg it's very good but which lzma code did you use on reflate u3? and which lzma method is the best lzma:max or another long method (using it in masked-ms).and what is the best compressor for after reflate? lzma nanozip fast paq8 (fp8) zpaq..

[Mini]

Mad Max

game24.arc - 812 mb

data.arc - 1.45 gb

time ~ 5 min

[Razor12911]

Originally Posted by Simorq

GTA V x64b.rpf
Input : 142 MB
Brute (Test) : 308 MB (2 min)
Brute+Srep+lzma:max: 55 MB (Monster Compressed )

Reflate u3 : 312 MB (10 min)
x64b.rpf (191MB) x64b.ref (359MB) x64b.srep (253MB) x64b.arc (162MB)

Brute (Test)


Reflate


Good

Have to work on that 4mb difference
Like I said. The result you'll be getting are false after applying srep and lzma, bug caused after I played a little with zlib source so basically, the output size is the only thing that tells us it could work at some point.

[panker1992]

unfortunately i give exams, and i havent had the time to test myself

hope you guys give some feedback, and help along the line

make sure to test and provide any feedback or glitches this soft makes ... keep an eye on ram and IO...

thanks Zee

[1234567890123]

Is it usable for decompression? i'm getting wclose: invalid argument (Bad file descriptor)
ERROR: CRC failed error.But veery good file size in (coh2 files) 2gb to 3gb.then srep lzma 146mb! i think something is wrong in brute.exe.with reflate 2gb to 3gb but then srep lzma 1100mb?

[Razor12911]

I wrote up there

don't apply srep or lzma and those other things

also

I already know that it doesn't work just as yet which is why I said just compare final size
with another zlib recompressor. (I edited zlib source to improve speed and that is what happened)
I get false data or repeated data making lzma compress the whole thing to a couple of kilos
and get 1% ratio which is false but working it though to be able to get correct data and make this work.

[ChronoCross]

Really impresive reasult razor. I was made some test and the results are very close with reflate but brute test with 1 core is more faster than reflate and the results are very close. Also some files that precomp doesn't detect in faster mode is detected by brute and reflate too. Very nice. I request the decode! For more test.

[Carldric Clement]

Originally Posted by Razor12911

Like the title says, it's a test.

What needs testing here is comparison with what reflate does and/or what precomp does.

I noticed a bug after compiling that when you compress the output of brute, you get a very small output
Well it's a false output so it's just a waste of time if you want to see how much you could get if this
program as working. I was going to ask someone to try this program on Mad Max and maybe get size which
is less than 4GB in less than a day, unlike precomp (brute), this is faster though it achieves 0.2mb/s
on data without zlibs (tested on audio files) and can push up to 40mb/s or more on an input with
a lot of zlib streams (Well it is 0.2mb/s on my 9 year old Core 2 Extreme QX9650).

I already know that it doesn't work just as yet which is why I said just compare final size
with another zlib recompressor. (I edited zlib source to improve speed and that is what happened)
I get false data or repeated data making lzma compress the whole thing to a couple of kilos
and get 1% ratio which is false but working it though to be able to get correct data and make this work.


What to test in comparison with reflate:
The output size, since some people say "reflate" is the only recompressor that can detect "invalid" streams
and others can't, well you could try this program. Again, if an stream is invalid, its invalid, there is
nothing further you can do, but as I said detection and restoration are the only factors that separate these
zlib recompressors. (You know yourself)

You can also test on GTAV, I got expansion on one of the rpf files meaning it does work on GTAV as well.


What to test in comparison with precomp:
The output size and speed.

Files that you test on:
Make sure they do have zlib streams else you'll get 0.2mb/s. (brute scanning speed)

If you got Mad Max, run the program on the game, don't apply srep or lzma and those other things,
if you know what files reflate doesn't work on but precomp (brute) works, try running this.

A test on a benchmarking file that I always use to test everything is the UI.sb file and here are the results:
Input : 82.3MB
// Bigger is better
Brute (Test) : 421MB (Without verification)
Precomp + Reflate : 421MB (Evidence that reflate isn't really brute, it had to work with precomp to get that size)
Precomp (Brute) : 421MB (Probably, I had to cancel it, it takes long)
Precomp (Intense) : 373MB
Reflate (Gozarck) : 361MB
pZlib : 348MB

In terms of speed while working on a file (UI.sb):
Input : 82.3MB
Slowest to fastest:
pZlib : 348MB (6.39 seconds)
Brute (Test) : 421MB (47.70 seconds) //Quite slow but working on it.

Work of Razor12911 and Panker1992

Razor. if you build some 'cls_brute.dll' will work with ISDone for extract files.

[Razor12911]

Originally Posted by ChronoCross

Really impresive reasult razor. I was made some test and the results are very close with reflate but brute test with 1 core is more faster than reflate and the results are very close. Also some files that precomp doesn't detect in faster mode is detected by brute and reflate too. Very nice. I request the decode! For more test.

Told ya, detection and restoration.

I still have to sort out restoration before I can make decode.

[Razor12911]

Test 2 uploaded.

You can now apply srep and lzma to get theoretical output size, fixed the so called bug.

Plus, I added Multi threading, I just want to see how much speed can I get out of this thing before I even start with restoration. Multi threading starts working if you got an input more than 64MB, It's 1 Thread = 64MB, watch RAM usage and CPU Usage, it uses the number of threads you have minus 1. So if you got an i7, it will use 7 threads which is 7 x 64MB =448MB, so expect +-1536MB RAM usage, but if you got crazy inputs like that game called Dirt Rally, then you're going to need about 6GB RAM.

Originally Posted by Mini

Mad Max

game24.arc - 812 mb

data.arc - 1.45 gb

time ~ 5 min

You can go again, I want to see what will happen to that 5 minutes you got. (Hopefully it is going to be less)

@Everyone
I'm taking a knee, Panker is busy with exams, but you can keep testing.

[Mini]

multi-th log

Compressed 1 file, 852,172,800 => 1,610,249,238 bytes. Ratio 188.96%
Compression time: cpu 1.75 sec/real 393.73 sec = 0%. Speed 2.16 mB/s

data.arc (1 th) - 1.45 gb
DATA.ARC (multi-th) - 1.49 GB

time (1 th) - ~10 min
time (multi) - ~6 min

[Simorq]

GTA V x64b.rpf
Input : 142 MB
Brute (Test) : 309 MB (2.5 min)
Reflate Bilawal (Refl8v100): 312 MB (3 min)
Brute+Srep+lzma:max: 115 MB (3.5 min)
Reflate+Srep+lzma:max: 116 MB (4 min)

Brute (Test)

Compressed 1 file, 149,448,704 => 324,310,946 bytes. Ratio 217.00%
Compression time: cpu 0.17 sec/real 221.05 sec = 0%. Speed 0.68 mB/s

Brute+Srep+lzma

Compressed 1 file, 149,448,704 => 121,086,757 bytes. Ratio 81.02%
Compression time: cpu 130.48 sec/real 259.64 sec = 50%. Speed 0.58 mB/s

GOOD

[Lucas65]

Thanks for very good job. I tested first brute with this results:

On file GTAV (x64a.rpf - 46.4mb)
Only brute
Compressed 1 file, 48,699,392 => 154,259,682 bytes. Ratio 316.76%
Compression time: cpu 0.06 sec/real 33.15 sec = 0%. Speed 1.47 mB/s
only Reflate
Compressed 1 file, 48,699,392 => 155,496,915 bytes. Ratio 319.30%
Compression time: cpu 0.05 sec/real 266.69 sec = 0%. Speed 0.18 mB/s

On Fifa 15 file (data_graphic1.big - 17.9 mb)
Only precomp brute
Compressed 1 file, 18,855,627 => 58,149,544 bytes. Ratio 308.39%
Compression time: cpu 0.02 sec/real 167.31 sec = 0%. Speed 0.11 mB/s
Only brute
Compressed 1 file, 18,855,627 => 58,142,118 bytes. Ratio 308.35%
Compression time: cpu 0.00 sec/real 1.05 sec = 0%. Speed 18.04 mB/s
Only reflate
Compressed 1 file, 18,855,627 => 59,147,086 bytes. Ratio 313.68%
Compression time: cpu 0.02 sec/real 234.76 sec = 0%. Speed 0.08 mB/s

I tried brute2. Thank again Razor!