Fast Zlib Recompressor by Razor12911 (*Test*)

[ChronoCross]

Like felice says ", also work on the compression threshold and increase the compatibility of file extensions, I think the interest of all."

panker and razor check the source code of quickbms have a lot of libraries of games to add on your fzr.

[Razor12911]

ChronoCross, quickbms? what are you on about?
Zlib is Zlib, I don't think quickbms is required.

Let me share what I know, I don't know how many times I've read the zlib documentation to know what I know so far, a lot of trial and errors and figured out how some things work.

Um, okay. Let's begin.

Firstly, invalid zlib streams, this doesn't really make sense at all.

The term invalid zlib stream I think defines whether the a stream is either zlib or not zlib, if it is invalid then it is not a zlib stream.
For example:
It's like Panker makes an algorithm called 1992, now you make a recompressor for all files that may have the 1992 algorithm then I come along and modify the 1992 algorithm for a certain archiver, I remove the headers the 1992 algorithm make and insert mine, do a couple of changes to the source so that is favours size. Now when you test your recompressor on my creation it doesn't work then comes along Ramiro, he makes recompressor like yours but that supports the recompression of my archiver and comes Felice and says your recompressor doesn't work on invalid zlib streams but the one by Ramiro handles them.

(Then it means they are not invalid but they are modified)

The term you used, "Invalid" is what I didn't understand at all because it only tells us whether it is zlib, or not.

If zlib streams used a modified zlib source, it's not that they can't be detected some can be detected and some can't, kinda like the GTAV scenario which is why precomp normally can't find anything on those game files, there is brute, precomp can detect them, just like reflate but the only difference is reflate is a tool that works does its detection using the zlib streams headers [Visit stackoverflow and look for this topic: structure - What does a zlib header look like?] + known file types that use zlib algorithm like pkzip archives, pdf, png and etc and precomp and fzr only rely on default zlib stream headers.

That's point number 1, detection.

Point number 2 is restoration.

You might think precomp can't achieve reflate's output size, but it can. This will explain why there is no such thing as processing invalid zlib streams. First if a stream cannot be decompressed then it is not a zlib stream but if it can be decompressed then it is a zlib stream.

Misconceptions with a lot of people is they think that recompressors like precomp and during processing is they think it's the compression process whereas its decompressing, reflate does the same. It's the opposite of what most people think. After decompression, the recompressor must check if it can compress the file (restore) and put it back the way it was before it was decompressed, CRC must match if it doesn't match, then the stream is discarded, the only superiority of reflate is it has a trick to overcome CRC because the creator carefully studied how the other file types that have zlib container operate which is why sometimes it's seen as faster but it knows how to restore the file without any CRC checkups.

I tried to summarize my knowledge as much as possible otherwise there are even more reasons but detection and restoration is what the difference is between precomp and reflate.

Now the difference between precomp and the fzr project (it's not frz by the way).
Maybe you missed the topic name, "Fast Zlib Recompressor", keyword, "Fast" it just offers speed. It just uses original zlib headers to detect zlib streams, the zlib length is quite small because what Panker and I picked up is most games have streams that are small so it doesn't make an unnecessary long length like precomp to detect streams.

The only difference between precomp and fzr is fzr is faster while precomp is slow, reason being fzr considers small zlib streams and precomp focuses on streams of certain length but which is more than fzr which is why precomp can perform better than fzr most of the time, not that fzr can't be adjusted so that it also gets the same size as precomp, it can be adjusted but I just decided to leave the option out. (The project itself was created in just a space of 4-5 hours) so you can imagine.

Can you explain how you think precomp and reflate variate, I wanna hear your version.

[ChronoCross]

ok the project is focus in zlib i understand. but...
the source code of quickbms Have a lot of lib compressions maybe can made you project more bigger.
imagine fzr detecting zlib headers and differents compression headers lz4 lzo and the others variants of compression used by games company.
that's would be great.
PD:the answer of precomp and reflate are in the previous thread.

[panker1992]

@ChronoCross

Good Evening,

i think you have misunderstood the entire concept, fzr like precomp , reflate , antiz and all of the kind of RE-Compressors.

what they do, they take the input check for recompresability (= the bility to unpack zlib for example ) then use the entire input to a better lz77 or other algorithm for better ratios.

what quickbms is, its a game pack/unpacker by no means this is a recompressor, doesnt guarntee anything crc and etc. And it does work for few games and not all.

it doesnt have any abilities and even if it could be implemented in reflate itself, reflate would be so slow searching for stuff that doesnt exist.

how reflate works and why quickbms cant be implemented. reflate and all derivatives and equals ( antiz, precomp etc )

they search for headers and try to see if this is zlib or not. if its zlib they try to unpack it if not its skipped. ( this is why invalid zlib doesnt stand up ) because its either a zlib or it aint. something in the middle doesnt exist.

also using a zlib recompressor on a texture file, its like using precomp on movie files this is why you get 100% ratio there is absolutely no zlib in textures.
For the sake of this i am providing a picture with every tool used just to see what fzr does to precomp on some occasions.

[ChronoCross]

Thnks panker for clarify. I really but really want fzr be the best zlib precompressor.
Pd:excellent results in full mode panker.

[Lucas65]

Hello to all. Meantime thanks Panker and Razor for their excellent work.
But I wonder: fzr is a recompressor as it is precomp and reflate?
Thinking they are "equal" I'm testing them all three in a FIFA 15 game files. (17 mb)
Reflate: (268%) (184 sec)
Precomp: (100%) (11 sec)
FZR: (101%) (1 sec)
More or less I understand the explanation just above of razor, these three search the same zlib files differently.
From what I see it seems to reflate the best while precomp, in this case, is useless.
Fzr works slightly.
Fzr in practice is very similar to precomp but much faster.
Fzr can replace precomp or work with precomp? And with reflate?

Thanks again for your work! Regards!

[Razor12911]

Glad you see it that way Lucas, fzr works a lot like precomp if set correctly, it does what precomp does but faster but in terms reflate, we were busy with brute version of fzr, so far it faster than precomp's brute by maybe 200x to produce more or less the same size, detecting headless streams

So far with regard to the UI.sb file which is 82mb
Precomp did 373mb in 204 seconds
Reflate did 358mb in 84 seconds using 3 threads
Fzr standard did 348mb in 40 seconds
fzr brute did 418mb in 99 seconds

[Lucas65]

I tried it with the file mentioned above with precomp into brute mode: 308.39% in 177 sec.
Therefore it found more streams of reflate (268% ratio) indeed working with lzma I have reached a higher compression ratio.
If I wanted to use fzr in brute mode how should I proceed? I have to edit the .ini files in what way?
Greetings.

[Razor12911]

Function doesn't exist internal,so you're gonna have to wait for another release

[Lucas65]

Ahhn. Ok ok Razor. Now I understand!
I will wait anxiously!
Thanks for good job!

[NexusIV]

The work you are doing is impressive. Thank you guys. I shall await developments for this interesting project.