Fast Zlib Recompressor by Razor12911 (*Test*)
 

This is as title says a fast zlib recompressor, works like antiz and precomp but it features better speed.

at the moment this doesn't feature a decompression this was only made yesterday.

*Important*
please consider to post feedback, No feedback means the program will never see the light of the world just as it happened with reflate.


[External compressor:fzr]
header = 0
packcmd = fzr <stdin> <stdout>

In short this tool takes input directly from Freearc, it doesnt slow down the process at all, it runs real life on processor to provide exactly copy paste speeds (Photo below with marked speed)

Yet another point for Zee-Panker Team... :p Nice work hermano...

Hell fast...Just like F12 ;)

Hmm...Did some more tests....

mfzr+reflate
Without Fzr....
mreflate
Its awesome... :D

It does not work on GTAV files. Try another input

Pardon me...In camp so using my PC via RDP client....

Also, I tested it on my traditional 30K textures zip....

Free Reflate Time: 24 min 16 sec
Antiz time: 1 min 18 sec
Skylake time: 59 sec 49 millisec :D
Reflate m1 time: 40 sec 23 millisec
FZR time: 26 sec 12 millisec :)

Some screenshots, but as Panker said, tool was created yesterday and we really haven't started with the decompression of files this thing makes, we still have to sort out the compression part so that we can prepare it for decompression.

WOW guys Nice team and good work this is just the begining. keep going.
Speed is very good (really awesome :) )

Hey ramirocruzo reflate always win in size because detects valid and invalid zlib streams. Also if fzr works like precomp and antiz those programs discard invalid zlib streams. I made some test with diferent texture and i cant see good result. So what kind of files suggest to test?

definitely not textures :D

texture files are by no means zlib files.

you will need to test ZLIB compatible files. for example dragon age CAS extension files can do it

Hey Panker finally back to action, great job as precompressor, in addition to good speed, also work on the compression threshold and increase the compatibility of file extensions, I think the interest of all.
Also thanks to Razor for the work done.

this is a free addition to the freearc exe free performance boost.

how it can work is

-mfzr+precomp+srep+delta+exe2+xz

it performs a light scan. so that precomp is abit speedy try it :D

very good! we wait unpack

Invalid zlib streams? Maybe you referring to headerless zlib streams.
reflate doesn't always win in size too, there is a file from NFS The Run, UI.sb
with reflate, you get 361MB, precomp gets 373MB, fzr gets 348MB and some other methods of reflate that I have created get 308MB.
If a zlib stream is invalid then it is not a zlib stream, not even reflate can do anything about it.

There are 2 problems with zlib recompressors
1: detection, finding all streams a file has is not easy but finding common is easy which is what fzr, precomp and reflate, the difference with reflate is it takes it too another level, it doesn't detect common zlib, it knows what programs that use zlib as compression for zip, for example 7-Zip or KZip, all those use zlib, detecting the zlib stream using common headers is not possible so the only thing that reflate (rawdet) does I think is detect the header of the zip file created by an archiver, e.g. 7-Zip then it knows that there is a stream here and there but precomp, antiz don't do any of that.

2: is after finding that zlib stream, after unpacking it, can you restore it the way it was (matching CRC) if so then precomp, fzr or reflate will unpack it but if you can't then it is ignored.

precomp, fzr and other recompressors have these problems, if it is not detecting then it is restoration luckily with reflate, it isn't much of a problem because it is a program that is mastered to detect normal zlib streams and zlib streams that have been tuned to work specifically for archivers. PNG's use zlib but I myself have problems restoring the zlib stream because of the variations done to the source of zlib.

**Note, this is what I know so far.

No no im right. 1 detection of zlib headers.

2 in zlib headers detect valid zlib streams.


So invalid zlib streams in reflate are false positive. Reflate dont have a valid or invalid detection of zlib streams (like antiz or precomp) in that case reflate always wins. (correct or incorrect detection and this is not good for reflate). So i prefers do not make a comparation frz with reflate because use diferent algo and still in alpha. (like ramirocomparation) i made some test with precomp and frz. I have good result with precomp in size and with frz i have only speed.

:D
http://i67.tinypic.com/fc75ur.png
:rolleyes:
http://i65.tinypic.com/iy2ypl.png

The optimum speed at the expense of the compression...:)

P.S :"-mfzr+precomp+srep+delta+exe2+xz" (Arc in Crash - Stops Working)...:confused:

Like felice says ", also work on the compression threshold and increase the compatibility of file extensions, I think the interest of all."

panker and razor check the source code of quickbms have a lot of libraries of games to add on your fzr.

ChronoCross, quickbms? what are you on about?
Zlib is Zlib, I don't think quickbms is required.

Let me share what I know, I don't know how many times I've read the zlib documentation to know what I know so far, a lot of trial and errors and figured out how some things work.

Um, okay. Let's begin.

Firstly, invalid zlib streams, this doesn't really make sense at all.

The term invalid zlib stream I think defines whether the a stream is either zlib or not zlib, if it is invalid then it is not a zlib stream.
For example:
It's like Panker makes an algorithm called 1992, now you make a recompressor for all files that may have the 1992 algorithm then I come along and modify the 1992 algorithm for a certain archiver, I remove the headers the 1992 algorithm make and insert mine, do a couple of changes to the source so that is favours size. Now when you test your recompressor on my creation it doesn't work then comes along Ramiro, he makes recompressor like yours but that supports the recompression of my archiver and comes Felice and says your recompressor doesn't work on invalid zlib streams but the one by Ramiro handles them.

(Then it means they are not invalid but they are modified)

The term you used, "Invalid" is what I didn't understand at all because it only tells us whether it is zlib, or not.

If zlib streams used a modified zlib source, it's not that they can't be detected some can be detected and some can't, kinda like the GTAV scenario which is why precomp normally can't find anything on those game files, there is brute, precomp can detect them, just like reflate but the only difference is reflate is a tool that works does its detection using the zlib streams headers [Visit stackoverflow and look for this topic: structure - What does a zlib header look like?] + known file types that use zlib algorithm like pkzip archives, pdf, png and etc and precomp and fzr only rely on default zlib stream headers.

That's point number 1, detection.

Point number 2 is restoration.

You might think precomp can't achieve reflate's output size, but it can. This will explain why there is no such thing as processing invalid zlib streams. First if a stream cannot be decompressed then it is not a zlib stream but if it can be decompressed then it is a zlib stream.

Misconceptions with a lot of people is they think that recompressors like precomp and during processing is they think it's the compression process whereas its decompressing, reflate does the same. It's the opposite of what most people think. After decompression, the recompressor must check if it can compress the file (restore) and put it back the way it was before it was decompressed, CRC must match if it doesn't match, then the stream is discarded, the only superiority of reflate is it has a trick to overcome CRC because the creator carefully studied how the other file types that have zlib container operate which is why sometimes it's seen as faster but it knows how to restore the file without any CRC checkups.

I tried to summarize my knowledge as much as possible otherwise there are even more reasons but detection and restoration is what the difference is between precomp and reflate.

Now the difference between precomp and the fzr project (it's not frz by the way).
Maybe you missed the topic name, "Fast Zlib Recompressor", keyword, "Fast" it just offers speed. It just uses original zlib headers to detect zlib streams, the zlib length is quite small because what Panker and I picked up is most games have streams that are small so it doesn't make an unnecessary long length like precomp to detect streams.

The only difference between precomp and fzr is fzr is faster while precomp is slow, reason being fzr considers small zlib streams and precomp focuses on streams of certain length but which is more than fzr which is why precomp can perform better than fzr most of the time, not that fzr can't be adjusted so that it also gets the same size as precomp, it can be adjusted but I just decided to leave the option out. (The project itself was created in just a space of 4-5 hours) so you can imagine.

Can you explain how you think precomp and reflate variate, I wanna hear your version.

ok the project is focus in zlib i understand. but...
the source code of quickbms Have a lot of lib compressions maybe can made you project more bigger.
imagine fzr detecting zlib headers and differents compression headers lz4 lzo and the others variants of compression used by games company.
that's would be great.
PD:the answer of precomp and reflate are in the previous thread.

@ChronoCross

Good Evening,

i think you have misunderstood the entire concept, fzr like precomp , reflate , antiz and all of the kind of RE-Compressors.

what they do, they take the input check for recompresability (= the bility to unpack zlib for example ) then use the entire input to a better lz77 or other algorithm for better ratios.

what quickbms is, its a game pack/unpacker by no means this is a recompressor, doesnt guarntee anything crc and etc. And it does work for few games and not all.

it doesnt have any abilities and even if it could be implemented in reflate itself, reflate would be so slow searching for stuff that doesnt exist.

how reflate works and why quickbms cant be implemented. reflate and all derivatives and equals ( antiz, precomp etc )

they search for headers and try to see if this is zlib or not. if its zlib they try to unpack it if not its skipped. ( this is why invalid zlib doesnt stand up ) because its either a zlib or it aint. something in the middle doesnt exist.

also using a zlib recompressor on a texture file, its like using precomp on movie files this is why you get 100% ratio there is absolutely no zlib in textures.
For the sake of this i am providing a picture with every tool used just to see what fzr does to precomp on some occasions.

Thnks panker for clarify. I really but really want fzr be the best zlib precompressor.
Pd:excellent results in full mode panker.

Hello to all. Meantime thanks Panker and Razor for their excellent work.
But I wonder: fzr is a recompressor as it is precomp and reflate?
Thinking they are "equal" I'm testing them all three in a FIFA 15 game files. (17 mb)
Reflate: (268%) (184 sec)
Precomp: (100%) (11 sec)
FZR: (101%) (1 sec)
More or less I understand the explanation just above of razor, these three search the same zlib files differently.
From what I see it seems to reflate the best while precomp, in this case, is useless.
Fzr works slightly.
Fzr in practice is very similar to precomp but much faster.
Fzr can replace precomp or work with precomp? And with reflate?

Thanks again for your work! Regards!

Glad you see it that way Lucas, fzr works a lot like precomp if set correctly, it does what precomp does but faster but in terms reflate, we were busy with brute version of fzr, so far it faster than precomp's brute by maybe 200x to produce more or less the same size, detecting headless streams

So far with regard to the UI.sb file which is 82mb
Precomp did 373mb in 204 seconds
Reflate did 358mb in 84 seconds using 3 threads
Fzr standard did 348mb in 40 seconds
fzr brute did 418mb in 99 seconds

I tried it with the file mentioned above with precomp into brute mode: 308.39% in 177 sec.
Therefore it found more streams of reflate (268% ratio) indeed working with lzma I have reached a higher compression ratio.
If I wanted to use fzr in brute mode how should I proceed? I have to edit the .ini files in what way?
Greetings.

Function doesn't exist internal,so you're gonna have to wait for another release

Ahhn. Ok ok Razor. Now I understand!
I will wait anxiously!
Thanks for good job!

The work you are doing is impressive. Thank you guys. I shall await developments for this interesting project.