PZlib (Alternative to Precomp)

[Razor12911]

Currently busy with exams

[Andrey167]

Multithread work?

[panker1992]

this is faster than anything on market with minimum usage of RAW power.

its blazing speed will leave you almost speechless ....processing 20 GB game in less than half an hour. for 1 core.

The Multithreaded version of this is reserved for me and Zee for now


@lolaya this tool decompresses all zlib from an input thus you gain better end compression in the end.

[felice2011]

Originally Posted by panker1992

this is faster than anything on market with minimum usage of RAW power.

its blazing speed will leave you almost speechless ....processing 20 GB game in less than half an hour. for 1 core.

The Multithreaded version of this is reserved for me and Zee for now


@lolaya this tool decompresses all zlib from an input thus you gain better end compression in the end.

WOW.....

As I have a little of time to test with pleasure...

[Stained]

Guys, is this awesome work ready to be used in a conversion? I mean can I use it ?
Also, if yes, I will need to include the file in decompression also, right? Or just using it, then lmza and then conversion will wok?

[Razor12911]

yea, then just add cls then done.

[Stained]

Thank you man. Will try it out tomorrow ��

[felice2011]

Hey Razor tell me which file extensions you've tried with PZlib posted in your tests ...

[Razor12911]

Cpk for pes, sb for nfs tr, arc for re5, bndl for nfshp and nfsmw...

[felice2011]

I took some files sample of games currently installed on my computer ...

1) Game = BAK / File = *.upk



2) Game = BAK / File = *.usm



3) Game = Dirt Rally / File= *.bk2



4) Game = Dirt Rally / File= *.pssg



5) Game = MGS5 / File= *.dat



6) Game = MadMax / File= *.arc



7) Game = ProjectCar / File= *.bff



You should be more precise, to specify what types of flows detecting in input, and what types of files, so use your new compressor with the desired result.
Except the positive result of the first test posted, with all the others I have not gotten the desired result.

************************************************** ************************************************** ****************
Edit: Test Only Upk Files

But here the situation is different, I wanted to test the only files *.upk, that in their results they have given a positive result to the new "PZlib"
Amazing speed in comparison to the use of the last precomp045, and better compression compared to reflate 1.4 ...

[Razor12911]

Haha, it aint about extensions sometimes mate, I scan files, I depend less on extension.

Strange, Dirt Rally does have zlibs and should have been detected by pZLib.

Game = Dirt Rally / File= *.bk2 (These are videos, no zlib can be located on that), same with usm.

Mad Max has headerless zlibs and those are not detected by pzlib.

[felice2011]

Originally Posted by Razor12911

Haha, it aint about extensions sometimes mate, I scan files, I depend less on extension.

Strange, Dirt Rally does have zlibs and should have been detected by pZLib.

Game = Dirt Rally / File= *.bk2 (These are videos, no zlib can be located on that), same with usm.

Mad Max has headerless zlibs and those are not detected by pzlib.

OH..OH....and this was you aware only you and possibly Panker

Ok you're the maestro of compression, I'm not trying to belittle or criticize your work, I'm tested your new compressor, and better understand its potential, maintaining a high feedback of the discussion, on the contrary would not be interested in anyone, what do you do you think?
I will do additional testing on complete games with multiple files.
Summing up your work is that of Panker is a great project, I hope in more news and updates.

[gozarck]

Simple batch for drag an drop folder to generate a list of files ready to test with reflate or pzlib. Generate 2 files rsd1.txt is a list of files with streams and rsd0.txt list of files with 0 streams. Maybe panker can do a mask for files exclusive to use reflate or pzlib for save time.
Drag the game folder into a batch and this batch generate the list of files.
Edit. I was forget to remove "" in some part of the batch sorry now is fixed. Now you can use [email protected] in arc.exe

[ChronoCross]

Thnx gozarck this help a lot to select the test files for pzlib.

[Razor12911]

Originally Posted by gozarck

Simple batch for drag an drop folder to generate a list of files ready to test with reflate or pzlib. Generate 2 files rsd1.txt is a list of files with streams and rsd0.txt list of files with 0 streams. Maybe panker can do a mask for files exclusive to use reflate or pzlib for save time.
Drag the game folder into a batch and this batch generate the list of files.
Edit. I was forget to remove "" in some part of the batch sorry now is fixed. Now you can use [email protected] in arc.exe

Great but I somehow can't make it work.

Always says:
0 Streams Founded Available to apply Reflate

@ChronoCross and gozarck
I don't think that this is going to work. pZlib detects zlibs using their magic headers.
http://stackoverflow.com/questions/9...ader-look-like
http://tools.ietf.org/html/rfc1950

For example, GTAV. Gozarck, your detector will work because rawdet has its methods of detecting that pZlib doesn't because pZlib itself has its own methods of detecting which I linked up there. Though I created Brute which works and the detector might work there, brute is still under development, restoring some streams is going to be hell, something big needs to be done first.

Originally Posted by felice2011

OH..OH....and this was you aware only you and possibly Panker

Ok you're the maestro of compression, I'm not trying to belittle or criticize your work, I'm tested your new compressor, and better understand its potential, maintaining a high feedback of the discussion, on the contrary would not be interested in anyone, what do you do you think?
I will do additional testing on complete games with multiple files.
Summing up your work is that of Panker is a great project, I hope in more news and updates.

I thought everybody knew, bk2 are bink 2 videos and usm is also a video format. It can't work on Mad Max because of the reason above I just explained.
We did try Dirt Rally though, recompressor needed 3.4GB from just 700mb worth of data and decompression failed, it needed 5GB somehow, Dirt Rally has huge zlib streams, we estimated that the game even when it is big already, originally with all the files updated, it is somewhere around 100GB just like GTAV.
Thank you for the test though.

It also works for Dragon Age BTW, Panker tested it a long time ago. Here are the results
http://fileforums.com/showpost.php?p=448875&postcount=6