PZlib (Alternative to Precomp)
 

Pre-ZLIB (pzlib)

Alternative of precomp

This program detects and unpack zlib streams for a certain input
giving you an option to apply a stronger compression such as LZMA.

This only detects zlib streams compressed using default and maximum level
which have headers. (It doesn't detect headerless)

Usage:
pZLib encode/decode [options] inputfile -o outputfile
inputfile/outputfile can be specified as "-" for stdin/stdout

Options:
-t#: Number of threads to use
-t#p: (default = 50) Percentage of threads to use
-v: Verification so that restoration keeps CRC the same
-ai: Prints out additional information

Note:
Input and/or Output can be "-" to signal stdin and/or stdout usage.
Multi threading only works during restoration.

Example:
pzlib encode $$arcdatafile$$.tmp -o $$arcpackedfile$$.tmp

Created by Razor12911 and Panker1992

good job! waiting CLS-PZlib ^_^

CLS-PZlib is not really necessary. It will work with Inno no problem as long as arc.ini is there, it will show progress etc.

CLS uploaded

Took about 20 minutes to make CLS, its not properly tested but it works.

CLS works on its own, it doesn't need exe alongside it.

This Method is actually a zlib killer :P

it can pack Dragon Age Inquisition, which precomp took 12 hours in about 1 our or so :D

i hope we see some results, this is the method i had once used to show reflate's speed packing dragon's dogma.

this is actually one of the 4 methods our fully fledged reflate has :D

enjoy !!

Linking Articles old test
 

Me and Razor decided to show you the progress we did so far,

this test is about how far we are going with reflate

input file was the entire game Dragon's Dogma

we only applied -mreflate and not fully repack for the sake of comparison

the game total input was 19.7 GB

the left CMD is HDD and right CMD is SSD

the process on HDD took 54 minutes, in which reflate took about 30 minutes to fully process the file

the process on SSD took 38 minutes, in which reflate took about 22 minutes to fully process the file


Edit : Extraction was Successful :D it took 28 minutes to decomp <stdin> <stdout> usage

Original Article here : http://fileforums.com/showthread.php...hlight=reflate

The effect of "-l" option.

Thank you panker and razor for the soft, I think your partnership is going well. I hope you come with more great works. Thank you again for this ;)

Is this work with ASC?

Yea, I tested.

Can you explain this sentence, "348MB instead of 82.3MB" ? Does it mean that the original file size was 82.3MB and resultant file size came out to be 348MB?

It means that's the new file size from the input, from the 348MB, you then apply stronger compression to make the file size drop even further, for example this file. Originally, it is 82.3MB, after applying pZlib, it becomes 348MB, from that apply LZMA it becomes 9MB. (For this files, other files behave differently)

9MB! That's amazing, man. And thank you, for explaining.😊

Very nice tool works correctly.

1. what news about decompression FZR?
2. whether there will be a support multithreading Pzlib?

Currently busy with exams

Multithread work?

this is faster than anything on market with minimum usage of RAW power.

its blazing speed will leave you almost speechless ....processing 20 GB game in less than half an hour. for 1 core.

The Multithreaded version of this is reserved for me and Zee for now :)


@lolaya this tool decompresses all zlib from an input thus you gain better end compression in the end.

WOW.....:eek:

As I have a little of time to test with pleasure...;)

Guys, is this awesome work ready to be used in a conversion? I mean can I use it ?
Also, if yes, I will need to include the file in decompression also, right? Or just using it, then lmza and then conversion will wok?

yea, then just add cls then done.

Thank you man. Will try it out tomorrow ��

Hey Razor tell me which file extensions you've tried with PZlib posted in your tests ...;)

Cpk for pes, sb for nfs tr, arc for re5, bndl for nfshp and nfsmw...

I took some files sample of games currently installed on my computer ...

1) Game = BAK / File = *.upk

http://i65.tinypic.com/2111ooh.png

2) Game = BAK / File = *.usm

http://i65.tinypic.com/dq6353.png

3) Game = Dirt Rally / File= *.bk2

http://i65.tinypic.com/2j4qz9c.png

4) Game = Dirt Rally / File= *.pssg

http://i66.tinypic.com/2elftso.png

5) Game = MGS5 / File= *.dat

http://i63.tinypic.com/28i23rt.png

6) Game = MadMax / File= *.arc

http://i65.tinypic.com/23j50df.png

7) Game = ProjectCar / File= *.bff

http://i65.tinypic.com/97mlj8.png

You should be more precise, to specify what types of flows detecting in input, and what types of files, so use your new compressor with the desired result.
Except the positive result of the first test posted, with all the others I have not gotten the desired result.

************************************************** ************************************************** ****************
Edit: Test Only Upk Files

But here the situation is different, I wanted to test the only files *.upk, that in their results they have given a positive result to the new "PZlib"
Amazing speed in comparison to the use of the last precomp045, and better compression compared to reflate 1.4 ...

http://i63.tinypic.com/2lj6dxe.png

http://i67.tinypic.com/2lwr3wz.png

Haha, it aint about extensions sometimes mate, I scan files, I depend less on extension.

Strange, Dirt Rally does have zlibs and should have been detected by pZLib.

Game = Dirt Rally / File= *.bk2 (These are videos, no zlib can be located on that), same with usm.

Mad Max has headerless zlibs and those are not detected by pzlib.

:rolleyes:OH..OH..:eek:..and this was you aware only you and possibly Panker:D

Ok you're the maestro of compression, I'm not trying to belittle or criticize your work, I'm tested your new compressor, and better understand its potential, maintaining a high feedback of the discussion, on the contrary would not be interested in anyone, what do you do you think?
I will do additional testing on complete games with multiple files.
Summing up your work is that of Panker is a great project, I hope in more news and updates.;)

Simple batch for drag an drop folder to generate a list of files ready to test with reflate or pzlib. Generate 2 files rsd1.txt is a list of files with streams and rsd0.txt list of files with 0 streams. Maybe panker can do a mask for files exclusive to use reflate or pzlib for save time.
Drag the game folder into a batch and this batch generate the list of files.
Edit. I was forget to remove "" in some part of the batch sorry now is fixed. Now you can use [email protected] in arc.exe

Thnx gozarck this help a lot to select the test files for pzlib.

Great but I somehow can't make it work.

Always says:
0 Streams Founded Available to apply Reflate :D

@ChronoCross and gozarck
I don't think that this is going to work. pZlib detects zlibs using their magic headers.
http://stackoverflow.com/questions/9...ader-look-like
http://tools.ietf.org/html/rfc1950

For example, GTAV. Gozarck, your detector will work because rawdet has its methods of detecting that pZlib doesn't because pZlib itself has its own methods of detecting which I linked up there. Though I created Brute which works and the detector might work there, brute is still under development, restoring some streams is going to be hell, something big needs to be done first.

I thought everybody knew, bk2 are bink 2 videos and usm is also a video format. It can't work on Mad Max because of the reason above I just explained.
We did try Dirt Rally though, recompressor needed 3.4GB from just 700mb worth of data and decompression failed, it needed 5GB somehow, Dirt Rally has huge zlib streams, we estimated that the game even when it is big already, originally with all the files updated, it is somewhere around 100GB just like GTAV.
Thank you for the test though.

It also works for Dragon Age BTW, Panker tested it a long time ago. Here are the results
http://fileforums.com/showpost.php?p=448875&postcount=6

Same thing, I tried one minute ago the idea of Gozarck but give 0 Streams on any file, even those they should work.

I don't get all the hocus-pocus around the Zlib term...So, here's some database about Zlib, Reflate, Brute, Precomp & pZlib.

@ChronoCross You should read this as there are no Bad-Zlib Stream or Invalid Streams.

Originally, data is stored in streams called Deflate Stream. Those streams are also called Headerless Zlib stream. Once, you wrap these Deflate streams into Headers, they become Zlib stream or Gzip Stream.

Zlib Stream contains Header + CRC + Data. (Header + CRC) = 15 position upto the stream. So Deflate stream = ZlibStream -15.

Now, why Deflate Streams are difficult to search for: As the header contains so called Magic Bytes which help to identify them, but when you seek for Deflate streams, its a bit more Brute & Slow...

Magic Bytes:
Now comes how Reflate works...
Rawdet reads all the offsets of the file & initiates Inflate from each of them. [Inflate = Decompression of Zlib Stream]. It usually gets error where the stream isn't present & where ever it successfully inflates, it creates buffer out of the data. Thats what .raw are, unprocessed Deflate streams. Raw2hif firstly Inflates the stream & then Deflates (Compresses) the streams with the level you have provided e.g. -c9 compresses the stream in Level9, thats why the higher the level, the smaller is the output.

Brute on the other hand, instead of creating buffers, it directly inflates them and creates the output.

Now, comes the Precomp part...Its a storehouse of processes. You can assume its slowness is due to it hit & trial method. It again & again reads the file to discover more & more stream & even to find info about stream it takes about 9 * 9 trials. Thats why its disk reading values are high e.g. for Brute of 1 GB file it approx. reads 98 GB data.

pZlib, it scans for the Magic Bytes & Inflates the streams, easy-peasy... :p Not in coding....

The Witcher 3 Wild Hunt


http://i.imgsafe.org/78f2bd7.png

Done at normal HD, I will post later on my SSD

Gozarck's tool works fine for me. Copy rawdet and bat file in the same path of the folder to test and drag the folder. This bat works with folder and not with files directly.
@ramirocruzo thnx for the info. Check the bug info on reflate by author and this link about invalid stream or non valid streams about bug or issue.
Precomp source code page 2 thread 41
http://encode.ru/threads/2411-Precom...n-GitHub/page2

Well, re-read his whole reply. With Non-Valid Streams. he meant streams which are of smaller size that the header size of Precomp. If you try to process them, the lossless restoration info will be greater that the original file. That means compression ruined. As, Reflate doesn't make headers, so yes...Reflate can process it...

Multi threading support comng

Update available

Changes

Added multi threading support

wait CLS ;) thx!!!

Mini you dont need CLS, use exe to stdout and magic is done

cls is only a lib for arc.exe its even slower than original exe itself and it is forced on smaller buffer.

Dark Souls 3
Data5.bdt - 11.4 gb

pzlib v1.0 (max level) + lzma = 9.95 gb
pzlib v1.1 + lzma = 11 gb (http://i.imgur.com/Jt9r2qn.png)

wtf