|
|
|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
plzip_modded - parallel 64bit lzma (de)compressor and a direct 4x4:lzma replacement
Plzip(not to be confused with pzlib) is a lzma compressor similar to xz, however fully scalable for multiple cores. Unlike other variants like 7zip/lzma2 or xz as well as many others, this one also fully support parallel decompression. It is therefore fully scalable similarly to "lolz -mt# -mtt1" but with pure lzma and therefore quicker. It is much better option IMO than 7z or xz as a replacement to FreeArc's internal (4x4)lzma. Plzip fully support <stdio> both ways and for both compression and decompression. Therefore I consider it best 64bit alternative to current 4x4:lzma available - as of now.
However, default plzip intentionally hinder itself on decompression by limiting memory for thread's splitter/muxer, therefore by default any bigger number of cores and bigger dictionaries result in using again only 1/2 cores during decompression, putting it basically back to xz/lzma2/7z level. That despite potential and design it pose. Or to be more precise, it does utilize all the core available, but only if you decompress from physical file to another, <stdio> will simply scale back. I had a mail exchange with plzip's original author about this where he confirmed this design. Which is where my mod comes. As FA's (ahead of its time) 4x4 can prove, things can be done with reasonable memory management and better. This mod change memory limits for decompression(for splitter/muxer not lzma's). Specifically, with this mod you can use block size up to 128m and up to 16 threads during compression and still decompress utilizing full 16t cpu load. You can of course use bigger block size and even more threads as much as default software support, but then decompression will start scaling back again. But I think this is enough for a foreseeable time. Furthermore and just like lolz, you can compress today with different number of threads and in the future when maybe 16t cpu will become normal, it will utilize it already without need for re-compression. Thats why -n parameter for decompression as well. Finally, I also modded lzma's lc parameter from default 3 to 8, making it equivalent to lzma:lc8 of FA. But, you can now also only decompress with this modded plzip only, because its hardcoded at compile time. Memore utilization for decompressing (minus for lzma * threads) should be around up to ~2Gb. I have tested it on 4t cpu with 128m block size and it loaded all cores upon decompression as expected. Original plzip utilized less memory, but was slower and only used ~1t with that setting. This mod should now be similar in behavior to 4x4, which you dont need anymore but if you want there is also simple minilzip exe which is single threaded 64bit lzma. That one you can use with 4x4 like: "4x4:b64m:t#:minilzip:s32Mi:...". Yes 4x4 support more than 4 cores ^_^. But you need block size of 4x4 2x of dictionary size of external packer or it will crash. Here is arc.ini example: Code:
[External compressor:plzip] default = -s 64Mi -m 32 -B 128Mi -n 4 packcmd = plzip.exe -c {options} - - <stdin> <stdout> unpackcmd = plzip.exe -c -d -n 4 - - <stdin> <stdout> [External compressor:lzlib] default = -s 64Mi -m 32 packcmd = minilzip.exe -c {options} - - <stdin> <stdout> unpackcmd = minilzip.exe -c -d - - <stdin> <stdout> Anyway, the mod(based on plzip 1.7 and lzlib 1.9): EDIT: New re-upped file include bad entropy skipping and source code to be in sync with original open source license. plzip-1.7_mod2_pre.7z PS(test for me decompression with above arc.ini if you have 16 core cpu with 128Mi block size archive and big file to confirm it really load 100% cpu sustained) Last edited by elit; 22-03-2018 at 11:57. |
The Following 7 Users Say Thank You to elit For This Useful Post: | ||
Andu21 (01-03-2018), doofoo24 (01-03-2018), EzzEldin16 (24-03-2018), Gupta (01-03-2018), Jiva newstone (02-03-2018), shazzla (01-03-2018), Simorq (08-03-2018) |
Sponsored Links |
#2
|
|||
|
|||
will it work with Edison007 isdone mod without exe ?
|
#3
|
|||
|
|||
This is a program not cls, you need also its libraries, if that what you meant.
|
#5
|
|||
|
|||
Quote:
|
#6
|
||||
|
||||
7zip 18.03 that was released, now supports parallel decompression on lzma2/xz
and also it supports up to 40% more speed on single thread except that how plzip fares on ratio ?
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45. |
#7
|
|||
|
|||
Quote:
In nutshell they are all pretty much same lzma's to me, its the smart techniques that make differences. Last edited by elit; 09-03-2018 at 14:55. |
The Following User Says Thank You to elit For This Useful Post: | ||
Simorq (10-03-2018) |
#8
|
||||
|
||||
I know how it works
me and Razor made a custom reflate tool looooong time ago at that time we didnt add entropic search inside !! (we also found that entropy that is above 5.5 signify compressed ~tiny packed ~ data) after that Razor took that code and made ZTOOL, ztool has that you seek inside which can bypass data that arent compatible with the program's parameters you can ask for his help!! i don't have the code to help you do it surely Bulat is ahead of his time ... hehehe
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45. |
The Following User Says Thank You to panker1992 For This Useful Post: | ||
elit (10-03-2018) |
#9
|
|||
|
|||
Thank you I do have entropy code already from Bulat, which in turn is from Matt's zpaq. Bulat also pointed at his own src code so there is no deficit anymore. Its only time that I need plus finishing many other things.
But to your question, I see plzip on par with 7z/xz when it come to compression, speed, quality and stability. Currently it doesn't expose all lzma params to user. But its absolutely simple low count code is great for modification. Its still more scalable I think as lzma2 probably doesnt support >8 cores whereas this is made from ground up design to any number. It also incorporate lzma1 which support lc8(lc>4) unlike lzma2. All in all this is perfect candidate for NxN:lzma and usable/stable already now if you don't need all lzma params or entropy skipping. |
The Following User Says Thank You to elit For This Useful Post: | ||
panker1992 (10-03-2018) |
#10
|
||||
|
||||
How to use 7z/xz Stdio + Multithreading (de)comp
I haven't tested your modded tool yet but i plan to
if you can digest that 99> entropy to skip data and make it somehow comparable to 4x4 but with ratio of 7z/xz (lzma2) then this tool has a shot !! Btw you can optimize 7z/xz to as many threads as you want but yours will have skipping which will make it faster by default Ex: [External compressor:lzma2] ; Block Size = -ms256m | Multithread = -mmt=8 | header = 0 packcmd = 7z a -txz -an -m0=lzma2{ption}:fb=273:mf=bt4:mc=100000000:lc=4: lp=0 -mmt=4 -ms1024m -mx9 -si -so <stdin> <stdout> unpackcmd = 7z x -txz -an -y -si -so <stdin> <stdout> [External compressor:lzma2_alt] ; Alternative Chunking Method for multithreading decompression header = 0 packcmd = 7z a -txz -an -m0=lzma2:d512m:c1024m:fb=273:mf=bt4:mc=100000000:l c=4:lp=0 -mmt=4 -mx9 -si -so <stdin> <stdout> unpackcmd = 7z x -txz -an -y -si -so <stdin> <stdout>
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45. |
#11
|
|||
|
|||
entropy.png
^_^ ,but wait a bit longer I want to finish this properly and add extra options for user(some lzma options like mc (& lc), memory limit for decompression slots option etc). PS(mods, I am not sure if this post belong to conversion tutorials I think I made mistake, pls feel free to move it if you think its appropriate). |
The Following User Says Thank You to elit For This Useful Post: | ||
panker1992 (21-03-2018) |
#12
|
||||
|
||||
Quote:
i know you want to do it properly maybe if you want you can add some filters, or maybe content modelling preferably sound for example lolz by profragger has textures modelling
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45. |
#13
|
|||
|
|||
Quote:
However, what I can do in future is to create variants, for example how would you like fully parallel Razor compressor that also support stdio, instead of lzma? I bet people would get crazy over it. But for that I would need support from its original author to access functions as it is closed source. Was also thinking lzss if its better or lzham and so on. I can inject different compressors but certainly not filters as this is not feasible with its current design. But I can say already this plzip is very solid(as a lzma). With all other options and better user control it will be better than either xz or latest 7-zip. If I can implement it as I think that is. It also support lc8, unlike those. |
#14
|
||||
|
||||
razor cannot be multithreaded as it is right now due to the fact that it doesnt create blocks, its near optimal parsing cannot allow for it (author himself explained)
as to stdio i was the one to ask Christian to make it and he said "no" because he goes for Unicode! i like your tool and will try it like your ideas and really wanna see how your modded stuff can be unique in a sense
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45. |
#15
|
|||
|
|||
Here is with finished bad entropy data skipping(raw copying), set at 97.9% currently. Anything above is copy. Raw data are still CRC checked. This part is considered stable by me now so let me know if you find anything.
From now on I will focus on changing/adding options. So next release should be completely new named compressor in new thread(I will refer link here). I dont know exactly how long it will take, maybe few days or weeks depending on how things go and my time. Btw arc.ini is like this: Code:
[External compressor:plzip] default = -s 64Mi -m 32 -B 128Mi -n 4 packcmd = plzip.exe -c {options} - - <stdin> <stdout> unpackcmd = plzip.exe -c -d -n 4 - - <stdin> <stdout> EDIT: vis first post Last edited by elit; 22-03-2018 at 11:58. |
Thread Tools | |
Display Modes | |
|
|