|
Masked Compression (Precomp,reflate,srep,lzt,zstd,nz,xz)
7 Attachment(s)
~~~~~~~~~~~{Intro}~~~~~~~~~~~ In compression its all about algorithms there are tons of algorithms that are suited all for a specific different purpose this project is all about utilizing the correct compressor on specific type of files thus gaining the upper hand on compression and decompression (example) there are types of data that compress better or doesnt compress at all so why use lzma or even reflate and precomp even if you gain nothing from doing so ?? masked compression will use specific compressors per file types it will gather all text files using extensions *.ini *.rtf *.txt and it will perform the right compressor to compress the better of them the same will happen with movies files and precomp file in the latter using precomp.exe to process only files that benefit from it on movies zstd will be used for maximum decompression ~~~~~~~~~~~{Tiny SFX Packing}~~~~~~~~~~~ 1)Choose you Program or Game make sure it is small to begin with : Tiny Tools cannot support large number of Files 2) Put it inside Tiny_SFX_Compressor.bat and choose whatever compression method you want : As I said all works Including Precomp, you can leave it as is its very Optimal 3) Run the bat and wait for magic 4) after you have the data.bin run make sfx.bat 5) the exe that is produced is ready to work without anything it will automatically Unpack whatever method i have provided so far 6) This tool is provided as is, it worked for me and i gladly share, further than that i cant guarantee that it work for you!!! ~~~~~~~~~~~{Bik, Bk2 Re-encoding}~~~~~~~~~~~ Read the Readme.log i provided, it has everything you need to know ~~~~~~~~~~~{Update log}~~~~~~~~~~~ [Update 1] Added new Zstd Compressor for windows, I splited this into 2 different compression settings Zstd ultra compression = srep + zstd used a pipeline and offer great compression similar to lzma2 max and even bypass ultra on >5gb files Zstd fast compression = pure zstd offering very very fast compression ideal for fast travel data Important: If you used my previous tools, you need to download Zstd Ultra Compression to maintain support, all your data that you have already compressed by this tool will be usable only with this option [Masked Compression 2.5] masks 2.5 are here but this time a minor release 1) third mask was beta last time, now it aint ...it works out of the box with great success 2)precomp 038 remade, and optimized from scratch, precomp 0.38 is custom and it is faster than all other precomp, on my test precomp 0.45 took 9 minues almost and precomp0.38 took 5 minutes and 35 seconds ( test file was 700 mb file ) precomp038 set to slow mode and prrecomp 0.43 for brute mode, more of this in arc.ini 3) legacy2 works again since precomp038 is added again 4) msc is disabled, but you can enable it yourself, i did a proper test with this and reflate doesnt work with msc plus the ratio was increased and not decreased (if you dont believe me i can post results and you can always test yourself) 5) zstd doesnt work with my config anymore... it took errors i had to reformat entire pc and again didnt ( that was enough for me to cast it aside tho i left the sonido ) and the option for you to re-enable it 6) msrsolid mask and void mask are better optimized !!! Masked Compression Archive Added including all masks up to 2.5 Masked Compression x86 Support Masked Compression Manual Stay tuned :) ~~~~~~~~~~~{Special Thanks}~~~~~~~~~~~ Special thanks to Razor for reflate and Inno(WPI) and Gozarck for his inspiration and tweaks :) |
I understood, in fact we all understand that your methods "LZ77", "XZ" are best of the old and obsolete method LZMA, thou hast repeated a hundred times in my and in your thread, but the file you posted www.fileforums.com/showthread.php?t=97303&page=5#67 changing folder "resource" sends error in my utility with different methods of choice, if we want to change something in my utility we can do together I hope you understand this you.:rolleyes:;)
|
i know what you mean Felice :) the file i uploaded was to show you the removed dll files and the updated srep that went to 3.93 just keep those files ..... i didnt update overall you work :P
I dont know if you need the dlls for your exe to work by the way ..... because i use commandline and you interface :) |
My utility uses a variety of combinations for different methods, and the files placed in the folder 'resource' are not included by chance..:)
I checked the files included in your 'Masked Compression' method bat console and arc.ini files modified, it is not difficult to insert and update all in my utility, but the combinations of choice increased considerably, and this may be a problem ...:rolleyes::D:p |
Felice if each compressor uses from 2~5 different switches then if you combine 5 programs
the amount of switches dramatically change and even a single modification is chained :P i suggest you toy with my bat file before you try to combine it in DLG :P I want to hear some thoughts on it ..... did you find it better ?? or worse ?? something that will help me too as you can see clearly decompression is still the same nothing will change there at this point masks only help the compression part :) hope you like this far i have some nasty ideas on compression but only razor can help me on this because i work solo i dont have any help besides this community. packjpg that is included in there is the exe from the latest release, i can also compile it in dll but cannot inject in precomp in order to upgrade it, if razor can inject it or better make the code use multithreading it would be great packjpg is GPL program by the way all can get its source !! |
Ok I will be working on the 'Masked Compression' I have a couple of titles to be compressed and are curious to know how it acts..:D
Meantime remove the zip file that you posted in my thread, if it is used by users and give error, by me come to break the balls for to malfunction..:D;):) |
ops you are right i assume you got the updates especially on srep you will need them :D
|
Quote:
|
:eek:truly the best so far
|
@oltjon if you have weak computer the compression speed will be like half or even better
the same goes for high end computers but it can run parallel hope you like it :) |
@PANKER1992, Really THANKS A LOT FOR THIS! ONLY one question, how to extract the archives in inno script?
And Bro, tell me with reflate and without reflate. I know how to compress with reflate and with reflate, only I am asking, how to decompress the files with an inno script. Please answer in detail! Again thanks for the Masked Compressor, it is really very fast |
@Amsal really thanks, i will update decomp tomorrow so no worries
generally in decomp nothing changes, masks only exist to instruct how a compressor behaves example: all text files are compressed next to each other and movies the same and so on thus gaining an upper hand in everything, trying to apply the best settings in each case so it doesnt affect decompression because decomp doesnt care what it extracts as long as it extracts :) DETAILS TOMORROW :) its 4 morning here :P |
Quote:
That mean, your compressor is amazing, i am shocked with results!! Please post the decompression method soon as possible? I will go on vacation in 2 days, so please post as soon as possible! |
now that is what i want all of you to do :)
very nice feedback copy all files from my 7z somewhere and make a unpack.bat file write inside unpack.bat arc.exe x data1.bin -y -pYourpassword and save this now see how fast truly gets :) Inno is coming up tomorrow ..... it works ok so no worries :) |
Quote:
|
There it goes :P Inno is up
As i uploaded right now i cant know for sure if you get errors tomorrow i will help !!! |
10++
|
Quote:
|
Quote:
An error occurred while unpacking: Unable to write data on disk! Unarc.dll returned an error code: 11 ERROR: archive data corrupted (decompression fails) |
http://i.hizliresim.com/orQRZ9.png how can i fix this?
edit:i have got same error in Split_Kompress_v1.3d lz77 |
panker1992 amazing work bro i compress in 1/3 with this method. now only waiting for decompression i tried with inno and it is successfully decompressed too but getting some problem with some games. by the way thanks for the method.
|
Quote:
|
1234567890123 the error you get is probably because you have to little ram try to lower dictionary a little
lz77 is set to use 9 gigs of ram any less produces this error so use d:512m for 5~5.5 gigs of ram and respectively d:256m for 2~3 gigs of ram required @Amsal if you downloaded the split compress from felice post it doesnt work that was only for felice to update srep and see how dll work lz77 only exists and allows for 7z.exe to use bigger dictionary for precompressed and reflated data for maximum results it uses 10 gigs of ram so carefully change dictionary |
Quote:
And can you give me a attachment of the .bat file where the dictionary size is changed to 256mb, because my PC RAM is only 4gb! |
Optimize for 4gigs and 8gigs of Ram
if your computer has 4G ram then use this
[External compressor:lz77] header = 0 packcmd = 7z a -txz -an -mcrc=0 -m1=lzma2:d200m:fb=273:mf=bt4:mc=10000:lc=4:lp=0 -mmt=2 -mx9 -si -so <stdin> <stdout> unpackcmd = dec x -txz -an -y -si -so <stdin> <stdout> [External compressor:xz] header = 0 packcmd = 7z a -txz -an -mcrc=0 -m1=lzma2:d100m:fb=273:mf=bt4:mc=10000:lc=4:lp=0 -mmt=4 -mx9 -si -so <stdin> <stdout> unpackcmd = dec x -txz -an -y -si -so <stdin> <stdout> if your computer has 8G ram then use this [External compressor:lz77] header = 0 packcmd = 7z a -txz -an -mcrc=0 -m1=lzma2:d512m:fb=273:mf=bt4:mc=10000:lc=4:lp=0 -mmt=2 -mx9 -si -so <stdin> <stdout> unpackcmd = dec x -txz -an -y -si -so <stdin> <stdout> [External compressor:xz] header = 0 packcmd = 7z a -txz -an -mcrc=0 -m1=lzma2:d256m:fb=273:mf=bt4:mc=10000:lc=4:lp=0 -mmt=4 -mx9 -si -so <stdin> <stdout> unpackcmd = dec x -txz -an -y -si -so <stdin> <stdout> |
thanks it works now :)
|
make sure you update to the Inno v2.... also i am posting other useful stuff here like lossy compression as well
enjoy !! :) |
Bro, if I didn't want to use PackJPG at all, then what I have to do?
|
-mprecomp+msc+srep64+rep:512+delta+exe+xz/$void=srep64+zstd/$text=srep64+zpaq
text and uncompressible only :P for Amsal |
Sorry Panker but i have this error with precomp:
http://i.imgur.com/WCM1ULW.png Have you got any idea? Thanks in advance. |
Sorry Panker but I have this error with precomp:
http://i.imgur.com/WCM1ULW.png Have you got any idea? Thanks in advance. |
Might I suggest you something amigo...The .bik to .bk2 encoding batch you are using might be a troublesome one...
Your batch file will just delete all the .bik files after the encoding...But its a common error that some .bik files aren't supported then your encoder will simply delete it too... Use this : Code:
FOR /F "usebackq delims=" %%i in ( `dir /b /s *.bik` ) do ( IF EXIST "%%~dpni.bk2" ( DEL "%%i" ) )Code:
Del /F /Q %%i |
@Lucas delete temporary files and rename data1.bin to data2.bin in Masked Compression.bat
if precomp finds same data it pops up errorss like these :P please do not hesitate to contact if the problem persists @Ramiro this is not a *.bik to *.bk2 this is a bik and bk2 RE-ENCODER it scans for bik and bk2 files and it downgrades them to 50% and 70% respectively :P sorry i didnt write that above :) |
Quote:
|
Quote:
Remember, I want to use Precomp, but I don't want to use PACKJPG and MSC! Thanks in Advance! |
Awesome stuff panker1992, hats off. looking forward with great anticipation!
Myself as a rookie, I mainly use UltraArc cuz of its reasonable GUI, any chance to see any simliar attempt, implementation? Keep up the good work, cheers! |
@Amsal i dont understand what you want to do exactly ....
maksed compression is all about masks and different algorithms if you want to skip a program simple edit this line -mprecomp+srep64+rep:512+delta+exe+xz/$void=srep64+zstd/$text=srep64+zpaq this is the command to skip MSC ......MSC only compresses textures wav sound mp3 sound and bmf format packjpg compresses jpg format completely different things @Kazamnuk i dont like GUI at all .... because i dont like GUI i am able to contribute projects like this i experiment i use many programs that i find i constantly update my entire Repack library BUT... Felice has made a nice effort to make a full yet user friendly GUI for newbies he will update to masks eventually so keep an eye out :) Peace !! :) |
Quote:
but I get the same error. I'm sorry for the trouble. I also disabled the antivirus but nothing. Thanks for your help. |
Lucas it seems that you are the only with this kind of trouble,
try to disable precomp for the test utilize all except precomp really wanna see if this is general or precomp specific problem. also move the Folder to an external drive and pack something small for this test dont use this in C:/ drive because almost everytime it will fail Upload a photo with the folder + the cmd with the problem please |
Hey panker i''m trying to test your new method all in one, so you called, with different titles small, to speed up the test, but it seems impossible not to have managed to compress a title, with each change of line gives me an error different: error in precomp, error in LZ77, the last error "disk full in delta compression algorithm":o
http://i63.tinypic.com/34grsia.png I think you have to redefine your new method so that it fits in most of the data files.:) I also noticed that most different algorithms are added to the mask, it is more subject to error.:cool: P.S the same titles that are not able to test with your method, I have successfully compressed using the common methods known "-mmsc+precomp+srep64+xz", "-mmsc+srep64+zstd" or "-mreflate+precomp+srep+delta+lzma". I hope for a solution..:rolleyes: |
| All times are GMT -7. The time now is 15:52. |
Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com