Simple splitting and compress by DLG - Page 3

RamiroCruzo · #1 04-12-2015, 00:04

Amigos if you wanna so fast compression method, why don't you take a look at LZ4 which goes beyond 1GB/s without I/O overwrapping & multithreading.....

panker1992 · #2 04-12-2015, 07:59

Quote:

Originally Posted by RamiroCruzo

Amigos if you wanna so fast compression method, why don't you take a look at LZ4 which goes beyond 1GB/s without I/O overwrapping & multithreading.....

I have tested lz4 and lz5 long time ago, zstd is a step ahead of lz4 lz5 and zhuff at the moment its steady and it has a lot of support the one i posted is v3.6 and the one i am testing now is 4.2 and it has 2.2 gigs/sec around
( 593mb/sec/thread = 2.2 gigs/sec )

felice2011 · #3 04-12-2015, 08:35

Quote:

Originally Posted by RamiroCruzo

Amigos if you wanna so fast compression method, why don't you take a look at LZ4 which goes beyond 1GB/s without I/O overwrapping & multithreading.....

Quote:

Originally Posted by panker1992

I have tested lz4 and lz5 long time ago, zstd is a step ahead of lz4 lz5 and zhuff at the moment its steady and it has a lot of support the one i posted is v3.6 and the one i am testing now is 4.2 and it has 2.2 gigs/sec around
( 593mb/sec/thread = 2.2 gigs/sec )

Friends you do your choices, I'm happy to update my utility just as you wish, and it however you like.
Panker do not look at only the speed, but also the compression level of the various methods, we must try to find a balance in both, to satisfy everyone.

panker1992 · #4 04-12-2015, 09:39

Quote:

Originally Posted by felice2011

Friends you do your choices, I'm happy to update my utility just as you wish, and it however you like.
Panker do not look at only the speed, but also the compression level of the various methods, we must try to find a balance in both, to satisfy everyone.

This is why i chose zstd ...because it features advanced API that can compress decent.

To be more specific zstd has at least 30% ~ 35% better compression ratio
tho everyone i see use my original -8 switch ....it can easily go as high as -20 but its ratio doesnt compare to lzma at all :P

All in all zstd gives 30% ~ 35% better compression AND it decompressed as fast as lz4

HINT : zstd and all tools i use have porpose nothing i choose is coincidence this all is part of my upcoming big project call "Masked Compression"

soon !!!

RamiroCruzo · #5 04-12-2015, 22:14

Quote:

Originally Posted by felice2011

Friends you do your choices, I'm happy to update my utility just as you wish, and it however you like.
Panker do not look at only the speed, but also the compression level of the various methods, we must try to find a balance in both, to satisfy everyone.

Lets compare ourselves which is better for our needs....

Source:Encode.ru

| Compressor name | Compression| Decompress.| Compr. size | Ratio |
| --------------- | -----------| -----------| ----------- | ----- |
| memcpy | 8533 MB/s | 8533 MB/s | 104857600 |100.00 |
| lz4 r131 | 480 MB/s | 2275 MB/s | 64872315 | 61.87 |
| lz4hc r131 -1 | 82 MB/s | 1896 MB/s | 59448496 | 56.69 |
| lz4hc r131 -3 | 54 MB/s | 1932 MB/s | 56343753 | 53.73 |
| lz4hc r131 -5 | 41 MB/s | 1969 MB/s | 55271312 | 52.71 |
| lz4hc r131 -7 | 31 MB/s | 1969 MB/s | 54889301 | 52.35 |
| lz4hc r131 -9 | 24 MB/s | 1969 MB/s | 54773517 | 52.24 |
| lz4hc r131 -11 | 20 MB/s | 1969 MB/s | 54751363 | 52.21 |
| lz4hc r131 -13 | 17 MB/s | 1969 MB/s | 54744790 | 52.21 |
| lz4hc r131 -15 | 14 MB/s | 2007 MB/s | 54741827 | 52.21 |
| lz5 r131 | 195 MB/s | 939 MB/s | 55884927 | 53.30 |
| lz5hc r131 -1 | 32 MB/s | 742 MB/s | 52927122 | 50.48 |
| lz5hc r131 -3 | 20 MB/s | 716 MB/s | 50970192 | 48.61 |
| lz5hc r131 -5 | 10 MB/s | 701 MB/s | 49970285 | 47.66 |
| lz5hc r131 -7 | 5.54 MB/s | 682 MB/s | 49541511 | 47.25 |
| lz5hc r131 -9 | 2.69 MB/s | 673 MB/s | 49346894 | 47.06 |
| lz5hc r131 -11 | 1.36 MB/s | 664 MB/s | 49266526 | 46.98 |
| zstd v0.3 | 257 MB/s | 547 MB/s | 51231016 | 48.86 |
| zstd_HC v0.3 -1 | 257 MB/s | 553 MB/s | 51231016 | 48.86 |
| zstd_HC v0.3 -3 | 76 MB/s | 417 MB/s | 46774383 | 44.61 |
| zstd_HC v0.3 -5 | 40 MB/s | 476 MB/s | 45628362 | 43.51 |
| zstd_HC v0.3 -9 | 14 MB/s | 485 MB/s | 44840562 | 42.76 |
| zstd_HC v0.3 -13 | 9.34 MB/s | 469 MB/s | 43114895 | 41.12 |
| zstd_HC v0.3 -17 | 6.02 MB/s | 463 MB/s | 42989971 | 41.00 |
| zstd_HC v0.3 -21 | 3.35 MB/s | 461 MB/s | 42956964 | 40.97 |
| zstd_HC v0.3 -23 | 2.33 MB/s | 463 MB/s | 42934217 | 40.95 |

ShadowEagle · #6 05-12-2015, 02:07

Used this method:

I tried to extract with inno setup. I copied all files from "Resources" folder into includes folder in inno setup script and added the reflate files into the script:

Code:

Source: Include\reflate.exe; DestDir: {tmp}; Flags: dontcopy
Source: Include\rawrest.exe; DestDir: {tmp}; Flags: dontcopy
Source: Include\rawdet.exe; DestDir: {tmp}; Flags: dontcopy
Source: Include\raw2hif.exe; DestDir: {tmp}; Flags: dontcopy

But i get this error:

Pls help. What i`m doing wrong?

panker1992 · #7 05-12-2015, 04:21

Quote:

Originally Posted by RamiroCruzo

Lets compare ourselves which is better for our needs....

Source:Encode.ru

| Compressor name | Compression| Decompress.| Compr. size | Ratio |
| --------------- | -----------| -----------| ----------- | ----- |
| memcpy | 8533 MB/s | 8533 MB/s | 104857600 |100.00 |
| lz4 r131 | 480 MB/s | 2275 MB/s | 64872315 | 61.87 |
| lz4hc r131 -1 | 82 MB/s | 1896 MB/s | 59448496 | 56.69 |
| lz4hc r131 -3 | 54 MB/s | 1932 MB/s | 56343753 | 53.73 |
| lz4hc r131 -5 | 41 MB/s | 1969 MB/s | 55271312 | 52.71 |
| lz4hc r131 -7 | 31 MB/s | 1969 MB/s | 54889301 | 52.35 |
| lz4hc r131 -9 | 24 MB/s | 1969 MB/s | 54773517 | 52.24 |
| lz4hc r131 -11 | 20 MB/s | 1969 MB/s | 54751363 | 52.21 |
| lz4hc r131 -13 | 17 MB/s | 1969 MB/s | 54744790 | 52.21 |
| lz4hc r131 -15 | 14 MB/s | 2007 MB/s | 54741827 | 52.21 |
| lz5 r131 | 195 MB/s | 939 MB/s | 55884927 | 53.30 |
| lz5hc r131 -1 | 32 MB/s | 742 MB/s | 52927122 | 50.48 |
| lz5hc r131 -3 | 20 MB/s | 716 MB/s | 50970192 | 48.61 |
| lz5hc r131 -5 | 10 MB/s | 701 MB/s | 49970285 | 47.66 |
| lz5hc r131 -7 | 5.54 MB/s | 682 MB/s | 49541511 | 47.25 |
| lz5hc r131 -9 | 2.69 MB/s | 673 MB/s | 49346894 | 47.06 |
| lz5hc r131 -11 | 1.36 MB/s | 664 MB/s | 49266526 | 46.98 |
| zstd v0.3 | 257 MB/s | 547 MB/s | 51231016 | 48.86 |
| zstd_HC v0.3 -1 | 257 MB/s | 553 MB/s | 51231016 | 48.86 |
| zstd_HC v0.3 -3 | 76 MB/s | 417 MB/s | 46774383 | 44.61 |
| zstd_HC v0.3 -5 | 40 MB/s | 476 MB/s | 45628362 | 43.51 |
| zstd_HC v0.3 -9 | 14 MB/s | 485 MB/s | 44840562 | 42.76 |
| zstd_HC v0.3 -13 | 9.34 MB/s | 469 MB/s | 43114895 | 41.12 |
| zstd_HC v0.3 -17 | 6.02 MB/s | 463 MB/s | 42989971 | 41.00 |
| zstd_HC v0.3 -21 | 3.35 MB/s | 461 MB/s | 42956964 | 40.97 |
| zstd_HC v0.3 -23 | 2.33 MB/s | 463 MB/s | 42934217 | 40.95 |

Impressive but this test utilize the HC format of the zstd program
currently i test zstd v4.2

i have eaten all my day to test, test and test this :P

download the lz4 installer for windows and rename zstd ~~> lz4.exe and
replace in programs/lz4 folder

use this and you will see that decomp speed is all but fake i/o cant handle
2.2 gigs/sec because of limitation unless you have some x6 SSD array :P

felice2011 · #8 04-12-2015, 10:10

Great Bro ... at least we know that LZ4 and LZ5 is now out

ShadowEagle · #9 04-12-2015, 14:32

Yes i`m stupid and bad guy

Razor12911 · #10 04-12-2015, 16:58

Quote:

Originally Posted by ShadowEagle

Yes i`m stupid and bad guy

not really, you just have to engage with the program even further and use logic, so far so good, you are showing interests with this program only thing left is to make attempts to make the program work using logic then you'll get there.

gozarck · #11 04-12-2015, 18:58

Yes felice your tool is very powerfull and easy to use.you need to make the UI more atractive for newbies. Do you was tried to integrate metroframework.dll (metroUI) in yours projects?.

sayang123 · #12 05-12-2015, 05:49

Is there any installer script which support the unpack command from this app?

sorry for my bad english

felice2011 · #13 05-12-2015, 12:17

Quote:

Originally Posted by sayang123

is there any installer script which support the unpack command from this app?

Sorry for my bad english

In attachment it is an example of installing proprietary 'not InnoSetup' with a data file compressed using the reflate method 'reflate+srep+delta+lzma:a1:mfbt4:d158m:fb273:mc10 00000000:lc8'
For what concerns me and for my use decompression or installation or setup of a compressed data folder, with the excellent method reflate, works beautifully.

coming soon, new universal extractor in new thread

Lucas65 · #14 10-12-2015, 14:26

Hello, thank you felice2011 for this application. It is fantastic!

In my windows 7 64 bit is good but in another my PC with XP SP3 32 bit does not work with this error: "Split_Kompress.exe is win32 application invalid".

There is solutions to this problem?

Thank you in advance, sorry for my english and congratulations again for your work.

felice2011 · #15 11-12-2015, 00:35

Quote:

Originally Posted by Lucas65

Hello, thank you felice2011 for this application. It is fantastic!

In my windows 7 64 bit is good but in another my PC with XP SP3 32 bit does not work with this error: "Split_Kompress.exe is win32 application invalid".

There is solutions to this problem?

Thank you in advance, sorry for my english and congratulations again for your work.

The program is optimized for 64-bit systems with minimum 4GB of RAM, XP is now obsolete, and makes the most of 3.5GB of RAM as it is a 32bit system.
Even if it would work 32-bit its potential for compression would be reduced by 60%.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Full Method, Splitting, Finalizing, Compressing	felice2011	Conversion Tutorials	55	16-10-2021 02:50

The Following User Says Thank You to RamiroCruzo For This Useful Post:
felice2011 (04-12-2015)

The Following 3 Users Say Thank You to panker1992 For This Useful Post:
felice2011 (04-12-2015), RamiroCruzo (04-12-2015), romulus_ut3 (04-12-2015)

The Following 2 Users Say Thank You to felice2011 For This Useful Post:
panker1992 (04-12-2015), RamiroCruzo (04-12-2015)

The Following User Says Thank You to panker1992 For This Useful Post:
RamiroCruzo (04-12-2015)

The Following User Says Thank You to panker1992 For This Useful Post:
romulus_ut3 (05-12-2015)

The Following 2 Users Say Thank You to gozarck For This Useful Post:
felice2011 (05-12-2015), RamiroCruzo (04-12-2015)

The Following User Says Thank You to felice2011 For This Useful Post:
Simorq (06-12-2015)

The Following User Says Thank You to Lucas65 For This Useful Post:
felice2011 (11-12-2015)

The Following 2 Users Say Thank You to felice2011 For This Useful Post:
Lucas65 (11-12-2015), panker1992 (12-12-2015)