FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   Helping thread :) (https://fileforums.com/showthread.php?t=98274)

MCWRX 13-10-2016 13:51

Hey guys. I am in a bit of a problem right here, this happened to me 2 times since I was compressing game files. I have 5 archives. First 3 unpack without any problems, then when it comes to the 4th one, at very ending of it I get this message - http://funkyimg.com/i/2i9Kc.png The 5th archive unpacks without any errors. I happened to me once before, I couldn`t manage to realize what it was, maybe you could help me?

I was packing with
Code:

-msrep:m3f:a16:l512+delta+lzma:a1:bt4:d400m:fb273:mc10000000000:lc8:lp2:pb2
Other archives use pretty much the same algorithms, on some of them I used precomp if it matters.

aswadd 13-10-2016 17:37

Quote:

Originally Posted by MCWRX (Post 452794)
Hey guys. I am in a bit of a problem right here, this happened to me 2 times since I was compressing game files. I have 5 archives. First 3 unpack without any problems, then when it comes to the 4th one, at very ending of it I get this message - http://funkyimg.com/i/2i9Kc.png The 5th archive unpacks without any errors. I happened to me once before, I couldn`t manage to realize what it was, maybe you could help me?

I was packing with
Code:

-msrep:m3f:a16:l512+delta+lzma:a1:bt4:d400m:fb273:mc10000000000:lc8:lp2:pb2
Other archives use pretty much the same algorithms, on some of them I used precomp if it matters.

I don't know Much about inno but did you try to extract using cmd to know exactly where the problem is ??
arc x archive.arc --adddir
if specific algorithm is causing this error arc will tell you
if it extracted without any problem so the problem is from inno script itself :)

MCWRX 14-10-2016 01:37

Quote:

Originally Posted by aswadd (Post 452801)
I don't know Much about inno but did you try to extract using cmd to know exactly where the problem is ??
arc x archive.arc --adddir
if specific algorithm is causing this error arc will tell you
if it extracted without any problem so the problem is from inno script itself :)

Thank you. Now I will know how to test this.

But it seems that the operation succeeded via the CMD, no errors :)

What problem can there actually be in my script, is there anyone who would like to take a look? The other 4 archives are unpacking without any errors. Strange.

aswadd 15-10-2016 03:56

Quote:

Originally Posted by MCWRX (Post 452806)
Thank you. Now I will know how to test this.

But it seems that the operation succeeded via the CMD, no errors :)

What problem can there actually be in my script, is there anyone who would like to take a look? The other 4 archives are unpacking without any errors. Strange.

try to make archive number 4 and 5 one archive :) so that you'll have only 4 archives :D Just ignore the problem you can't solve
Update:
-msrep:m3f:a16:l512+delta+lzma:a1:bt4:d400m:fb273:m c10000000000:lc8:lp2:pb2
you are using large dictionary in your lzma :D the difference will be a couple of megas
& For Srep you are using a16 why not using a64 it's the fastest & best ratio ~ same ratio as a0/a1 :D ~

aswadd 15-10-2016 04:46

Guys , I need to understand something
Precomp+msc
msc+precomp
what is the difference ??
is Precomp+msc better to be sure that msc find all media & compress it ?? or Just leaving it msc+precomp because msc will also find all media but will be faster because input is smaller than precomp input ??

SAM2712 15-10-2016 23:17

Quote:

Originally Posted by aswadd (Post 452830)
Guys , I need to understand something
Precomp+msc
msc+precomp
what is the difference ??
is Precomp+msc better to be sure that msc find all media & compress it ?? or Just leaving it msc+precomp because msc will also find all media but will be faster because input is smaller than precomp input ??

Hello aswadd :D
I think Precomp+msc do the best :rolleyes: As precomp do decompress so msc piping is easy to find out maximum media files with effective speed of finding:D

Razor12911 15-10-2016 23:31

^Exactly what he said.

But it's general knowledge, think of it this way.
can deflate streams store media streams? yes, deflate is a compression algo and it can contain media streams.
can media streams store deflate streams? no, media streams is not a compressor or a packer of some sort therefore it cannot contain deflate streams.

ChronoCross 16-10-2016 12:21

Testing only One idea in fa Next with LUA code.
The first idea:
I take a folder with 62MB into -m0 option "data.arc"
and then split the file with razor's split tool in three parts.
data.part1 30 mb
data.part2 30 mb
data.part3 2 mb
OK well the first idea is said to fa next with LUA code:
that's this three files are "data.arc"(with a trick or patch i don`t know).
well the problem is fa next
search the file in directory and seek all the file.
doesn`t see this three files like into one("data.arc").
The principal idea is not join the files is create an emulation of file or something like that(and you know what?, i donīt know how to do :D ).
but i fail because, fa next needs data.arc present in the folder and with his size(and his correct hashes).
OK this is the LUA code for fa.ini to join the files (without Split.exe).
Quote:

onProgramStart(function ()
a = "none"
f = command.arcname
f = io.open(f, "a")
f:write()
f = io.open("data.arc", "r+b")
f1 = io.open("data.part1", "rb")
f2 = io.open("data.part2", "rb")
f3 = io.open("data.part3", "rb")
f4 = f1:read("*all")..f2:read("*all")..f3:read("*all")
t = io.output (f:write(f4))
f = assert (io.open("data.arc", "r"))
f:close()
end)
and the code for extract in cmd

Quote:

fa x -dp.\ data.arc
well another problem here (help) you need to run the bat two times.
the first is an error like data.arc is not a valid fa next file.
because the code first create data.arc with 0b.
but the second time you run the bat, fa next extract correctly because the program found a joining data in data.arc.
If someone have an idea to fix this or help to the first idea please go on!

gozarck 16-10-2016 16:43

HI ChronoCross Nice idea and a litle complicated for me :D
the first Idea is awesome, but i can't help you :( .
I started to read LUA and looks easy to undertand.
Quote:

Originally Posted by ChronoCross (Post 452895)
well another problem here (help) you need to run the bat two times.
the first is an error like data.arc is not a valid fa next file.
because the code first create data.arc with 0b.
but the second time you run the bat, fa next extract correctly because the program found a joining data in data.arc.
If someone have an idea to fix this or help to the first idea please go on!

let me help you in this part.
add some lines in your fa.ini code repeat and until ~=
Code:

onProgramStart(function ()
    repeat
        a = "none"
    f = command.arcname
    f =  io.open(f, "a")
    f:write()
    f =  io.open("data.arc", "r+b")
        f1 = io.open("data.part1", "rb")
        f2 = io.open("data.part2", "rb")
        f3 = io.open("data.part3", "rb")
    f4 = f1:read("*all")..f2:read("*all")..f3:read("*all")       
        t = io.output (f:write(f4))
        until f ~= assert ( io.open("data.arc", "r"))
        f:close()
end)

Now can join and extract the files correctly. :)

Bulat 17-10-2016 07:18

btw, if you not found it - the classic Lua learning book is http://www.lua.org/pil/contents.html

it's third edition is available online ;)

ChronoCross 17-10-2016 19:00

i cleaning the code
Quote:

onProgramStart(function ()
f = io.open(command.arcname, "wb")
f1 = io.open("data.part1", "rb")
f2 = io.open("data.part2", "rb")
f3 = io.open("data.part3", "rb")
f4 = f1:read("*all")..f2:read("*all")..f3:read("*all")
t = io.output (f:write(f4))
f:close()
end)

gozarck 20-10-2016 15:40

HI @ChronnoCross Thanks for the code. Well i was thinking and reading LUA code and maybe i resolve your request(with minimal knowledge).
--how to read parts without merge those files?
Well the only trick i made is send the data to Buffer.
Example:
FA Next request the file to extract, so the code create the file "data.arc" with 0MB(like a temp file but isn't)
and send the data of data.part1, data.part2 and data.part3 to buffer. FA Next read the collected data from buffer and extract the files.
the file data.arc always have 0MB because with the LUA code "flush" you flush the buffer.
Code:

onArchiveStart(function ()
        f =  io.open(command.arcname, "wb") 
        f1 = io.open("data.part1", "rb")
        f2 = io.open("data.part2", "rb")
        f3 = io.open("data.part3", "rb")
        f4 = f1:read("*all")..f2:read("*all")..f3:read("*all")       
        io.output(f:write(f4)):setvbuf("full")       
end)

onArchiveDone(function ()
    f =  io.open(command.arcname, "wb") 
        f:flush()
end)

maybe we need to check the code or add some more lines :)

ChronoCross 20-10-2016 17:05

Wow awesome gozarck now we don't need extra space to extract multi parts!. Works great! You emulate the arc file or whats?.
I made a test in my usb flash drive: i was fill it to the limit of space and then i added the files to extract with 10mb of free space in the usb flash drive to test if the file data.arc increase his size but not!!! Works perfect! Also extract correctly.You create multiparts extraction for faNext!!

gozarck 21-10-2016 08:14

We need to work a little more with the code to add features like unpack from DVD , remaining time for while collecting data, request part if some part are in diferent folder, make the code for IS. Also we need to add the option exclude full path -ep1 with lua code for fa next.
----Update------
Unpack from DVD or iso DONE!
----------------------
For unpack from dvd you need The
data.parts fa.ini fa.exe and unpack.bat
example:
-------------------
data.part1
data.part2
data.part3
fa.ini
fa.exe
Unpack.bat
------------------
the trick is set data.arc file in temp folder.(%temp%\data.arc)
----Unpack.bat----------
Code:

fa x -dp"D:\MyGame" -o+ %temp%\data.arc

ChronoCross 21-10-2016 18:54

This is nice gozarck!. From DVD - ISO works greats!
Now we need a code to "request part if some part are in diferent folder" or another DVD!.
I don`t know how to add "-ep1" option in LUA code.
If someone knows please help us.

ChronoCross 22-10-2016 10:31

Oh no... My test are wrong!. Data.arc need extra space!.

Blade2239 24-10-2016 22:30

http://fileforums.com/attachment.php...3&d=1477332855
Anyone help me with this........ Please.....

Blade2239 27-10-2016 11:37

Anyone can tell me more forum link which do work like fileforms
Like conversion tutorial and more.....
pakrat2k2 you may know......
For compression I know encode.ru
Please

aswadd 27-10-2016 12:22

Quote:

Originally Posted by Blade2239 (Post 453179)
Anyone can tell me more forum link which do work like fileforms
Like conversion tutorial and more.....
pakrat2k2 you may know......
For compression I know encode.ru
Please

Most of then are Russian forums I don't remember the names but search google & you will find them

oltjon 27-10-2016 12:53

Quote:

Originally Posted by Blade2239 (Post 453179)
Anyone can tell me more forum link which do work like fileforms
Like conversion tutorial and more.....
pakrat2k2 you may know......
For compression I know encode.ru
Please

http://krinkels.org/ russian forum godluck :D

Blade2239 28-10-2016 00:41

Quote:

Originally Posted by oltjon (Post 453184)
http://krinkels.org/ russian forum godluck :D

Can I get more option beacuse Google sending me to warz groups.....

And anyone can tell me which is the most beautiful installer is available or made .... Like we made with ic asc and other....

Which one of the latest codex/plaza installer

Is anyone know how fitgirl installer looks......

Edit:
On krinkels most user are not active from a while!!!!!!

Edit:
Why fileforms user are not online. is everyone's exam start exactly together........

Edit:
Can anybody tell me how to extract or get source of wpi.....
There's an innoextract but it just give DLL exe file no code or source

MCWRX 30-10-2016 05:44

Hey guys. Can someone tell me why ISDone tells me that there is Unsupported compression method when I am using srep+delta+exe2+lzma.

The problem is with exe2, but I don`t understand why. When I use exe instead, it`s ok, but with exe2 I can get rid of some kbs.

aswadd 30-10-2016 06:01

Quote:

Originally Posted by MCWRX (Post 453274)
Hey guys. Can someone tell me why ISDone tells me that there is Unsupported compression method when I am using srep+delta+exe2+lzma.

The problem is with exe2, but I don`t understand why. When I use exe instead, it`s ok, but with exe2 I can get rid of some kbs.

why even using exe :D for only 1 or 2 megabytes ?? & For delta don't use it unless you make sure it will give u better ratio

BTW this problem is related to arc.ini there is something wrong with it
& if you are using fazip for exe & delta & lzma make sure you are including it's configuration is arc.ini also don't forget to include fazip.exe & dll files

MCWRX 30-10-2016 06:06

Quote:

Originally Posted by aswadd (Post 453276)
why even using exe :D for only 1 or 2 megabytes ?? & For delta don't use it unless you make sure it will give u better ratio

BTW this problem is related to arc.ini there is something wrong with it
& if you are using fazip for exe & delta & lzma make sure you are including it's configuration is arc.ini also don't forget to include fazip.exe & dll files

I just had one executable and a bunch of dll`s and ran them through 16 algorithms, and chose the one with smallest size.

I just wanted to know what is causing the problem, to know where to dig.

What can be wrong with .ini? Do you have any clue?

Thanks.

p.s - Through console it`s running fine, damn.

VickNet 05-11-2016 11:35

Hi everyone. For a long time I`ve been searching how to make the unpacking process faster. There is one guy whose repacks are unpacking very quickly, comparing to mine or others.
http://funkyimg.com/i/2jewZ.png
By looking at this screen we can see the (almost) full CPU load and high load on my SSD (on HDDs the load is also high. It can go higher depending on type of SSD/HDD). The same pattern is seen in CODEX / RLD setups.
I couldn`t come up with such results. I tried to take a look in the temp folder of that setup, and haven`t seen anything new. Just saw facompress_mt.dll which did not give me any good.
Is there anyone who can point me to the right direction or these are some secret techniques? Thank you.

FitGirl 05-11-2016 12:38

arc.exe -m4x4:lzma

VickNet 06-11-2016 00:15

Thx, but is it possible to combine 4x4:lzma with other parameters? Like this -m4x4:lzma:a1:mfbt4:d400m:fb128...
- I tested like this but the process of packing is not starting

FitGirl 06-11-2016 06:17

The example string works just fine
srep+4x4:t4:b128m:lzma:128m

Never use large dict size for 4x4
4x4 means you're running 4 instances of LZMA, each with 400Mb dict size, that leads to 400*4*11 = 17600 MB of RAM to compress.

On user machine ISDone will require four continuous memory blocks of 400 MB each.

Also, using 4x4 hurts compression ratio due to blocking. That's why "some guy" repacks are larger than any other. You can try to use LZMA2 instead of LZMA1 (gives you 2 decompression threads by default), but it has it's own downsides.

I've personally used 4x4:LZMA in some of my early repacks, but gave it up. Number of SSD users is not that high, and unpacking in four streams on HDD often leads to overload.

aswadd 06-11-2016 07:53

as fitgirl said 4x4 is not good to use for public repacks you can use it for your personnel repacks it require 4X what normal lzma require but also faster ,
one more thing RLD didn't use 4x4:lzma they used 4x4:tor :D you don't have to use 4x4 for lzma only ;) but it will be a good option for a fast storing with a light compression ratio :)

MCWRX 07-11-2016 09:56

I tried this string
Code:

srep+4x4:t4:b128m:lzma:128m
but I am not able to unpack it.

aswadd 07-11-2016 10:49

Quote:

Originally Posted by MCWRX (Post 453660)
I tried this string
Code:

srep+4x4:t4:b128m:lzma:128m
but I am not able to unpack it.

why you're making it like this 4x4:t4:b128m:lzma:128m
why not 4x4:lzma:b128m:128m :D

if you used fazip for 4x4 , lzma then you must use it again for unpacking
see (arc.ini)

MCWRX 07-11-2016 11:00

Quote:

Originally Posted by aswadd (Post 453663)
why you're making it like this 4x4:t4:b128m:lzma:128m
why not 4x4:lzma:b128m:128m :D

if you used fazip for 4x4 , lzma then you must use it again for unpacking
see (arc.ini)

thanks, I will test.

I use LZMA x64

aswadd 07-11-2016 11:12

Quote:

Originally Posted by MCWRX (Post 453664)
thanks, I will test.

I use LZMA x64

one here mentioned that don't use lzmax64.exe it's not good i don't remember why but he said that as he tested :D use 7z lzma2 (xz) or fazip for (4x4,lzma)

MCWRX 11-12-2016 13:57

Hi everyone. What can the problem that after using precomp, the decompression process starts very slow? I press "INSTALL" and it stays at 0% for 2-3 minutes then precomp starts. It is not very good I think. Is there any solution?

Bulat 19-01-2017 03:22

Quote:

Originally Posted by MCWRX (Post 454382)
Hi everyone. What can the problem that after using precomp, the decompression process starts very slow? I press "INSTALL" and it stays at 0% for 2-3 minutes then precomp starts. It is not very good I think. Is there any solution?

it's ok - precomp is in beginning of your compression string, i.e. smth like "precomp+srep+lzma", and this means that on decompression it's last algo in the line, so all other compressora are done when precomp starts

still, you can use cls_precomp and cls_srep to make all decompressors work simultaneously

MCWRX 20-01-2017 13:37

Quote:

Originally Posted by Bulat (Post 455312)
it's ok - precomp is in beginning of your compression string, i.e. smth like "precomp+srep+lzma", and this means that on decompression it's last algo in the line, so all other compressora are done when precomp starts

still, you can use cls_precomp and cls_srep to make all decompressors work simultaneously

Thanks for your response!

I've got something else - what can cause this error: Not enough memory. unarc.dll returned error code - 5. It appears for some users. It appears when files compressed with 4x4:lzma start to unpack. Yes, I know, you guys told me it's not very ok to use it but what alternatives do I have? It's really fast comparing to default compression. I personally haven't got this error, it's rare, but the fact that it exists, annoys me and I can't move on without making it clear. Have any of you got ideas?

Bulat 21-01-2017 02:33

well, it may be bug in my program (unarc.dll) or yiu use it wrong way. so, let's start with lt command output and computer that fails. can you make some checks on this computer so i can understand the situation?

write me to Bulat.Ziganshin********com since i rarely read this forum

Bulat 21-01-2017 03:05

Quote:

Originally Posted by FitGirl (Post 453612)
The example string works just fine
srep+4x4:t4:b128m:lzma:128m

Never use large dict size for 4x4
4x4 means you're running 4 instances of LZMA, each with 400Mb dict size, that leads to 400*4*11 = 17600 MB of RAM to compress.

On user machine ISDone will require four continuous memory blocks of 400 MB each.

Also, using 4x4 hurts compression ratio due to blocking. That's why "some guy" repacks are larger than any other. You can try to use LZMA2 instead of LZMA1 (gives you 2 decompression threads by default), but it has it's own downsides.

I've personally used 4x4:LZMA in some of my early repacks, but gave it up. Number of SSD users is not that high, and unpacking in four streams on HDD often leads to overload.

i agree with almost everything you said except for two things:

1. 4x4 doesn't mean "use exactly 4 threads both for compression and decompression". let's see - data are split into independently compressed blocks, so you can use any number of threads at compression stage and, independent of that, any number of threads at decompression stage. both arc and unarc find largest contiguous block available (in 64-bit windows it will be 2047 MB), divide its size by the memory required for single (de)compression thread, and run so many threads. so, on 64-bit windows it will use more threads, on 32-bit windows it will use less threads

2. lzma2 is different from lzma in two things:
- it stores incompressible chunks of data
- like 4x4, it can split input data into independent blocks. unfortunately, it always uses only 1 thread for decompression. When manual says "LZMA2 uses: 1 thread for each chunk in x1 and x3 modes; and 2 threads for each chunk in x5, x7 and x9 modes" it means only comperssion stage

So, compared to 4x4:lzma, lzma2 may give slightly better ratio and single-thread speed, but it can't decompress even in 2 threads. but probably you meant compression w/o splitting into independent blocks, where lzma2 indeed is slightly better, but still single-threaded on decompression side. unfortunately, it's inherent property of lzma algorithm - you can't use morte than 1 thread to decompress single lzma compressed stream

3. that said, inability to decompress data compressed with 4x4:lzma is a bug in my program. I already have a better algorithm for InsertTempfile procedure, but ATM it's not yet ideal and still wrtitten in Lua. But i will fix that sooner or later.

4. these days, most of user computers run Win64, so probably it's time to switch to 64-bit unarc.dll in most repacks? i will try to build one.

Bulat 21-01-2017 03:11

Quote:

Originally Posted by aswadd (Post 453663)
if you used fazip for 4x4 , lzma then you must use it again for unpacking
see (arc.ini)

it's opposite - fazip can implement any internal arc.exe algorithm as external one. this means that you can compress with fazip and decompress internally, or vice versa.

in practice, i recommend to use fazip for large compression methods (rep, lzma, ppmd). you may also prefer to use fazip for decompression (even in installer) since it's yet another way to break 2/4 GB barrier of 32-bit programs

MCWRX 21-01-2017 07:29

Bulat, send me please a PM with your email.


All times are GMT -7. The time now is 12:45.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com