Another thing I thought of in this area... is there any way to compress RAW-ISO data? I know this is strongly tied to software piracy, but it was something I was always confused by. We can compress execuatables, text, music, images to a great degree, but raw data can't be compressed. WinRAR/ZIP rarely shave more than 1% off a cd-image of 800mb.

Is this because nobody has attempted it, or because there is a fundamental difference that makes cd-images uncompressable?

When I learnt of the failings of CD-R's in another thread, I thought of using CloneCD to take an image of the cd, storing it on my HDD. Thing is, the image is the full size of the cd's, it can't be compressed in the same way as ripping the music to waves then using MAC. ???

here's the link to UHA binaries
Enjoy
And maybe even post some good results

i just think it depends on the data on the cd.

if the data is compressable so will the image.
However you will offcouse not get mac compression on an iso audiotrack as
1: no generel compresser, compresses as well as mac on audio.
2: You have a lots of error checking data in between the audio destroying the "pattern" of the audio data (this goes for data tracks to)



but i have a nero image her which are compressed down to 59% of the original size.

--edit--
minor spelling errors

madah,

Could you run some of those test suites with just a "-t7z -mx -r" on them? I wonder how well a humane switch set would do instead of these suites that call for > 512 MB of RAM.

As for CD image compression: Many CDs (especially software) already have compressed data on them, so trying to compress the image doesn't get you much. Otherwise you'll need a compressor with a pretty large data dictionary to get good compression, because of the error checking bits and other stuff that appears in the datastream.

It would be interesting to see a specialized CD image compressor for handling things like keeping images on a PC and mounting as virtual drives. The only possible problem would be the compression extracting a toll on the CPU. Maybe with a good asymmetrical compression scheme, where decompressing is much less comlex than compression, the losses would be small enough that the overwhelming speed differential between CD and HD would win out. <ponder ponder ponder> Image mounting software would need some modification to handle this too... Hmm...

Here is a second link: http://dragoonhome.de/down/uharc04.zip

There you can download a GUI for UHARC (both in german & english).

Roughly the same set of .xm-modules I've tested on earlier (461 MB uncompressed, but this time organised into several folders). Time measured on 450 MHz celeron:



(*) = 7zn.exe a -t7z -r -y -m0=BCJ2 -m1=LZMA -m2=LZMA -m3=LZMA -mb0:1 -mb0s1:2 -mb0s2:3 -m1a=2 -m2a=2 -m3a=2 -m1d=16M -m2d=4M -m3d=4M -m1fb=128 -m2fb=128 -m3fb=128 archive.7z

(**) = uharc.exe a -m3 -md16384 -r archive.uha

WinRar 3 seems to be blazing fast, but it could be because of some problem with the timing...

Also there seems to be a problem when there's many files in many different folders; compared to the last test with exactly the same files:

zip: 2 MB larger
rar: 5 MB larger
7z: 17 MB (!) larger (was only 177 MB in the last test, compared to 194 MB in this one)

Edit: added uharc

Found a nice french site testing rar,7z,uharc: http://cmetge.dixinet.com/archive.asp

Sometimes 7-zip maximum compression (no special atributes) is better than with these atributes.

Madah, I thought -mx was the best compression UHA had to offer?

Thanks for the links to UHA, I searched some more before work, and came up with the same 0.4beta. The GUI is handy.

If people are still interested - I can do some testing in a week when I get a new PC. Everything on my current HD is compressed as it is to save space!!

I think UHA is at least a rival for 7Z, possibly better.


And on the CD Image business.. I hadn't thought about it for image mounting! I remember I had a use for it once, but never had the gig's for more than a few images. Maybe with compression it would be usable..

All I'm hearing though is "a cd image is raw data, you can't compress raw data"... I can never find out why though!!

this arguments sound like BS.

if the data has a repeatable pattern it can be compressed. its has nothing to due about it being "raw". data is data. only the pattern count about compressability.

How does it explain winrar 3 taking an 800mb ISO and making it 800mb? On maximum compression it does little or no different to on store mode (no compression) ?

Hey JuanK, thanks alot for that link. At last i have a UHA GUI .

you must be doing something worng or have you only tested with thise single is file ?

I just compressed a Clonecd image of 612mb down to 59% of original size with winrar3 (best/solid/4m)
the CloneCD was rip'ed with raw+96 reader. you just dont get more "RAW" then this (unlees you go do mdm which contains jitterinfo of the physical track)
I have taken a nero image down to 59% as well from 636mb with winace.

both these images are more "raw" then iso (iso is at a higher level)

-mx uses the PPM mode which is mostly useful only for text-files (I think).

-m0 = store
-m1 = ALZ:fast
-m2 = ALZ:default
-m3 = ALZ:best
-mx = PPM
-mz = LZP

For best possible compression with uharc you can try:

uharc a -r -y -m3 -md32768 archive.uha

But lots of RAM is needed, otherwise uharc will use a lower amount of dictionary, like 16 or 8 MB.



7z supports files and archives larger than 4 GB, filenames can be in unicode and it's open-source. Uharc has nothing of that...



This depends on the data. If there are installation files on the ISO they are certainly already packed with CAB or similar. What was the data you were trying to compress?

Oh, I thought from the document that PPM was the best compression mode for anything. I did a little test compressing a 2mb installer, PPM came out smaller than M3 (and than rar & 7z)..


I was just using a simple uharc a -mx archive.uha!

How much ram would -md32768 need? Are we talking the insane 512mb requirements I was seeing in some of 7Z's most hardcore modes?


When I say better, I mean purely by compression ratio. For day to day things, I'm going to stay with RAR, to me it is 'best' in terms of a simple GUI, multi-volumes, passwording, context menus, and better speed


Hehe, it could well be something done wrong.

I've tried it with an audio cd, games (compressed files), old games, which are runnable from the cd, and come with .wav music files! It was a while ago, but I remember getting no compression from anything.. I'll try again and see if I can get results like you

You're welcome.

http://sjeng.org/ftp/7zboost.exe

I had a try at optimizing things, extract the above into your 7zip program dir.

I didn't benchmark. If you do, please post here.

Garf,

When I try to use the 7zan.exe from your compile, I get a "libmmd.dll" not found. 7za.exe runs fine. Where can I get that file at?

That said, I did a VERY rough test with GhostScript 7.04. Clocked in at 1:10 for the default, and 0:54 for Garf's.

http://sjeng.org/ftp/libmmd.dll

It's from the compiler, I'm surprised it's needed.

I've been doing some comparisons between RAR 3 and 7zip and usually 7zip wins out in compression ratio, but very slightly, and it seems (quite a bit?) slower than RAR.

The better-than-zip zipper is very nice though.

BTW. Can anyone tell me how to configure Total Commander (EX-Windows Commander) to work with 7-zip?

Maybe consider sharing Your otptimization ideas with author. Next beta will be faster

just curios but how did you optimize this
faster compiler ? mmx/sse/sse2/3dnow optimizing ?


--- edit ---

i just tried with you optimized files and the ddl file.
when compressin files or foldes by the context menu (rightclick) it say "specified module not found" nd closes it self instead of starting to compress files

Works fine here, did you download the second file also? (see thread above)

It's compiled with Intel C 6 + profile feedback. The source needed several modifications before it would compile in the first place, but most of those are farily trivial.

BTW

7-zip forum:

http://sourceforge.net/forum/?group_id=14481

yes i dowloaded "libmmd.dll" into the 7-zip after i extracted the file from 7zip boost into the 7-zip folder.

my comp spec
dell latitude c810
p3 1.13ghz/733mhz tulatin cpu
Geforce2go 64mb
30gb toshiba disk
815 based motherboard
2x 256mb of pc133 ram cas3

-- edit --

finally got it to wrok be uninstalling and reinstalling a couple of times

more compressiin benchmarks


33 folders and 396 .jpg's = 87,2mb

.rar = 87.1mb
.7z = 87.8

using
0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=16M 2d=16M 3d=16M 1fb=64 2fb=64 3fb=64
This was as high as i could get with only 512mb of ram on my laptop

is still seems 7z is a no go for already compressed data. the size actually GREW. (please also se my last test with .aac files)

...this is a bad configuration.
Try

ok thanx.

I just stole the linie from Zombiek at page 4
0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=16M 2d=16M 3d=16M 1fb=64 2fb=64 3fb=64


i question.
what the hell are all those parameters standing for ?

i kinda got the
1d 2d 3d 4d = dictionary size
1fb 2fb 3fb = fast bytes (whats is that)

whare are the rest

and why is there no FB in you line ?

I'm just DARN curios :-)


some more benchmarks


4282 files og 22 folders = 5.43gb
rar = 1.12gb
7z = 877mb

this is based on
0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=16M 2d=16M 3d=16M 1fb=64 2fb=64 3fb=64

it shaved aditionalle 260mb of the file then winrar. Nice

i just teste zombiek vs sk1's line

SK1 (0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1d=25 2d=20 3d=20 1mf=bt4b 2mf=bt2 3mf=bt2)
28.1mb


Zombiek (0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=16M 2d=16M 3d=16M 1fb=64 2fb=64 3fb=64)
72.mb


original size 91.2mb

The reason why SK1's line was so much better could be that it uses 32M instead of 16M dictionary.
Can you try this line, if you have ~702 MB ram:

0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=64M 2d=4M 3d=4M 1fb=64 2fb=64 3fb=64



I suggest reading the 7-zip manual, section "Command Line Version / Switches / -m (Set Compression Method)"

1a=2 2a=2 3a=2
Means to use maximum compression for that stream:
a=[0|1|2]
Sets compressing mode: 0 = fast, 1 = normal, 2 = max ratio. Default value is 1.

1d=25 1d=32M
Means dictionary size, in this case both are 32M. 25 = 2^25 = 32M.
Default value for LZMA is 20 (1MB) in normal mode and 22 (4MB) in maiximum mode (-mx).
When BCJ2 is used dictionary sizes for methods 2 and 3 can be set to 1/8 of dictionary size of method 1.

mf={MF_ID}
Sets Match Finder for LZMA. Default method is bt4.
Memory requirements: d*9.5 + 6MB, where d = dictionary.
For the line at the top: 32*9.5+6 + 4*9.5+6 + 4*9.5+6 = 614 + 44 + 44 = 702 MB of memory.

0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3
I think this means which compression mode each stream will use and how they are connected. It is explained in the manual that BCJ2 has one input stream and four output streams.

1fb=64 2fb=64 3fb=64
Sets number of fast bytes for LZMA. It can be in range from 5 to 255. Default value is 32 for normal mode and 64 for maximum mode. Usually big number gives a little bit better compression ratio and slower compression process.


Edit: Quoted some from the 7-zip manual.

thanx for the info.


I'll chew on them later




-- edit --
finnish chewing adn it helped a little

if
1a=2 2a=2 3a=2
Means to use maximum compression for that stream:
and
Default value is 1.

why are sk1''snot having them ? are they runnig with only 1 or by 2 (if u have chossed maximum in the gui)


-- edit --

BTW

does someone noe som switches that might decrease decoding time at coast of encoding time/ sligty bigget size

encoding time is not an importne issues for me
size and extracting time IS.
as I'm compiling driver cd's for mig works which alon uses 410mb of nvidia drivers.

i only have 650mb of space to use for all my drivers and test programs

sofar im using
0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=32m 2d=1m 3d=1m 1mf=bt4b 2mf=bt2 3mf=bt2

if thise can ber improved for faster extraction please let me know

Nobody else benchmarking UHA ?

I've done a small test... but it took my system so long I got bored of cleaning up my flat!


I might get accused of not giving 7Z a fair go by using the 20 switch command line, but I have 256mb of ram and so can't use a enourmous dictionary!


I think it's fair to say, out of the 3, they are pretty similar!! UHA gave the best compression and least memory usage, WinRAR had the fastest compression & nicest features, 7Z had the fastest and least demanding decoding.

Personally, until a *BIG* drop in compression without loss of speed & frills I won't be switching from Rar



-=-=-=-=[my test]=-=-=-=-

Original Files :

277.1mb - Quake 2 / ICQ folders

1276 files / 97 folders (exe/dll ~13% -=- zip/pak ~24% -=- jpg/avi/mp3 ~8% -=- wav/tga/bmp ~28% -=- bsp/dm2 ~25%)



WINRAR 3.0 beta 2 (compression - best, solid, 4096kb dictionary, optimizations on auto or checked)

-Compression 124.3MB (44.9%)
-Time : ~9 min
-Memory : ~50 mb needed

-Decompression : time ~1min 30
-Memory : ~30mb needed



7-ZIP 2.30 beta 4 (compression - maximum, solid)

-Compression 120.8MB (43.6%)
-Time : ~18 min
-Memory : ~50mb needed

-Decompression : ~1min 15
-Memory : ~8mb needed



UHARC beta 0.4 (compression - ppm, solid, 4096kb dictionary)

-Compression 116.0MB (41.9%)
-Time : ~31 min
-Memory : ~25mb needed

-Decompression : ~15 min
-Memory : ~25mb needed

I'm using the following command "presets" for the different memmory usage/availbility)
there are all offspring of SK1 recomendet line but with adhustet dictionary size

(128 MB)
0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=6m 2d=19 3d=19 1mf=bt4b 2mf=bt2 3mf=bt2

(256 MB)
0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=16m 2d=1m 3d=1m 1mf=bt4b 2mf=bt2 3mf=bt2

(384 MB)
0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=32m 2d=1m 3d=1m 1mf=bt4b 2mf=bt2 3mf=bt2

(512 MB)
0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=48m 2d=2m 3d=2m 1mf=bt4b 2mf=bt2 3mf=bt2


But i think there might be wasted memmory on the sorting algoritme in the low memmory "presets", that might be better used for dictionary size.
Does someone have a better 256 or 128mb preset. or maybe a 64mb preset ?


maybe we should have a preset switch like the --alt-preset. Something like SK1=xxx wher xxx is the size of memmory usage :-D

I just wanna say that 7-zip has slightly better compression and speed now:
"Some optimization recompiling for code size reducing."
"In max mode 7z now uses 8 MB dictionary instead of 4 MB."

Also:
"Self-extracting capability for 7z format."
AES encryption for files.

And more

Look there: http://www.7-zip.org/

Edit: If you want really best ZIP compression, use additional switches: fb=255 pass=4.

it might be the best compression software, but it sure is slooooooooooooooooooooooooooooooooooow

w4r3z kiddies don't care about compression speed, and that's where 7z rules

WinRar gives the use the ability to split the file after a given size. Is this possible with 7z?

This option will be supported soon...

As soon as I find out that this option is supported, I will remove WinRar from my system and use 7z on its own!

Funny - sometimes tar.7z is better than normal .7z

7zip doesn't have native solid archive support?

It does, which is what makes it funny.

I'm guessing file ordering may be the problem, i.e. sometimes tar is more lucky than 7z in ordering the files for good compression.

is there any visual indication to how it's grouping them? Does it do like WinRAR and group them by extension and then alphabetically? Or just throw them all in the mix?

Best known line for compressing Foobar2000:
7z.exe a -t7z foobar2000.7z -m0=BCJ2 -m1=LZMA -m2=LZMA -m3=LZMA -mb0:1 -mb0s1:2 -mb0s2:3 -m1a=2 -m2a=2 -m3a=2 -m1d=4M -m2d=4M -m3d=4M -m1fb=128 -m2fb=128 -m3fb=128 -mhcf=on -m1mf=bt4b fb2k_winnt\* fb2k_win9x\* extra\*

Best for packing PSI Jabber client:
0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=8M 2d=8M 3d=8M 1fb=128 2fb=128 3fb=128 hcf=on 1mf=pat3h with solid and "maximum"

Lol! W4r3z kiddies want compression rate, which is what 7-zip.org claims 7z gives

QUAKE2CD:\install\data\*.*
(the instructions for installing q2 on win2000 is to copy this dir to your harddisk!)

Didn't measure speed or memory this time round, just pure size. The 7-zip.org quote was annoying me after a friend was trying to convince me that 7z is the highest indeed the highest compression

RAR = 46.86%
7Z = 44.62%
UHA = 43.35%

As previously stated, UHA ()wnt j00!

However, as the smug friend pointed out... this was a 388mb uncompressed archive, and the benefit of UHA over 7Z was 4.95mb

And in the extra time it took to encode the 7z file over the rar file... I could of sent him 15mb, when 7z only saved 8.5mb Swings and roundabouts!

I'm a smug w4r3z kiddy tho for now


RAR = max compression, solid, all special compression included
7z = 0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1d=25 2d=20 3d=20 1mf=bt4b 2mf=bt2 3mf=bt2 (SK1's line)
UHA = PPM 4mb dictionary

Try 0=BCJ2 1=LZMA 2=LZMA 3=LZMA b0:1 b0s1:2 b0s2:3 1a=2 2a=2 3a=2 1d=16M 2d=16M 3d=16M 1fb=128 2fb=128 3fb=128 hcf=on 1mf=bt4b (!)

IIRC the author of 7-zip said it's not necessary to make all d-Values (1d, 2d, 3d) equal. Instead of
1d=16M 2d=16M 3d=16M it should be better to use something like
1d=32M 2d=2M 3d=2M (or 32M/1M/1M as in SK1's CL)

Have you tried this?

16 8 8 for FB2K archive was WORSE than 4 4 4

That line ends up bigger by 0.8mb...

174,358,723 instead of 173,552,904

Once my friend has recieved the archive, I'm sure he won't sleep until he's found a line that beats UHA

Ok you tried compressing one set of files with 7-zip and UHA. Have you done any others? Before you get all smug perhaps you should try more file sets. There are sets that RAR beats UHA and 7-zip. Likewise there are sets that 7-zip beats RAR and UHA. And then of course there are sets that UHA beats RAR and 7-zip. But quite often it seems 7-zip does a bit better than others. But not always mind you.