Hydrogenaudio Forums > WavPack 4.41 beta available

bryant

Feb 27 2007, 23:11

QUOTE(Synthetic Soul @ Feb 27 2007, 00:03)

I don't really understand the MMX, SSE stuff; however would I be right in saying that, due to the integer arithmetic nature of WavPack, SSE code is pointless? Anything else up your sleeve?

Right. Because WavPack encoding is integer-only (and generally not parallelizable) there's little or nothing to be gained by SSE and his brothers. MMX should be able to make a nice improvement because the left and right channels can be done together, but because of some optimization available for 16-bit data that doesn't work with MMX, the MMX stuff only really helps with 24-bit (and float) data.

I did get an idea from one of the FLAC 1.1.4 improvements that might be applicable to WavPack and I'll give that a try. You never know when something's going to make a big difference...

QUOTE

Anyway, I'm sure all your users will agree, the extra benefit provided by this beta with no loss in compression is extremely welcome. Many thanks for your continued efforts.

Ah, well it's my pleasure...

DARcode

Feb 28 2007, 08:21

QUOTE(Synthetic Soul @ Feb 27 2007, 10:03)

[...]Anyway, I'm sure all your users will agree, the extra benefit provided by this beta with no loss in compression is extremely welcome. Many thanks for your continued efforts.

Definitely extremely welcome and surely worth thanking whole heartedly for.
Haven't begun re-encoding from 4.3x to 4.4x for better DAP support yet, so the added speed is greatly appreciated.

DARcode

Mar 5 2007, 08:48

QUOTE(bryant @ Feb 28 2007, 07:11)

[...]I did get an idea from one of the FLAC 1.1.4 improvements that might be applicable to WavPack and I'll give that a try. You never know when something's going to make a big difference...[...]

Are you going to try and include the above into the 4.41 changes or can we expect a separate release please?
Thanks a bunch once more for your time and brilliant work.

bryant

Mar 5 2007, 23:30

QUOTE(DARcode @ Mar 5 2007, 06:48)

QUOTE(bryant @ Feb 28 2007, 07:11)

[...]I did get an idea from one of the FLAC 1.1.4 improvements that might be applicable to WavPack and I'll give that a try. You never know when something's going to make a big difference...[...]

Are you going to try and include the above into the 4.41 changes or can we expect a separate release please?
Thanks a bunch once more for your time and brilliant work.

I tried it the other day but wasn't able to make a big enough improvement for how ugly the change is. I'll work at a little more, but I'm not sure it's going to work out.

bryant

Mar 11 2007, 15:45

I have created a second beta that I believe makes another worthwhile improvement in performance. This involves accessing the bitstream in 16-bit chunks instead of 8-bit chunks, so it should affect the faster modes more (at least on a percentage basis).

Like the first beta, it should be bit identical to 4.40.0 in all circumstances.

Thanks in advance for any testing...

Here it is.

Madman2003

Mar 11 2007, 17:05

Some sourcecode would be nice for us non-windows users.

Mangix

Mar 11 2007, 17:06

the svn server is at http://svn.slomosnail.de/wavpack

grab subversion(if you don't already have it) and get it.

A_Man_Eating_Duck

Mar 11 2007, 19:56

i'm just wondering if there is any harm having the --optimize-mono to be on by default?

Mangix

Mar 11 2007, 21:08

IIRC, --optimize-mono only breaks compatability with any version of WavPack below 4.31

shadowking

Mar 12 2007, 02:03

--optimize-mono is also buggy in lossy mode - bitrate drops , though only on dual-mono content.

A_Man_Eating_Duck

Mar 12 2007, 02:35

i did a quick comparison using the --optimize-mono switch and without it in lossless mode, and the file sizes are the same, this is on normal stereo audio.

shadowking

Mar 12 2007, 02:39

It only kicks in on dual-mono content or similar, so its safe to use in lossless if your player doesn't need < 4.31 decoder. From what I understand is that there is a new decoder that fixes this without the hack and will be used at some point in the future.. or maybe the --optimize hack will hardcoded instead ?

bryant

Mar 12 2007, 13:21

QUOTE(shadowking @ Mar 12 2007, 00:03)

--optimize-mono is also buggy in lossy mode - bitrate drops , though only on dual-mono content.

Have you actually seen this lately? I thought I fixed it before I released 4.40 but I never mentioned it in the changelog because it (the bug) never made it into a real release.

Now that the DirectShow filter is based on the latest library, the only decoder that can't handle the new mode is the XMMS plugin that I provide. Once I figure out how to build that with the latest library I could make the --optimize-mono the default, with perhaps a new switch to turn it off (for maximum compatibility).

Unless you're using the XMMS plugin (or are encoding files for a large audience), it's perfectly safe to use all the time.

Madman2003

Mar 12 2007, 16:25

Encoding speeds are amazing compared to flac, wavpack -hh seems much faster than flac -8 (order of 4 difference), decoding is about half the speed (only 40% difference with mode -h compared to the 100% difference of -hh). Good work, wavpack has the potential to beat flac. Keep up the good work.

shadowking

Mar 12 2007, 17:51

QUOTE(bryant @ Mar 13 2007, 05:21)

QUOTE(shadowking @ Mar 12 2007, 00:03)

--optimize-mono is also buggy in lossy mode - bitrate drops , though only on dual-mono content.

Have you actually seen this lately? I thought I fixed it before I released 4.40 but I never mentioned it in the changelog because it (the bug) never made it into a real release.

Now that the DirectShow filter is based on the latest library, the only decoder that can't handle the new mode is the XMMS plugin that I provide. Once I figure out how to build that with the latest library I could make the --optimize-mono the default, with perhaps a new switch to turn it off (for maximum compatibility).

Unless you're using the XMMS plugin (or are encoding files for a large audience), it's perfectly safe to use all the time.

http://www.hydrogenaudio.org/forums/index....ost&id=2888

When encoding that sample with optimize-mono bitrate drops 60k or so. SNR is similar with or without the switch, but really should be much higher. If I add the missing 60k - i.e. change b320 to b380 I'll get what should be , I think..

WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.41.0-beta2
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.

ave noise = -59.18 dB, peak noise = -51.82 dB
created 001__08__Help_Me_Rhonda___Beach_Boys_cut___dual_mono.wv in 1.40 secs (lo
ssy, 301 kbps)

WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.41.0-beta2
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.

ave noise = -60.85 dB, peak noise = -53.20 dB
created 001__08__Help_Me_Rhonda___Beach_Boys_cut___dual_mono.wv in 1.40 secs (lo
ssy, 243 kbps)

bryant

Mar 13 2007, 00:02

QUOTE(shadowking @ Mar 12 2007, 15:51)

When encoding that sample with optimize-mono bitrate drops 60k or so. SNR is similar with or without the switch, but really should be much higher. If I add the missing 60k - i.e. change b320 to b380 I'll get what should be , I think..

That's actually the intended behavior. In the original release that supported --mono-optimize, the bitrate would drop to about half the specified bitrate and the noise would go up significantly compared to the stereo encoding, which was obviously wrong. In the newer versions the target was to keep the noise constant and let the bitrate drop (but not as much as before).

My thinking was that a worst-case scenario for this would be a track that was basically pure mono, but with an occasional sample different (in one channel) by one LSB. In this case, most blocks would be encoded in the new special mono mode, but the blocks with the single sample difference would have to be encoded in the regular stereo mode. In this case I think the ideal situation would be to have the noise remain about the same, which would mean that the bitrate would jump up in the stereo blocks because the "mono" mode is more efficient (the whole reason it exists). I think the bitrate jumping around would not be an issue, but the noise jumping around might be audible (and if it wasn't audible then why waste bits to make it even lower in the "mono" blocks?)

This way I can say that the --optimize-mono option can reduce the bitrate (in either lossless or lossy mode) with tracks that contain mono audio, but should never have any audible effect in lossy mode.

David

Josef Pohm

Mar 14 2007, 08:03

QUOTE(bryant @ Mar 11 2007, 23:45)

I have created a second beta... Thanks in advance for any testing...

Here is some evaluation of latest WavPack binaries conducted on my PIV Prescott 2.8ghz.
Ratio is calculated on my SetF.

CODE

|      | Ratio |    4.40.00   |   4.41.0b1   |   4.41.0b2   |

|  f   | 65,66 |  96,7  115,6 | 100,4  124,4 | 102,4  137,6 |
| fx1  | 65,24 |  59,9  116,1 |  64,2  123,2 |  65,0  137,6 |
| fx2  | 65,17 |  45,1  117,1 |  49,1  123,8 |  49,4  137,6 |
| fx3  | 65,13 |  27,8  117,1 |  31,1  125,6 |  31,2  138,3 |
| fx4  | 65,08 |  12,0  114,5 |  13,5  124,4 |  13,5  139,8 |
| fx5  | 65,07 |   9,7  115,0 |  10,9  123,8 |  10,9  139,0 |
| fx6  | 65,07 |   8,3  115,6 |   9,3  123,2 |   9,3  139,8 |

|      | 64,00 |  82,0   98,1 |  85,8  105,7 |  87,0  115,0 |
|  x1  | 63,56 |  44,4   95,9 |  49,5  102,0 |  50,1  110,6 |
|  x2  | 63,51 |  29,4   95,6 |  34,0  101,6 |  33,9  111,6 |
|  x3  | 63,52 |  15,9   95,6 |  19,0  102,0 |  18,7  112,6 |
|  x4  | 63,27 |   4,3   95,9 |   5,1  100,8 |   5,1  110,2 |
|  x5  | 63,25 |   3,0   95,6 |   3,5  100,0 |   3,5  109,7 |
|  x6  | 63,20 |   1,5   91,5 |   1,8  100,4 |   1,8  105,3 |

|  h   | 62,88 |  57,1   75,4 |  64,8   74,7 |  64,8   78,8 |
| hx1  | 62,78 |  30,0   76,5 |  34,7   78,3 |  34,4   83,1 |
| hx2  | 62,70 |  18,3   76,2 |  21,9   77,8 |  21,7   82,5 |
| hx3  | 62,66 |   9,2   76,2 |  11,4   77,8 |  11,3   82,3 |
| hx4  | 62,43 |   2,6   75,4 |   3,1   77,8 |   3,1   83,1 |
| hx5  | 62,39 |   1,9   76,2 |   2,3   78,3 |   2,3   82,8 |
| hx6  | 62,38 |   1,3   76,5 |   1,6   78,3 |   1,6   83,9 |

|  hh  | 62,47 |  49,7   60,9 |  52,5   62,7 |  53,0   65,8 |
| hhx1 | 62,28 |  22,2   61,2 |  26,6   61,8 |  26,7   64,8 |
| hhx2 | 62,22 |  12,9   61,8 |  15,8   61,9 |  15,8   65,0 |
| hhx3 | 62,17 |   6,3   60,9 |   7,9   61,8 |   7,9   65,0 |
| hhx4 | 62,01 |   1,7   62,2 |   2,0   61,5 |   2,0   65,3 |
| hhx5 | 61,96 |   1,0   62,1 |   1,2   61,8 |   1,2   65,5 |
| hhx6 | 61,96 |   0,7   62,4 |   0,9   61,8 |   0,9   66,0 |

A nice 16% encoding speed improvement and 12% decoding speed improvement, when going from 4.40 to 4.41b2.
Most encoding speed improvement is achieved with 4.41b1, most decoding speed improvement with 4.41b2.
Encoding speed improvement is slightly more focused on slower modes, while decoding speed improvement is the other way round.

shadowking

Mar 14 2007, 08:51

QUOTE(bryant @ Mar 13 2007, 16:02)

QUOTE(shadowking @ Mar 12 2007, 15:51)

When encoding that sample with optimize-mono bitrate drops 60k or so. SNR is similar with or without the switch, but really should be much higher. If I add the missing 60k - i.e. change b320 to b380 I'll get what should be , I think..

That's actually the intended behavior. In the original release that supported --mono-optimize, the bitrate would drop to about half the specified bitrate and the noise would go up significantly compared to the stereo encoding, which was obviously wrong. In the newer versions the target was to keep the noise constant and let the bitrate drop (but not as much as before).

This way I can say that the --optimize-mono option can reduce the bitrate (in either lossless or lossy mode) with tracks that contain mono audio, but should never have any audible effect in lossy mode.

David

Thanks for the explanation.

Martin H

Mar 14 2007, 09:45

1 image file ~ 11 tracks ~ 300MB | Intel Celeron 1.7GHz. :

CODE

D:\Temp>wavpack -h *.wav

 WAVPACK  Hybrid Lossless Audio Compressor  Win32 Version 4.40.0
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

created Evanescence - Fallen.wv in 129.54 secs (lossless, 32.76%)

D:\Temp>wavpack441b2 -h *.wav

 WAVPACK  Hybrid Lossless Audio Compressor  Win32 Version 4.41.0-beta2
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

created Evanescence - Fallen.wv in 118.27 secs (lossless, 32.76%)

D:\Temp>wvunpack *.wv

 WVUNPACK  Hybrid Lossless Audio Decompressor  Win32 Version 4.40.0
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

restored Evanescence - Fallen.wav in 91.87 secs (lossless, 32.76%)

D:\Temp>wvunpack441b2 *.wv

 WVUNPACK  Hybrid Lossless Audio Decompressor  Win32 Version 4.41.0-beta2
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

restored Evanescence - Fallen.wav in 80.67 secs (lossless, 32.76%)

Result :

WavPack v4.41b2 encoded the image file 11.27 seconds faster than v4.40.

WvUnpack v4.41b2 decoded the image file 11.20 seconds faster than v4.40.

Nice work, David - and many thanks for your continued efforts, my friend

DARcode

Mar 15 2007, 03:17

Nice improvement indeed!
Many thanks, David!

bryant

Mar 15 2007, 11:08

Thanks Josef, Martin, and DARcode for the tests!

These numbers are very much in line with what I am seeing here and I think a new release is certainly justified at this point.

Mangix

Mar 18 2007, 22:26

QUOTE(bryant @ Feb 27 2007, 22:11)

Right. Because WavPack encoding is integer-only (and generally not parallelizable) there's little or nothing to be gained by SSE and his brothers. MMX should be able to make a nice improvement because the left and right channels can be done together, but because of some optimization available for 16-bit data that doesn't work with MMX, the MMX stuff only really helps with 24-bit (and float) data.

what about for PCM Float? SSE should be able to make a nice improvement there, shouldn't it?

bryant

Mar 20 2007, 17:44

QUOTE(Mangix @ Mar 18 2007, 21:26)

QUOTE(bryant @ Feb 27 2007, 22:11)

Right. Because WavPack encoding is integer-only (and generally not parallelizable) there's little or nothing to be gained by SSE and his brothers. MMX should be able to make a nice improvement because the left and right channels can be done together, but because of some optimization available for 16-bit data that doesn't work with MMX, the MMX stuff only really helps with 24-bit (and float) data.

what about for PCM Float? SSE should be able to make a nice improvement there, shouldn't it?

Well, no. WavPack actually uses no floating-point operations even when operating on IEEE float data. I did this to ensure lossless operation even with slight variations in rounding behavior and to allow integer-only platforms (like Rockbox) to play IEEE float files even without an FPU.

he-jo

Mar 21 2007, 10:34

Hi David,

there has been an idea in my mind for a while, and now I'd like to ask you, if you think this would be possible

As I read it from your docs, each block in the encoded data stream is independent from the previous one. So, if two blocks have the same parameters (which should be almost always the case in your current implementation - except for –optimize-mono), we could decorrelate them at once with SSE2 or AltiVec, couldn't we?

Regards,
Jo.

bryant

Mar 22 2007, 06:15

QUOTE(he-jo @ Mar 21 2007, 09:34)

there has been an idea in my mind for a while, and now I'd like to ask you, if you think this would be possible

As I read it from your docs, each block in the encoded data stream is independent from the previous one. So, if two blocks have the same parameters (which should be almost always the case in your current implementation - except for –optimize-mono), we could decorrelate them at once with SSE2 or AltiVec, couldn't we?

I'm not completely up to speed (so to speak) on SSE2 or AltiVec, but what you suggest sounds reasonable. Unless the user specifies a -x mode, all blocks will have exactly the same sequence of terms and deltas.

I have also thought that the existing MMX code could also be used for 24-bit mono data by doing multiple blocks in parallel.

Of course, these changes would involve a lot more internal shuffling than the current stuff did.

BTW, please feel free to e-mail me directly on your ideas. I really do appreciate your input (and work so far).

David

edit: spelling

he-jo

Mar 23 2007, 10:23

QUOTE(bryant)

Of course, these changes would involve a lot more internal shuffling than the current stuff did.

Yes

That's why I hoped to get some help from you

I need to think a bit more about it. You know, absence of spare time is the biggest problem...

QUOTE(bryant)

BTW, please feel free to e-mail me directly on your ideas. I really do appreciate your input (and work so far).

Thanks for the invitation! I think the developer mailing list would probably be a good place for discussion.

DARcode

Mar 30 2007, 01:23

Is it release time yet please

?

bryant

Mar 30 2007, 15:06

QUOTE(DARcode @ Mar 30 2007, 00:23)

Is it release time yet please

?

Well, there's going to be a beta3 to fix the issues raised in this thread, and I'm also looking at quickly throwing in the --skip and --until commands from the FLAC decoder. So It'll be at least a few weeks before an official release...

David

DARcode

Apr 2 2007, 12:54

QUOTE(bryant @ Mar 30 2007, 23:06)

QUOTE(DARcode @ Mar 30 2007, 00:23)

Is it release time yet please

?

Well, there's going to be a beta3 to fix the issues raised in this thread, and I'm also looking at quickly throwing in the --skip and --until commands from the FLAC decoder. So It'll be at least a few weeks before an official release...

David

Fantastic! Thanks a lot!

Synthetic Soul

Jun 1 2007, 02:25

QUOTE(bryant @ Feb 27 2007, 07:16)

I just wish I could get a few more percent on the decoding speed to break 100x. The way it looks now it's easy for people to get WavPack mixed up with the riff-raff!

I've finally got around to uploading my test results for 4.41 final.

You may be pleased to see -f(x) achieving 104x when decoding.

It's just a shame (for WavPack marketing) that my CPU doesn't benefit as well as many.

Bruno Monteiro

Jun 1 2007, 04:03

QUOTE(Synthetic Soul @ Jun 1 2007, 08:25)

QUOTE(bryant @ Feb 27 2007, 07:16)

I just wish I could get a few more percent on the decoding speed to break 100x. The way it looks now it's easy for people to get WavPack mixed up with the riff-raff!

I've finally got around to uploading my test results for 4.41 final.

You may be pleased to see -f(x) achieving 104x when decoding.

It's just a shame (for WavPack marketing) that my CPU doesn't benefit as well as many.

Just checked, nice tests. I'd like to see this test run under a dual-core CPU, to see how the codecs would scale. I don't know how many (if any) support it. We would see a huge speed improvement. AFAIK, WavPack doesn't support it (yet)...

Maybe I'll run that tests in some time, maybe broading the test corpus to include other types of music!

Regards!

GeSomeone

Jun 1 2007, 07:39

QUOTE(Bruno Monteiro @ Jun 1 2007, 11:03)

... run under a dual-core CPU, to see how the codecs would scale. I don't know how many (if any) support it. We would see a huge speed improvement.

You don't understand. On multi core machines you should run multi instances of the wavpack executable. Your hard disk will become the next bottleneck quite soon.