First of all, I'd like to thank Bryant for this wonderful codec. I switched to Wavpack for my whole collection (2000+ CDs) since the 4.0 release and I never looked back. The compression/speed ratio is a killer, and the ApeV2 tags don't hurt either.

However, while running some general tests on lessless codecs, I stumbled on some results that may be of interest. I happen to listen to a lot of old mono popular music on CD (so technically it's stereo, but you see what I mean) and I was surprised to see Flac -8 frequently outperforming Wavpack -h, which to my small knowledge never happens with stereo music. Something weird, too: sometimes, Wv -f outperforms by a very slight margin wv default. Here are some numbers:

* On 5 out of 10 CDs tested, Flac -8 outperforms Wv -h (actually, Flac -5 does too)

* The biggest gap is happening with a recent Presley remaster where Wv -h does 32,52% while Flac -8 does 28,09%.

* On 2 out of these 10 CDs, Wv -f outperforms by a slight margin Wv default, and on 1 of these CDs even Wv -h! (-f = 42,01%, -n = 42,76%, -h= 43,30%).

I uploaded an Excel sheet on my site with detailed results. If you have any questions, please let me know.

This seems interesting. I'll be interested to hear from more in-the-know people!

A-Ha. So you guys are close to discovering one of the changes in WavPack 4.3

I won't give out details now as it involves some compatibility goop. I'll let David explain better, with his own words.

I was actually shown the "mono" problem several months ago by Guruboolez and recently figured out what was happening. The problem happens on tracks that are perfectly mono but in stereo format. I would have caught this myself, but many "mono" CDs actually have slight differences between the channels (like if you ran a mono signal into a stereo ADC; not all samples would be identical). It ends up costing about 5% compression (compared to the mono equivalent), and also happens on stereo tracks that have sections where the left or right channel is totally silent (this would not be too common).

One solution is to convert such tracks into mono using Audition or some other program before encoding. These would play fine and I would guess that most software (like CD burners) could figure out how to deal with this. Obviously not ideal, but a possibility.

There is no way to fix correctly this without breaking the decoder. I actually did fix it in preparation for the 4.3 release, but then decided that breaking the decoder would be a bad idea. I suppose that I could add an option that would check for mono during encode, and then reencode the track if it was mono. The problem would be a possible compatibility issue with the resulting files and, of course, the tracks might actually have a handfull of mismatched samples that would throw the whole thing off.

Unfortunately, no perfect solution...

I personally think that at this stage when WavPack isnt that big yet that breaking the old decoder to fix better compression on mono material and implement smart noise-shaping on lossy wouldnt be that much of an issue. At least for me.

'Egad'!

So the only solution would be to check the whole track for absolute correlation between the two channels before encoding? We definitely can't have any incorrect samples.

But how come FLAC outperforms WavPack on these pseudo-mono files? Is it something to do with the joint-stereo algorithms involved?


Possibly... but these may be the kind of features for the version five release, maybe?


Edit: Addition

I may be (and probably am) a bit dumb, but.... I'm not sure what do you mean by "breaking the old decoder". Does that means that files coded with the hypothetical new encoder will not be decoded at all or that they will be incorrectly interpreted by the old decoder?

The second possibility is of course the worst one. Is there a provision in the format for a "version number check", that is a check for a particular signature in the file at decoding time so that the decoder can say "I don't know this sub-format, it is likeley to be a new one so check and eventually update me"?

If this is not the case one possible solution would be to change the file extension for the new format, maybe something like .wvp (that afaik it is not used yet) so that the old decoder will not be associated with it.

I agree that this will definitely call for a major Rev. stamp update.

Sergio

Thanks for your answers, Bryant. Sure, breaking the compatibility on such a mature and widely spread codec like Wavpack is not a minor issue and I perfectly understand your choice.

As I intend on sticking to Wavpack, and since maybe 25% of my collection is in mono (I'm an old geezer ), I hope nonetheless we'll see one day this optimization. In the meantime, take care and keep up the awesome work.

I love WavPack. If you have found a way to fix the mono compression error, please use it, even if it breaks older versions compatibility.

I disagree. I think if wavpack has an ambition to be mainstream format then first thing in mind is compatibility. MP3s are still so damn popular because they have best quality/smallest size?
Joe Average would only get pissed when a wv file he's got just won't play in his fav player. As a consequence he will try avoid "buggy/corrupt" wv files and'll stick to flac/ape for lossless.
In my opinion, sacrificing the 100% compatibility for wavpack to make a few percent size reduction in certain (rare) situations is a bad move.

I also disagree. It may well be a consideration for WavPack 5 but I don't think it'd be good to break compatability any sooner.

What I don't understand is why WavPack performs so bad on these almost-mono files. Is this, as I asked earlier, some kind of joint-stereo problem? Bryant, an explanation for a dumb person would be appreciated.

Anyway, downmixing first is not an idea solution for files that are 'almost' mono as, technically, you are changing audio data, right?

I don't wish to take sides here, because I think it's Bryant's choice, but I wouldn't call these situations "rare". Sure, I ran my tests with only a 10 CD sample, which is not much, but half of them were concerned by this situation. If you consider how much mono music is still bought and heard today, be it blues, classical, jazz, or even pop/rock, it's definitely not a "rare" situation.

Actually you've got it backwards. On the "almost" mono files WavPack does fine. The special case that WavPack does poorly on is the "perfect" mono files. These are the ones that I should have detected during encode as being mono and stored them as a single stream. Instead, I just store them as regular stereo where one stream (the "side" stream of joint stereo) is all zeros. The entopy encoder of WavPack can detect when streams have long runs of zero, but only when both streams do. If only one stream is all zeroes, then it consumes 1.5 bits / sample, which translates to almost 5% compression (1.5 / 32).

So, these are the files that could be losslessly converted to mono wavs before encoding (assuming they were mono all the way through). I suppose I could add an option to the encoder to check each sample during encode, and then make a second pass to create a true mono WavPack file if there were no left / right differences in the entire file. It would be interesting to know how many applications (like players and burners) actually accept mono source files and also what percentage of "mono" tracks are really this pure.

I notice that on BoraBora's speadsheet (very nice, BTW!) only 3 of the 10 mono albums show this 4-5% difference. Perhaps the others have a mix of pure mono and almost mono tracks, or the tracks themselves alternate between pure mono and almost mono. I guess I should try out some of my mono CDs (being a bit of a geezer myself ).

Accomodating this on a block-by-block basis would require the addition of a flag to the format that would cause all existing decoders to display "not compatible with this version of WavPack!" when they encountered one of these files. This would be fine for most HA members because they probably keep up with all the recent releases. But it might be a while before all the other programs that play WavPack have been updated, and this would be inconvenient for some and, like jetpower says, could annoy Joe Average.

An even bigger concern of mine is that if any other product developers (especially of hardware devices) are considering (or are actively working on) WavPack support, the news that the format is changing would not be welcome. Both Monkey's Audio and pre-4.0 WavPack were famous for breaking the decoder every release, and this was probably not so bad because all the tools and plugins were supplied by us. FLAC was different in that Josh finalized the specs at the first release, and I believe that even the most recent FLAC file will play fine on the very first decoder. One could argue that this is an important reason that FLAC enjoys such wide hardware support, and it was my idea that WavPack 4.0 would follow this model (even though I haven't got the chance to make a formal spec ).

Of course, this is a very simple change on the decoder side and does provide a significant improvement to the files it applies to. What I may do is incorporate the flag into the decoder now but not use the mode (this is already the case for some other potential features like VBR lossy). After several months, if the decoder is in wide use and most apps have been updated and no hardware devices have surfaced, then I could add the encoding feature to take advantage of it.

Thanks for the explanation - I finally understand! I doubt there will actually be many pure mono files, will there?

Hopefully you'll find a suitable solution. I understand that you don't want to break decoder compatibility now and I'm not too worried - I'm too modern for mono!

This is mostly true. Pre-1.1 versions of FLAC will choke on embedded cue sheets, due to a decoder bug. In addition, there seems to be some issue with 24-bit files; for example, the 24-bit file discussed in this thread, encoded by 1.1.1, decodes incorrectly with 1.0.2.

Perhaps you should consider adding the feature to both the encoder and decoder immediately, but have it disabled on the encoder side unless a command line option is given to enable it. This would be akin to the --lax option to FLAC--if the user selects this option then he is taking responsibility for the fact that his file might not decode everywhere. At some point in the future you could make it the default.

You could even generalize this to a format-version option that would specify version of the file format the encoder would generate. Something like -v4.0 (or -v4.1 or -v4.2 since the format hasn't changed) would specify the current format and -v4.3 the updated version. This would allow you to evolve the format, keeping the default at some reasonable level for compatibility purposes, but still allow the advanced user to take advantage of new developments. You could even have the user specify an earlier format-version than the default if he uses an old decoder somewhere or just for maximum compatibility. The obvious downside would be a fair bit of added complexity in the encoder to selectively enable all these format improvements.

Glad to hear it.

Is there an easy and possibly automated way to bit compare left and right channels of a wav file with a free tool? If there is, I could sample 50, 100 or even 200 of my CDs so you could have a more exact idea of the "pure mono"/"not so pure mono" ratio. I don't know nothing about programming and not much about audio, but if I can help you assessing your choices, I'll gladly help.

I don't understand why there is a push to emulate FLAC in respect to this issue.

I have recently encoded over 500 mono tracks with Wavpack Hybrid - and I can assure you - that for an average 5-7% difference in compression on true mono files, it is hardly something to get stressed over.

You know what they say: If it ain't broke - don't fix it!

WavPack is ready to fly in popularity, and creating compatibility issues over this issue could be a serious blow to its acceptance and credibility.

Cheers