1.01j encoder vs 1.15u, listening test inside Options

[guruboolez]

Yesterday, I had some free time to play with various encoders. Like many people on this board (I guess) I’m still waiting on a major AAC release (Apple’s VBR encoder; major release of Nero AAC or at least the ‘green light’ for using the ‘fast’ and high quality encoder), and I have therefore decided to wait before starting a new personal multiformat comparison. This disappointing situation gives me an occasion to play with other encoders, especially musepack audio.

First, I’ve encoded many classical samples with latest encoder, with –quality 4 (--radio) profile. After two encouraging results on collective tests, musepack’s quality and reputation are now well-established at this bitrate (~135 kbps). I’ve experienced it myself, and I clearly admit that MPC at mid-bitrate performs really well… with some exceptions.
In reality, the same collective tests revealed serious issue with musepack, especially with classical music: bloated bitrate with some samples first (up to 180-210 kbps on harpsichord), and the opposite with some other samples, especially low-volume moments. With the sample named Debussy.wav (opera part: two people speaking/singing with a small orchestral accompaniment wreathed in recording noise), musepack performance was much worse than other competitors during Roberto’s 2nd 128 multiformat collective test (worse than WMA and atrac3!). This unusual low performance was probably correlated to a strange bitrate plunge (~100 kbps), not bad by itself (as long as it not hurts, lower bitrate are always welcome), but really questionable when quality is so strangely bad. Of course, Debussy.wav is only one sample, but it’s not a hard task for someone enjoying classical music to find numerous moments of low-volume including recording noise…

My own experience of musepack at mid-bitrate is not a big one, and a good half of it could be found on the first (and partially flawed) listening test I’ve published twenty months ago (here). In October 2003, musepack 1.14 performed equally with vorbis, slightly better than lame 3.90.3, but worse than WMA9Pro (winner) and QuickTime AAC. In other words, the relative performance of MPC –radio was not as enjoying than quality obtained with less specific music genre. Last but not least, with all progress made during this time by AAC (both Nero & Apple implementations – and faac too), lame, vorbis and maybe WMA(pro and only pro), musepack could seriously pretend to be the potential looser among all other modern audio format.

Yesterday, by listening quickly to various classical samples (I’m building a full library of very short samples, yet unfinished), I was simply amazed by poor performance of musepack at this bitrate on my music (I insist: the encoder performs differently and much better with everything else than the general category of ‘classical’). I’m now used to live with ABR encodings coming from LAME 3.97a, and I could swear with a great confidence that LAME is free of most issues audible with musepack: ringing, lifeless sound, and a typical artifact of mpc which reminds me for unknown reasons a washing machine ). I have suspected one moment the PNS tool (enabled by default up to –quality 4.99) to have a bad impact with some of these samples, but a quick try (---pns 0 disables it) showed this algorithm to be innocent. Then, I have decided to try older versions of the encoder. I’m used to amass a plethora of encoders and keep them on my hard disk, but recently I’ve burned them all, and had the good idea to loose the disc (lost somewhere in the shambles I’m using as an apartment). I’ve only kept mppenc 1.01j (please, don’t ask me why!). And result obtained with this encoder is completely different: less ringing, lower artifacts… quality is much better with mppenc 1.01j --radio than with mppenc 1.15u --radio with classical music.

Few words about musepack –radio history (and correct me if I’m wrong). Frank Klemm worked on musepack at mid/low bitrate few time after he released mppenc 1.00 (it must be 1.02, 1.04 or 1.06), by introducing a tool called PNS (different beast IIRC from MPEG-4 AAC Perceptual Noise Substitution). As a consequence, the use of PNS lowered the bitrate of some MPC profiles (radio, thumb, telephone) without jeopardizing the output quality. It means that 1.01j --radio will systematically be weightier than 1.15u --radio (~15 kbps) and it implies that both settings are not directly comparable. The comparison I made is logically unfair. But it nevertheless reveals unexpected drawbacks of former musepack tunings, and (I hope so) will maybe open the door of further improvements

As example, Debussy’s encodings (replaygained with high value) could be downloaded here:
• 1.01j
• 1.15u
• Debussy (lossless)


Ok, and then? Now that you’ve read my awful literature, you’re probably disappointed to not see any table of results, ABX score or ANOVA analysis. No? It’s simply because I haven’t made a complete comparison between 1.01j and 1.15u at --radio profile… in order to make one at the famous --standard setting. It might sound totally absurd to compare a three-year old version of mppenc (released on May 2002) to the most recent version of the encoder (February 2005). But if tunings made three years ago hurts on several classical music samples at --radio setting, it is natural to speculate about possible regression with other profiles. But I don’t really like unwarranted speculation (probably as dangerous -if not more- than placebo), and I’d rather make a listening test in order to obtain results, whatever they are, than being suspicious or throwing doubt on current release of musepack. My biggest risk is to hear nothing (good or wrong) between both encoders. But of course, if I have created a long topic and smashed the record of grammatical mistake per square centimeter, it’s not to tell to everybody that I can’t hear something wrong between two encoders considered for several years as transparent to most people

This post has been edited by guruboolez: Jun 18 2005, 06:56

[guruboolez]

• samples: 15 classical samples, used on several previous listening tests (multiformat, AAC, Vorbis and LAME), and coming from a selection of best technical & artistic recordings of the years made by a French magazine.

• hardware:
— Creative Audigy2 soundcard with official driver
— Beyerdynamic DT-531 soundcard
— Onkyo amp

• ABX conditions:
— schnofler amazing ABC/HR software (0.5 alpha 5)
— fixed number of trials: eight; sixteen if problems (results are hidden until the end of the test)
— no blind ABX sessions between reference file and encoded one ; ABX comparison between 1.01j and 1.15u encodings only ; REFERENCE is nevertheless accessible for playback during ABX session in schnofler’s software (so, please, don’t complaint on the theme of “you’ve rated the most pleasant to your ears and not the one which was closest to reference”)
— first second was systematically discarded from evaluation and ABX comparison (if not, it was a mistake; see log files to check it: the listening range is stored)
— consequent rests between most tested samples (4h30 to test the complete set of 15 samples)
— a lot of coffee, water, soup and even ice cream

• Encoders and Decoder:
— mppenc.exe 1.01j from (file date) 2002.04.02 (YYYY.MM.DD)
— mppenc.exe 1.15u from 2005.02.23
— mppdec.exe 1.93h from 2002.10.22 (default settings of ABC/HR 0.5a5)



RESULTS





ABX logs are here.


Surprisingly, there’s an audible regression with latest encoder when compared to an older one. Statistically, it’s apparently safe to say that – to my ears and on these samples which are not really killers – mppenc 1.01j is better than 1.15u. Bitrate can’t be invoked to explain this difference: 1.01j encodings are systematically smaller than 1.15u (12 kbps, corresponding to a +6.4% inflation for latest encoder). So what happened? From what kind of problems suffers 1.15u?


First, it should be important to keep in mind that I haven’t rated any sample 5.0 ; none was transparent, and all of them revealed at best minor issues with both encoders. 1.15u doesn’t introduce new artifact, but rather increase the level of audible distortions. But what form of distortion? It’s hard for me to give a precise description of what I’ve heard. But one issue appeared several times.

• rinaldo.wav: “now, it's distorted, fluctuant, a bit acid too”
• hercules.wav: “aggressive ringing on some parts”
• bach BWV 1014.wav: “flute is not clean, and suffers from asperities (irregular or unconstant line), like a very subtle ringing”
• compostelle.wav: “It suffers from something like chirping. Reference sound is very tonal, continuous, regular and smooth; encoded one is very messy, irregular, inconstant, like millions bees flying in the church. Weird, and aggressive”
• Bruhns.wav: “Irregular organ note (now…); aggressive ringing. ”


Ringing, fluctuant, irregular, inconstant… On five samples, I’ve described the problem as being a form of ringing
Other problem:

• Brahms: “ slightly lifeless (missing noise). Pre-echo is perceptible”
• Van Wilder: “ pre-echo appears clearly ; sound is also a bit synthetic and lifeless, probably a consequence of missing noise or excessive lowpassing process. But pre-echo is clearly the biggest issue on this encoding.”

Here, something like an excessive lowpass (probably) or ATH threshold lead to a synthetic sound (very subtle) and also pre-echo issue. I’ve encountered similar problems when I’ve tested some problematic versions of lame: aggressive noise reduction was often correlated to higher level of pre-echo (and of course it involves the feeling of synthetic, lifeless, synthetic, claustrophobic ambiance).


For other samples, I couldn’t be more precise (acidity, false or wrong color, and of course distortions are filling the log files).


On the other side, 1.01j appeared to sound worse on two samples:
• Mozart – Dies Irae: sibilant consonant (“s”) were distorted (clear difference)
• Avison: the solo violin appeared as more “acid” (subtle difference)



LINKS

Samples are as usual uploaded for complementary tests (highly welcome).
• not available anymore for download

mppenc.exe (v.1.01j) could be downloaded here

This post has been edited by guruboolez: Dec 29 2005, 22:28

[Vertigo]

Guruboolez, you could have saved us the trouble of all that bothersome reading and just say, "Everything you know about Musepack is a LIE!"

....thanks for dashing my dreams, I enjoyed living in ignorance.

This post has been edited by Vertigo: Jun 18 2005, 08:32

[NumLOCK]

Just a remark - as Guruboolez said: replaygained with high value, if it was just a very quiet sample which got amplified a lot, does this really represent usual listening conditions ?

[Garf]

Just a remark - as Guruboolez said: replaygained with high value, if it was just a very quiet sample which got amplified a lot, does this really represent usual listening conditions ?

Common for classical music, which was what the test was about.

[madoka@ex-sounds]

BTW, where is 1.01j binary?

[guruboolez]

BTW, where is 1.01j binary? 

I've uploaded it but forgot to mention the link:
http://guruboolez.free.fr/MPC/mppenc-windows-1.01j.zip

[guruboolez]

Just a remark - as Guruboolez said: replaygained with high value, if it was just a very quiet sample which got amplified a lot, does this really represent usual listening conditions ?

Debussy.wav was tested during latest 128 multiformat test by many people, and artifact were clearly perceptible, even without replaygain correction. Replaygain is just an help; it amplifies the sound and the artifact (in case of ringing), but even without it it could be apparently heard by many people.
BTW, ReplayGain on track mode isn't something unusual, and there's nothing wrong to evaluate the quality of encodings with a corrected gain

This post has been edited by guruboolez: Jun 18 2005, 14:00

[madoka@ex-sounds]

BTW, where is 1.01j binary? 

I've uploaded it but forgot to mention the link:
http://guruboolez.free.fr/MPC/mppenc-windows-1.01j.zip

thx guruboolez

[Digisurfer]

Keep up the good work guruboolez. I always find your tests interesting, as well as enlightening.

[JeanLuc]

I'm so glad that all my Musepack encodes have been carried out at -braindead ...

[CiTay]

Certain replies split to this thread.

[Gabriel]

Does the current MPC encoder adjust its ATH level according to content?

[Gecko]

Very interestig test, Guruboolez, thank you! I guess the next interesting question is when the regression was introduced.

Perhaps to find that point, testing of only one or two samples would suffice. At some point in development, bitrates dropped dramatically (and where raised again afterwards). That could be a good candidate. I think it was version 1.13 that introduced the bitrate drop, but I could be wrong. I don't have time to test now; I am in serious need of sleep.

Guruboolez, I still have many older mppenc versions on my hdd. If you want to investigate this regression, I could email them to you. Ca. 5mb: 0.90e to 1.15u.

[guruboolez]

I think it was version 1.13 that introduced the bitrate drop, but I could be wrong.

I can't remember any 1.13, but mppenc 1.04 was notorious and had frighten many users when Frank Klemm tried to improve MPC efficiency. Interesting messages by the way in this old topic, especially this one

EDIT: I like this one too...

I used the "--radio" profile for all tests, to get usable results (--standard is already too good sometimes).

I really like the "usable" concept. Maybe CiTay could tell few words about it to current MPC developer?

This post has been edited by guruboolez: Jun 20 2005, 01:15

[Lefungus]

Does someone have the sources of mppenc from 1.01j to 1.14 ?
Because it'll be much more difficult if not impossible to see where the regression was without those

[shadowking]

Wouldn't it be easier to start with 1.14 and test quality in relation to 1.15 and 1.01 ?

[guruboolez]

Lefungus> mppenc was closed source when 1.01j was released. Source became public very recently, one year after the release of 1.14 and 1.15r. You should maybe directly ask to Frank Klemm to obtain older source.

This post has been edited by guruboolez: Jun 20 2005, 10:11

[Peter]

Case mentioned having at least 1.14 sources some time ago. However we've lost contact with him recently.

edit: Thanks to Case, I've got this: http://www.foobar2000.org/mirror/sv7-1.14.zip