Very disturbing indeed.

I'll tell you what's disturbing:

1. 1.15u is not the latest version of mppenc. 1.15v is. It's pretty obvious that using 1.15v would not have made any difference in the test, but it does hint at the level of involvement the community has in Musepack's developemt and the lack of attention to the official site.

2. Classical music is murdered by radio stations. This is not new. Conducting a test for such genre at --radio is pointless, and performance at --standard is much more interesting for the majority of listeners who aren't half deaf.

3. 44.1 KHz samples (all the ones in the pack) are not perfectly resampled to 48 KHz by cards like the Audigy 2. This leads to different aliasing on the original and encoded sample. Resampling to 48 KHz before a test is a must. Example of a sample that is horribly affected by many Creative cards (BitTorrent link only)

When the amount of interest and support in the current development equates to almost zero and the members of HA who continually complain about the lack of improvements to mppenc do not participate in such a small test, the community cannot expect the development team to do it all on their own.

The test was carried out at --standard, not --radio.

Aren't the >=1.15r Musepack encoders tunned more for the insane profile than standard?

True. The preface to the test hinted that --radio fails badly on such a genre. That's just irrelevant, since
both Andree and Frank never intended to optimize the encoder for such a preset. I don't see why this fact has
to be mentioned so many times. There are probably very few (1%-2%) users who have .mpc files created with
this preset.

It's unfair to compare Musepack to other, specially-tuned codecs, at these bitrates, despite the
fact that one of Roberto's tests proved otherwise. For the record, even --standard wouldn't be the choice of
classical music lovers. They'd go for lossless or at least --braindead. --standard being close to transparency
is a pipe-dream that Frank Klemm had. I never supported those claims and I find that those with golden ears
shouldn't even bother with such tests.

Typos such as "Concierto violon" in the graph just indicate to me that
the test was a hurried one. Proving that --standard cannot handle classical music or electronic music by
Autechre is stating the obvious. It's *not* a revelation. How can --standard be improved? By cooperating
with the very few who still desperately try to work on mppenc, not by stating the same obvious facts over and
over throughout many similar tests that basically reach the same conclusion.

There has been NO perceived
improvement in --standard over the last 3 years for classical music and several other styles, except for
improvements due to bugs fixed in the last year.

Edited bad paragraph positioning

Since the sample you posted is so low in size I've uploaded it for direct download here.

You're the only one who downloaded it, which maybe means that everyone accepts guru's test methodology as valid without questioning the way they are performed. Is that what I should expect from a community that prides itself on its objective and correct listening tests? no.

Since I don't want to be dragged into any fight, I'll sum up my points:

1. Inbetween all the --radio talk in the first post, I missed the fact that the test was actually performed at --standard. My bad. That is not to say that I find the test more interesting because of it. There are probably hundreds of classical samples where both 1.01j and 1.15v fail at --standard. You just have to listen and find them.

2. Much like Frank did not get any significant feedback on his 1.95z67 binary, the current developers (who are much less talented or experienced in this) do not get help at all. It's impossible to assume that an encoder can mature and improve when only 2 Musepack team members perform ABX tests and others fail to submit their impressions regarding the samples the devs wish to improve the handling of.

3. I'm sorry if guruboolez finds my posts a personal attack on him. I only wish to criticize the way the tests have been performed and mostly the insistence of everyone in poking at Musepack's performance at --radio and --standard. Yes, Frank does not hear as well as Andree, and as a result, reports of failures at --standard could not always be handled perfectly. Yes, significant *perceived* progress (quality-wise) has been largely made from the first days until the last version from Andree. Frank contributed *many* improvements to the codec, some of which can be heard at higher quality levels, but --standard is still *far* from ideal. Face it. If you can hear no problems with it, be glad. If you do, you are just one out of many.

You probably don't know it, but I don't have internet at home. Therefore, I couldn't connect to www.musepack.net to check if 1.15u was the latest version or not. Sorry if it appears as disrespectful. Using my free time to test mpc is a mark of respect I suppose, and there's no need I must say to suspect "a lack of attention".


--radio is the name of the profile, and doesn't reflect the habits of radio (which are probably not using or even know anything about mpc.


Supposing that the internal resampling of Audigy could trigger audible artifacts (I could compare with my DMX6Fire which doesn't resample, and I never heard any of them), I am experienced enough to distinguish potential aliasing from lowpassing, pre-echo, chirping and ringing. The problem is internal to musepack, and the possible impact of soundcard is totally marginal, and probably inexistant.
Last but not least, I've compared mpc vs mpc, and not mpc vs PCM. Your argument have in these condition no sense at all.


Same thing for Vorbis developers, LAME, Faac, etc... You're not alone in this sad situation. Most people can't ABX musepack, and some other don't necessary have free time or simply interest to do it. But when I sent results to Quantum Knot, Gabriel, Bryant or Ivan Dimkovic, I always obtained a polite answer, and the promise to check what happened. You're the first developer I saw that consider the methodology as wrong, and who clearly claim them to be uninteresting. I also very surprised to see that you're not interesting at all by the quality of --standard profile, by far the most popular one.
Saying that --standard is not suitable for classical is not the answer. Suitable or not, excellent or piece of shit, the problem is that 1.15u showed regression compared to an older version. As developer, I supposed that you would be interested to check the problem, and to try to improve your encoder. I was wrong. Sorry.

I suppose that I will spend my free time to help other developers (LAME, WavPack lossy or Vorbis) which are more interested by individual listening tests. I probably can't help you to tweak --braindead profile. My skill is limited to profile which don't apparently interest you (at least with classical). Have a nice week-end

I insist, but it just to say that I find these words really offending. You've apparently commented the whole test without reading anything I've described (you even missed the fact that I've tested --standard). As I said it, I spent more than four hours to conduce this test on 15 ridiculously small samples, in order to respecting all rules of blind test (fixed number of trials, rests when needed, discarding first second of VBR encoding). Then, I spent many time to write a boring text to give all precision in a foreign language.
Suspecting the validity of a listening test and denying all efforts I made for a simple typo (I can't see it ) is absurd, and offending.



Most people on this board that would claim that musepack --standard distord classical music would risk bannishment. No listening test on this board revealed that (except maybe mine). For most people, your claims could either appear as a revelation or be interpreted as a troll. The "official" HA.org recommendation is very clear:

http://www.hydrogenaudio.org/forums/index.php?showtopic=1927

No mention about flaws with classical music. I can't find anything mentionning this apparently well-known issue on www.musepack.net.



How do you expect any form of cooperation with your attitude?



Plus 11 samples which were *perceived* as regression compared to an older version.

Thank you guruboolez for your very informative listening test. I normally don't use musepack, but enjoyed reading the test nevertheless because it is - as usual - clear, precise and methodologically sound. I think you make by far the most useful and systematic contributions here on HA to support codec development. Seed should really be ashamed for his grossly disrespectful reaction to your tests, for which you sacrificed your free time.

First, I gotta say I'm dumbfounded. I've never, ever seen a developer reacting so badly to useful feedback. I saw developers getting pissed at some of my test results - for instance, when Vorbis clearly won over AAC one year ago. But they never tried discrediting my tests after a cursory read. (Actually the only knowledgeable person - not counting some slashdot morons - to ever try to discredit my tests was Julius Thyssen, but that's just like him)



<...>


How interesting. First you claim comparing at --standard would be "much more interesting", later you say that the results with this same preset are "obvious. It's *not* a revelation."

Care to decide at once what is your opinion about testing with --standard?



Resampling would make the test less useful, for that's not what most people do when they play their tunes at home. And tests are about emulating the average usage.

Also, you musepack guys whine so much about resampling, and still there is not a single word about that necessity in your site (at least, none that I could find at the pages or at the forum). If that issue only surfaces when musepack pushers want to whine about a listening test, it can't be that serious then.



As Guruboolez himself pointed out, you're not alone there. It's not like people here are ignoring Musepack in favour of FAAC, NERO AAC, Vorbis and Lame.

The bottom point: whining about it won't help you the slightest - quite the opposite, as you probably already noticed.



If my tests proved otherwise (your own words), how unfair can it be??!



Hoho. I guess my tests were done in a couple of minutes then, because there are typos everywhere in the results page. I have a particularly nasty tendence of writing "it's" even when I should be writing "its".



Ohhh, you're approaching messianism there. Honestly, if I were you, I wouldn't get desperate over an audio codec. It'll make you stressed and you'll die at an young age of heart attack.

Also, what many similar tests are you talking about? I simply can't recall any test comparing Musepack at --standard preset versus an older version with classical samples only. Care to post a link?


Edit: split the post in two otherwise the quotes would break

I think Guruboolez has a little more experience on conducing and participating in listening tests than you do

I can say, from the top of my relatively vast experience conducing tests and producing results, that his methodology is excellent (I would say flawless, but that's pretty much impossible - as far as I know, he doesn't have an acoustic room for instance). It has been that way since tests he conduced years ago, and it doesn't seem to me he deliberately changed his methods in this test to make Musepack sound bad or some such bullshit.



Of course, poking at Musepack's performance has always been a taboo here. Now the most experienced tester in this whole forum is taking the plunge, and the Musepack advocates have no idea how to proceed. The first course of action that comes to mind is, of course, discrediting.



This is insanity. The official speech about Musepack quality changed overnight. Before, --standard was the answer to all of one's questions. Now, many can hear problems with it. (freely rearanging your own words)



That really made my brow raise. Are we to expect the "Musepack nanny" really believes Andree and Klemm listened a lot to radio and tried to emulate it perfectly in that preset, and not just named it trying to imply Radio's lack of quality?



Correct.


Thank-you very much for yet another informative test, Guruboolez.



Edit: hahaha, my post number 6666. How appropriate

OMG, I can't believe it's happening, but I completely agree with Roberto.

Anyway, I just wanted to state my support for guru. Rest assured that there are people that very much appreciate your hard work.

I dont want to choose sides here, but there is clearly a lot of frustration. So i guess it would be best to discuss the technical aspects and not blame the testers/developers.
There has been a lot of effort in this test AND in the development of mpc. Lets be gratefull for BOTH.

Agreed, people are quick to judge, even more when some of them obviously know only 15% of the facts but still care to comment.

Complaining about MPC uninterest while it's going to get his first hardware support thanks to Rockbox...

And i don't see the point of argueing with Guru who has posted here very helpfull and very reliable feedback...

:/

Letting this degenerate into just another flamewar will be to the benefit of noone.

Thanks for the work and efforts of both guruboolez and the Musepack Development Team!

Thanks for all friendly words. I wondered what I did wrong this time by testing an encoder and pointing some regression.

Nevertheless, Seed's answers are very interesting. Seed being an official musepack developer, I wonder if we shouldn't simply consider as official his following positions:

- standard is not a transparent encoder, and will never be so (excepted maybe for people who are at least "half-deaf").
- braindead should be the recommended encoder for classical music, offering noticeable audible improvements compared to --standard, --extreme and also --insane.


I have also serious doubts about the extreme difficulty of encoding classical music (apparently, a simple flute should be as critical for perceptual encoders as Autechre sharp electronic music). There are 7 or 8 collective listening tests that were performed by ff123 and Roberto, and I guess that people could confirm that classical samples never appeared as a very-hard-to-encode music, comparable to killer samples like Death2 or Fatboy.

To sum up, this thread is very informative. Without any posted ABX test (from Seed - mine are apparently useless), we know yet that:

- braindead profile isn't braindead at all, but is the minimal setting to obtain transparent sound
- audible improvements are possible between --standard and --braindead, as long as you prefer violin to electric guitar
- Frank Klemm had limited abilities to tweak musepack (he will certainly appreciate it)
- Classical music must be considered as a killer-genre for perceptual encoders, even if most people experienced something else
- Radio profile is so poor that people should stay far from it (it will be probably easier to remove it from mppenc code, and I wonder why developers maintain such awful profiles)
- that people considering --radio as a good encoding solution at 128 kbps are "half-deaf" - which implies that all people having difficulties to ABX AAC, Vorbis or MP3 at this bitrate (all are close or inferior to MPC --radio) are also "half-deaf". How many people on this board have a full-audition? Nobody I suppose (nobody on HA.org claimed -and respect TOS#8 at the same time- that braindead was necessary to reach transparency with a specific musical genre)
- Audigy2 high quality resampling leads to audible artifact when listening to MPC (the only listening test I've seen on Creative soundcard is this one from Pio2001, and it appears that internal resampling of Creative soundcards -even older- have no impact on quality).


That's cool, isn't it?

Just a question: does TOS also apply to developer? I'm browsing HA.org since the beginning, and I can't remember any developer of any perceptual encoder making such claims that are so totally different from the experience accumulated by thousand members without bringing any element of proof.
I respect developer works, but today, I have to confess my perplexity.

Thanx guruboolez. Really strange and disappointing sense. Now I'm trying to analyze the problem of musepack... Maybe, it's indeed in levels of classical sound... I've tested in recent days mppenc1.15v on very silent samples and heard great distortions. I think, result you have got is due to wide dynamic range of classical music... Maybe three year old version is just compressing low level sounds more carefully...
Seed, you brought some misunderstanding in my mind with your posts. I'd like to see really serious arguments from your side... In other hand, after some thinking I almost understand your position... But you making obvious things that were not so.
(forgive me for my unpracticed english )

To Seed's defense, i think he overreacted a little bit, but it's no use to get cynical about all the points, the positions are hardening and nobody will get anywhere. Generalizing at either side is pointless, as this seems to be a rather specific issue, which should of course be taken seriously. I'll try to make sure that - when things have cooled down a bit - this will be looked into. Until then, classic nuts should consider at least --quality 6, but just like Seed, i guess that's a redundant advice for that clientele.

I could confirm the fact that mpc has some problems with low volume music: obvious on many samples with --radio (Debussy.wav as example), and it also could be noticed in unusual listening situation with --standard.
But during this test, I've evaluated many samples which don't appear to have a low overall volume. The ringing problems was nevertheless audible. Apparently, there's another problem with 1.15 encoders and classical music at --standard profile, and it was introduced somewhere between 1.01j and 1.15.

What is most obvious here is that Seed took feedback intended to be constructive, and took it as a personal attack. This sort of thing certainly is understandible (your code is like your child, you don't want people criticizing it), but it isn't desireable at all, and I hope Seed softens his position.

We all appreciate your work, Seed, Guru was just trying to help.

So encoding trance at -extreme will still surfice a good rip?

Apologies, I don't get this ABX stuff. I think Guru, despite pointing out errors in a good and informative manner, could have been a little more creative on what to put right rather than what to put wrong.

Seed...well when I first read his post, I had no idea he was an official dev...he dived into the thing and took it as an argument - admittedly it must be a bitch to have your work exposed...but he could have been a little more positive.

Sorry, it seems i also came over as generalizing. Experience with previous situations has shown that problems with a certain group of samples can be decreased by forcing a higher bitrate. However, this might as well be a problem that can't be cured like that, and rather needs tweaks to the codec (for instance Ogg Vorbis has such problems where a higher bitrate doesn't solve anything). However, using a higher bitrate preset can never backfire (except for filesize) and often helps. So take it like i said, you should consider for now. I'm sure we'll find out more, and then a specific answer can be given. In the meantime, maybe people can try to reproduce the test results. After all, the difference is not that alarming as some people make it seem.

I beg you pardon, but what do you mean by creative? I don't understand? I made a listening test, and it's not really a "creative" task, but rather a form of research.



Just "a little more"? A little less arrogant, and a little more respectful on HA.org Term of Service (backing up his claim, respecting the necessity of double-blind listening test and not considering this methodology as a flawed one) might also help I would say.



Are you sure to understant Seed? He personaly complaint about the lack of reactivity of MPC basis, the lack of feedback; now he have one he simply does't try to read it, but simply discredit the entire work with arguments coming from an unknow source (certainly not HA.org experience, recommendations and history).

Guru, take serious consideration to being one hell of a lot less argumentative and confrontational - I was only trying to summarize in such a way that both parties can meet an argumentative compromise

Seed does need to take the critisism though.

What codec should I use? AAC? Ogg?

OK, fine. But could you explain your summary. You said that my attitude wasn't creative enough. I repeat: I don't understand. I made a test, which was intended to be the first step to further investigation. What did I miss according to you?

P.S. It's not a cynical reply: I really don't understand what you tried to say previously.

You are probably missing some flowers and a little bit more of blue color. This would have been more creative...

I see. For the next test, what is better: writing the report in alexandrine or in haiku style?

Guru I didn't mean to imply that Seed was justified, just that he responded agressively when he interpreted your listening test as an attack on his "baby". At least that's my impression. He's totally wrong in doing so of course, and I hope your (highly enlightening) listening tests continue, since I read each and every one of them avidly.

I say many thanks to Guruboolez' continued efforts, great publishing work !
And subscribe the old answers of Roberto and Guru to the aggressive posts of Seed.
But I don't wanna heat the discussion up again, it is really time to cool down (yeah, maybe it is the weather, very hot days recently in germany and Europe). Of course it is difficult to to view his own work from a distance, but necessary in technical questions, as especially the work on the best lossy encoder of the world will never be finished... (regard the different profiles, bitrates, from "really transparency" to 99.999% population of critical ears at higher bitrates down to "transparency for 95% of the population", down to best compromises for low bitrates for portable devices, where bitrate counts.
So, we see here, that the work on an encoder is hard work, and needs to be apprecited a lot, so many thanks to all previous developers and the current dev.-team, especially Seed.

Blind alexandrine?