no audiophile will encode in 128kbps anyway...so that was a real world test with real people that use that bitrate...nothing wrong with it and well done

You keep saying "invalid" and "inaccurate." But in what way? As in the rankings would have produced a different order of rankings, or another winner or loser? I don't think so. The effect of having different setups, in my opinion, is to add random variability to the results, so that the uncertainty is greater. But I don't think it would add a bias, i.e., change the order of the rankings by much, if any.

ff123

I just meant that most of these people will not bother testing, not that they will do the test WITHOUT knowing what an ABX test is.

First you say there are not enough participants, then you want to conduct a test with a very selective group...

The test does not claim to be more than it is, it is not the definite anwer to which codec is "best". However, it does give an good indiciation what is good and bad (or perhaps i should say "not so good"?) for the average user. There is no more inaccuracy in the test than the error bars in the graphs suggest.

A tests which proves codec A does better than codec B on some 10.000$ piece of equiptment is completely useless for most people.

It would have meaning for people with poor equipment as well; the quality headroom will be bigger with the winning codec. Even if the current equipment isn't good enough to reveal flaws at 128kbps, I bet most people would always want to encode with the best format at that bit-rate, rated by people where the equipment/hearing isn't the bottleneck.

When hifi mags test speakers, they tend to use the best possible cables, amplifiers and most trained ears. That doesn't make the test useless to people with average hearing/equipment

This is exactly my point. Crappy equipment, and poor ears don't provide and accurate data. Yes, it may be real world to the majority of listeners. But, nevertheless, it does not prove anything substantial. For all we know these users were guessing. I would trust good ears and good equipment with a small majority of users, over poor ears and crappy equipment with a large majority of listeners.

Do you even know how ABC/HR testing works?

Nice one...

Please read up on how the test was performed. You cannot make any valid conclusion from it if you do not understand how to interpret the data.

@rjamorim: As I see, you couldn´t resist the oportunity.

@jmitch: Well, that´s what a listening test´s supposed to be, isn't it? A "so good" audio equipment is not that relevant.


SoNiX

The accuracy of the answers is given in the test results. It is 95 %.

Explanations : http://www.hydrogenaudio.org/forums/index....howtopic=16295&
The ABC/HR method : http://ff123.net/abchr/abchr.html
The Anova analysis (which gives the 95 % above) : http://www.psychstat.smsu.edu/introbook/sbk27.htm

Just to satisfy yourself, why don't you take the test with your gear? All the samples and ABX software are still there. when you are done, just for yuks, go slumming and borrow someones "crappy" sub $100 phones and see if that makes a difference (not in how good things sound, just in how they affect your ability to abx a codec)

Let us know how it went.

edit: fixed quote markers

I participated in this test myself (when it was run). I used:

- RME Digi 96/8 PAD (professional level 96kh/24bit sound card with 1:1 bit accuracy and very nice analog measurements)
- High quality minimum capacitance shielded interconnects
- Meier Audio Pre head (very high quality solid state headphone amp)
- Sennheiser HD600, AKG K271s, Ultrasone HFI-650 and Etymotics ER4p/s headphones (Ultrasones and etymotics getting most of the listening time)
- A quiet room with a silenced computer

My ears have been tested to be flat to 11 kHz (with less than average attenuation after that up to 14kHz, the maximum that the test equipment at national hearing clinic was able to test).

I have c. 4 years in trying to get into lossy audio, perhaps 3 of that with slowly increasing listening acuity. I've gone through several training sessions with the AES "Perceptual Audio encoders - what to listen for" CD, as well as many example samples from here, mpeg development archives and previous listening tests. I've also purchased and gone through the Moulton labs "Golden Ears" hearing training cd set. In addition, I regularly audition new hifi and high end gear (and also write about hifi to a national publication). I think my hearing (both as an instrument and as a skill) is better than average.

While I'm far from being a "golden ear" I can invalidate the above argument by saying that neither my equipment or hearing is crap.

My results didn't significantly differ from that of the statistical averages in this test.

While neither my hearing or equipment are not "best in class", I think they are clearly better than average population in both cases. It is debatable how good they are, but surely not crap.

As such, I don't think the test can be in invalidated by only referring to "crappy equipment and poor listeners".

Had I magnificiently surpassed every other listener in this test by picking out artifacts other couldn't hear, I could _perhaps_ be willing to entertain the possibility of the argument being right.

But alas, I wasn't even among the best listeners in the test. Surely equipment at least wasn't a limiting factor in my case.

I must say that I was also a wee surprised that spotting artifacts in a 128kbps ABR test was so difficult. I knew it was going to be difficult, but it was even more so than I initially had imagined.

friendly regards,
halcyon

PS I really should not even have needed to reply with this defense, as ad hominem type attacks don't really need refutation. I think arguments should be evaluted based on the evidence available (and the logic of reasoning). Not on the basis who makes the argument, UNLESS there is strong proof to show that the author is not to be trusted (which in this case is non-existent). Conjecture is not enough. Arguments need evidence, not prejudice as their support.

Tell me you're kidding. Hifi mags may *claim* that the ancillary equipment they use is the 'best possible ' (though it's almost never been subjected to a controlled listening test) and that their listeners are 'trained' (no proof of that either...but I guess it's how guys like Robert Harley can hear the directionality of the crystalline structure of cables)... but that's no reason to believe what they claim.

And it's always funny when the caution that the listener who isn;'t using a $10,000 amp and $100/ft cabling might not hear the amazing microdynamics they hear...thereby covering their asses.

FWIW, Floyd Toole, Sean Olive, et al, who are doing controlled comparisons of speakers using trained listeners at their facility at Harman/JBL, seem rather more credible to me than any 'hi fi' mag in this area