Recently I posted on this thread:

http://www.hydrogenaudio.org/forums/index....showtopic=36432

I quickly realised I'd like to begin a discussion / debate on ABX.
Now, before I get reported for trolling, I would like to say that I use ABX. I'm not against using it.

The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?

Someone once pointed out that choosing in ABX involves concious thought. Listening however also involves the subconsiousness.

Let the mayhem begin

As far as I know, with an ABX test, you can only prove that there is a difference, not that there is no difference.

If that was possible, I could simply prove that an encoder is transparent by not listening carefully during the test (failing the ABX test on purpose).

So the hypothesis is: "There is a difference between A and B" and it is up to the tester to prove it. Two samples are the same unless proven otherwise

It can be defined as "transparent" for that particular person. Thats why people shouldn't always follow recommendations blindly, they should make the effort to find their particular format settings for what they descover is transparent to them, and for this you need ABX testing.

No, because its impossible to "prove" that something does *not* exist. It is only possible to prove that something *does* exist.

So, strictly spoken, you can never be 100% sure that something *is* transparent. But you also can never be sure that in 3mins weird aliens will not kidnapp you ;-) It all comes down to probabilities - if a significant amount of skilled testers were unable to prove that there is a difference, then it is probable that the majority of people also cannot detect a difference - this is what usually is called "transparent".

Thus, there is no such thing as absolute certainity. You trust/make decisions based on probabilities - not on "real" certainities. Thats not something specific to audio - you handle things like this countless times per day. For example, if you would only cross the street if it is 100% safe, then you could never cross it - because there would always be the possibility that you erred. So, based on probabilities you "hypothetically asume" that something is certain - the problem is just that people tend to forget that this "certainity" is only hypothetical.

Personal opinion following:
Concerning objective vs. subjective methods, it depends on the case. There is a reason why humans have heart as well as mind - because both have their individual strengths and weaknesses. The mind tends to be more "error-safe" but is also more restricted. Heart and intuition can be useful in unknown situations but are more prone to error. If heart, mind or a combo is most effective depends on the case.... with the topics discussed on ha.org, a scientific approach(mind) is more efficient imho, and thats why i do agree with TOS#8. Subjective means *in this case* would do more harm than good.

"Listening however also involves the subconsiousness." - ching3

How are you conscious of this subconsiousness?

I think 'transparent', when referring to lossy codecs, is a great simplification and a confusing term to use. IMHO a more correct term (variations of which is repeated throughout this thread/ha.org) would be 'effectively transparent for this person and sample'.

Popper said:

Thus in audio encoding the only thing which can be defined as transparent is lossless.

Here are a couple of interesting research papers in the area of perception.

The first is an overview of research regarding conscious vs. unconscious perception, where most of the work has been done with the visual system:

http://watarts.uwaterloo.ca/~pmerikle/pape...tudies.1998.pdf

This one is focused on the auditory system, specifically how we might perceive and react to rhythm (consciously and unconsciously):

http://www.apa.org/journals/features/xhp273600.pdf

Wow! Thanx for the references!

It's generally a better idea in a task of analysis to rely on as many reliable sources as you can. But ABX is among the best sources because it uses double-blind testing to generate empirical results. And yes, if the listener cannot differentiate between the compressed test sample and the uncompressed reference in enough test cycles to be meaningful, the compressed encoding is shown to be perceptually transparent to that listener.


Because you can hear when you're not conscious. This isn't clear proof, only an indication in that direction.


What I generally say is "perceptually transparent for the listener", and just saying "transparent" in this context is generally understood to mean the same thing.


That would only be the case if the context were either infinite or so vast as to not be reasonably measured. Any scientific analysis must have a defined scope to be meaningful. One person with a measureable hearing ability using ABX to test a few seconds of music for perceptual transparency can certainly identify what is or is not transparent to their ears within the scope of (1) the sample being tested, (2) the encoding format and setting used, (3) the individual's health and hearing ability, (4) the rigorousness of the test (i.e., the listener follows the rules of the test and applies all of their abilities) and (5) the number of test cycles executed.

... and (6) the playback environment and equipment being used.

I guess the bottom line here is that while ABX is an excellent tool, but not perfect. Also, if the masses hear things one way and I hear something else, then we do have a clear winner: C'est moi !

True that you need a quiet and controlled environment, but I was thinking this is provided in the "rules" of using an ABX comparator. I could certainly be wrong on that, though.

But the equipment used really doesn't matter as much as people may think. In my experience the vast majority of artifacts that can be heard on a $10,000 system can just as well be heard on an off-brand sound card and $8 earbuds.

I see, well I'm guessing now for ABX to be trully perfect, one would have to run through an infinite number of trials!

But of course it seems ABX is very reliable, despite minor uncertainties.

Isn't there some kind of method in scientific research that goes along the lines of 'prove this is incorrect and thus the opposite must be correct'...? Something to do with an anti hypothesis, I learnt this in high school, but it's all very fuzzy Perhaps there is a way to be (nearly) certain that an artifact or difference is not noticible, ever, by the listener...

Edit: Punctuation

The outcome ABXing is a probability, not a "yes" or "no". There is no perfectness in that.


In a way this is already what ABX is about. You are trying to statisticaly prove that you CAN hear a difference. If you fail in statistically proving this, the opposite must be true: you CANNOT hear a difference.


ABX can do that for one listener, but it wouldn't be fair to extrapolate this answer to everybody else. Even when a group of listeners has the same ABX result, you cannot be absolutely sure that this will count for everybody else.

A certain uncertainty will always remain in this, you will just have to decide for yourself with how much incertainty you want to live.

As for this comment:

If your subconsious would play a role in normal life, why not during an ABX test?

No - this is generally not possible, because it is impossible to prove that something doesn't exist. Absolute certainity simply is not achievable. All you can do in your daily life as well as science is to rate probabilities - define a degree of unprobability at which "it is quite safe to asume that it doesn't exist".

In the end, we are all just gamblers who try to get the most safe deal. But some uncertainity is part of the game ;)


Exactly.

I think he just worded a bit wrong what he was trying to say. What he probably meant is that a "quick ABX-test" leaves out mid/longterm-feedback.

Nothing stops one from doing relaxed longterm ABX-tests which could als include subconscious "feedback" (some would call that intuition or just "feeling"). But in practice, such tests are rare because the effort and time needed is multiple times higher than with a "quick" ABX.

So, its not a limitation of the ABX-methodology. Its a limitation of how much effort people are willing to put into such a test.

- Lyx

Is there any sofware to abx for videocodecs?
I know it's audiorelated so I created a new topic here. I don't want to crosspost but guys who are interesting about audio compression don't open this subforum frequently.

http://www.hydrogenaudio.org/forums/index....showtopic=40537

Excuse me guys, I'm a newbie but there is something I can't understand.

- On one hand you can measure what you can hear. I mean, there are "machines" that can measure the frequencies that an specific human can hear. That´s objective.

- On the other hand you have a song encoded at x kbps that at the end of the day (correct me if I'm wrong) will deliver different kind of frequencies. This element is also objective.

Honestly, I know nothing about ABX test but by definition, some frequencies of the song will fall within the range you can hear, and some won't.

What I'm trying to say, is that despite the fact that I can say that I cannot hear a difference in an ABX test, objectively I must hear it if my ears are able to process the frequencies. Am I so wrong?

Yes you are wrong. Your ears/brain are only processing some information based on a complicated set of rules and their own physical limitations. Furthermore, while you may be able to hear all frequencies within a band, you may not hear certain frequencies under certain conditions (for instance, very quiet sounds may not be noticed in a noisy room).

If you can hear the difference, you should eventually be able to ABX the difference given enough time and motivation.

All you can do is say that "in this test, the listener(s) did not make any distinction". No more, no less.

no. if n (listeners) is high enough and tests are held under reproducible and standardized circumstances minimizing other variables that might interfere with the results then you can say with a certain amount of confidence (e.g. 95%) that results are valid to the whole population.

Since when has Hydrogenaudio become the birthplace of "magical thinking"? It's kind of like attempting to argue the case of Free Will Vs. Determinism. I personally just like to settle for Compatiblism and call it a day.

I agree.
I disagree. Because of the structure of ABX testing, the negative case (no difference can be heard) is can be trivially affected by expectation bias. For the positive case, however, it reduces the effect of expectation bias to an insignificant level. For example, If I am comparing 32kbps MP3 versus 44.1/16 wav and I truly expect there not to be a difference, I can bias the result of the test in such a way as a negative result is produced.

Surely, based on this, you can't prove a negative hypothesis with the ABX method. If you think I am wrong, please share your reasons.

You can always sample population, but you can never make more than a probabilistic statement in either direction, please.

You can use controls to test the sensitivity of any given test. You can make a probabilistic statement based on any given test. There is no "proof" in any case, either way, only stronger and stronger (as results and 'n' go up) probabilistic proof.

I personally take "proof" to mean what one "proves" in a mathematical proof, and there is no absolute certainty in subjective testing, only probabilistic certainty.

Does that count as properly performing an ABX test - deliberately saying the files are indistiguishable when you can actually hear a difference (selecting B when you know the correct answer is A)?

To me that is more akin to making a concerted attempt to bias/sabotage the test. Any experiment will become useless if testers makes a decision to ignore, or say the opposite, to what their senses are telling them.

If a sufficient group of testers perform the same test, it is unlikely that all of them will share the same bias, and thus it is likely that more accurate results will out-weigh what would be later considered an anomaly.

If a hundred people are asked what colour the sky is on a clear day, and I say red, knowing I actually see blue, yet the other 99 people say that it is blue, then we can be a lot more confident the sky is blue, rather than red. The 'bad' data gets filtered out in the end.

It might not be a conscious effort. Expectation bias can play a huge role in non-blind test results without the tester conciously being biased. That is why we choose a double blind testing methodology. Surely if the tester expects that there is no difference, he can be biased towards a negative hypothesis, even subconciously?

In my mind the ABC/HR methodology is stronger than ABX against this bias. ABX doesn't prevent this from happening at all. It doesn't mean that positive ABX results are any less valid - it's still extremely good at removing expectation bias in the positive direction. ABX result statements like "there is a 99.99% chance the tester can distinguish between these samples" are still valid.

In my mind, ABX results like "the tester could not distinguish between these samples" are not truly the results of a double-blind test. I might be wrong about this, but it seems logical to me.

Do you know the term "controls"?

Quite aside from that, are you saying that the subject knows which signal 'x' is?

Indeed, but how is this relevent?
No, the subject doesn't know what X is. However, the subject expects that the difference between A and B is not audible and is therefore biased towards selecting X at random. The point I am trying to make (perhaps not clearly) is that expectation on the part of the subject can bias them (consciously or subconciously) into not putting their "best effort" into identifying X. Surely the ABX methodology is based on the hypothesis that the test subject made their best effort?

Expectation depends completely on what the test subject knows about the material before the test starts. I don't see how ABC/HR is different from ABX in this regard, as the effort required to come to a positive result is not different and the effort required to select at random will be equal between the two too.

Then how do the controls not detect this?