I'd like to start by saying that while I recently registered, I've been both an avid fan of the ABX concept and long-time reader of HA.

The test itself is simple and straightforward enough. Listen to said samples, then play a random one and decide which it was you heard.

That said, I have some issues, not with the ABX testing methodology itself, but with the conclusions that are frequently withdrawn from them.

All too often, I feel they are given more weight than they truly deserve (other times, not enough weight, though that is rare here thanks to TOS-8).

An example:
---------------------
User 1 rips his favorite album via EAC using his Dell Laptop. He used "Secure" mode and concludes the rip went fine(it very well may have). He encodes everything using the latest recommended version of lame, whatever that happens to be at the time. He originally used 192kbps CBR with forced full-stereo since he read somewhere that this would result in higher quality. Later on upon learning more he switched to VBR preset-standard / V2, with no other command line options besides --verbose because he likes seeing all the info.

User 1 was previously unaware of ABX tests. He was using winamp to play everything. Upon visiting HA, he quickly switched to foobar realizing it's apparent superiority, and now has an interesting ABX tool. He decides to conduct some ABX tests, to see what he truly "Can and Cannot" hear. He hurries away, re-rips some of his favorite albums to WAV or FLAC then proceeds to convert them to lots of different bitrates with the intention to compare them. So far so good. Right now he has a few dozen versions of the same song in a "Testing" folder, all encoded at various bitrates, with various codecs. His test stops short when he realizes he cannot ABX the 64kbps AoTuV vorbis encode from the original. He concludes that his hearing simply was not as good as he thought it to be, that his bitrate-needs were greatly inflated. While a bit upset at what appears to be like his lack of accurate auditory perception, he is happy that he can now store more music with less bits & bytes.
---------------------

People often tout his failure as a triumph with jestfull but serious statements like, "Don't worry, be glad, now you can take more tunes with you!", as if to reference that he should try to remain satisfied with the results of that test and use it as a benchmark for what he can and cannot hear.

There are plenty(almost infinite) variables that can and likely did inhibit his ability to hear the differences between the tracks.

To me, the only logical conclusion one could withdraw from the test that was conducted is something along the lines of:

"Using my present on-board AC-97 sound chip, with this specific version of it's drivers, with my current $20 radio shack headphones(or worse, laptop speakers), with the background ambient noise of my laptop, and the nearby refrigirator(was sitting in the kitchen), with this specific song, at this specific bitrate, at this specific point in time with my current physical and mental state, I was unable to discern the differences between the two files."

That conclusion I just withdrew from the test is radically different however than what most people conclude from failed ABX tests, I find.

While I fabricated most of this scenario, it is based on a post I remember reading not too long ago(I'm sure some others will recall it).

In summary, while the test itself is sound(pun intended), the conclusions people tend to withdraw from them(most notably, failed tests) are often not what I would consider accurate.

Its widely known that ABX cannot disprove a difference.

You surely have a point there ... on the other hand, you should take into account that the major public listening tests (and these are the ones being really representative) are being carried out under certain fixed conditions:

1) the amount of test samples always covers a wide range of musical styles with different demands towards the participating codecs' qualities.

2) every test participant is well aware of the fact that subpar sound controllers, in combination with notebook speakers or el-cheapo headphones won't be able to reproduce differences in a way decent headphones or decent stereo sets would.

Given these two boundaries, public listening tests are quite accurate and reliable IMO.

On the other hand, personal listening tests (and you are right on that point) are often being overrated and sometimes lack accuracy. But I think we can safely assume that people being aware of the ABX method know when (and why) a personal ABX test might show a lack of accuracy due to test conditions.

The quality of your listening system is, more or less, unrelated to how well you can ABX. LAME --aps work, allegedly, was primarily conducted on a laptop with onboard sound and inexpensive ($50 or less) headphones.

In fact, one can make the argument that better equipment makes ABX tests harder to perform, in some circumstances. Because the frequency responses and distortions are so much tighter on better gear, the psychoacoustic assumptions the encoder makes actually become more valid.

I think that everybody engages in a little bit of reading between the lines when they see an ABX test. You need to understand the context. If you have 10 ABX tests in a row saying that (for instance) the audibility of ultrasoind has not been observed, the actual numbers say that audibility may still exist - but you'd be hard pressed to find anybody who would actually defend that argument. The lack of a positive results restricts how important any audibility is compared to your own listening skills. If for whatever those listening skills aren't that good, then there are very few restrictions on interpreting an ABX failure, and it would make sense to encode well past what you think is transparent, if your skills are expected to improve.

Yeah, its a good point - Although in my personal listening tests, Lame -V3 was adequate on all but 1 sample (and even that one was just a problem sample; hardly indicitive of something i'd actually notice in normal listening), I find myself using -V2. The reasoning?

More people find -V2 transparent, and when/if I train myself to be a better listener, I will also be unhappy with -V3. The fact is, there IS an audible difference, and I just haven't spotted it yet. When I've got my music cranked at a party, I want to KNOW that everyone is hearing CD-quality sound.

There are limits to this rationale (I'm not going to use lossless on a portable just because Guru can ABX practically anything), but generally speaking I want my music to be un-ABXable by ANYONE, not just me.

I can almost guarantee that the people at your party aren't going to be paying as much attention to sound quality as you did during your ABX test.

Not only that, but at any given party, the sound quality is going to be much poorer than a quiet room with good headphones at loud volume.

That is, at any decent party.

. I think that once the tests have been performed a general consensus can be reached amongst the community, then valid conclusions can be drawn from that.

Coming to an audiofool store near you: The PortaBX! You can easily ABX a song you hear and an automatically-downloaded lossless version on the go! Impress your friends with your ears using PortaBX!

I am also of the opinion that things should continue to improve measurably, unless there is a specific reason not to do so.

Example: No one questions the superiority of recording at, say, 96/24 or 88.2/24 for the purpose of mixing and/or mastering, before a final mixdown to 44.1/16. The work is already done at high resolution. Ideally it should be released in whatever format it was recorded in, avoiding any transfer to a lower sample rate or word length. Simply put, taking a 96/24 recording, and converting it to any other samplerate/bit-depth cannot IMPROVE quality, therefor it should be left alone. However, that is not accounting for all the pieces of the puzzle. A very real and valid piece is that if this were to be done (and it is already done in the form of DVD-Audio), Content producers have and surely will abuse the new format as a vehicle to introduce more ways to curtail consumers rights on how they can and cannot use that media. I see this as a very real and very legitimate reason to AVOID, for example, DVD-Audio & SACD's. Having said that, considering I can buy a 750GB hard drive for a few hundred dollars, with terrabyte drives on the very short horizon, I see no reason storage-wise why I cannot store higher resolution audio. Assuming that each redbook CD is roughly 400MBish as FLAC/WavPack, one could easily store roughly 500 CD's on a 200GB hard drive, which can be had extremely cheaply. Since encoding them to a lossy format cannot increase the quality, and space is of no concern in many cases(aside from portables), there is no reason not to use lossless in most cases. Not because the lossy versions cannot be transparent at some point, but simply because they cannot make it sound better. Best case scenario is you will not notice them being worse.

I suppose the core of what I was trying to convey is that, in my opinion, failed ABX tests should not stop measurable quality from improving, so long as there exists no specific reason to avoid it (increased restrictions on our already dwindling content-rights is a very valid reason to avoid it).

As for the environmental parameters I think it's clear that a personal listening test that is to find out about what encoding parameters to use must use an environment of at least the quality that is available when real listening later. I think everybody who goes into the effort of ABXing is aware of that.

There are practical questions like 'is my AC97 sound sufficient for testing while I will listen later using a high quality DAP?' Maybe this is the most important question cause other environmental parameters (like which headphones to use) usually can be done with production parameters. As for the PC sound system if in doubt one has to find a friend with a decent soundcard (or buy one).

I think more crucial is the problem of which test samples to use. Any test is limited and in the end the tester can only decide on these samples. My personal solution to this is to use special test samples which I know that as a set are a problem to many different encoders (sure everybody knows I use harp40_1, herding_calls, trumpet, and now: eig).

Of course this opens the question about how to think about problem samples. Gambit once said he doesn't want to train towards hearing artefacts. He is absolutely right - I made this experience with my biggest listening test so far where after a day of intensive listening I was able to abx harp40_1 at cbr320 mp3 which I couldn't in the 200kbps range just one day before.

This shows that ABXing is necessary to get at non-subjective conclusions, but we also have to make up our mind about how to think about the results, and about how to perform the ABX process.

For getting productive encoding parameters I demand to get transparency with regular music in a short ABX test (that doesn't bring me too far away from the concentration used with regular listening), and I demand for very good quality with my standard problem samples as a safety margin.

I believe that ABX can be useful but it has it's shortcoming. It's really a personal choice and a matter of equipment used. This is not a clear cut debate like cables. If you have the space and want use lossless.

The main interest of ABX is that it provides statistically significant results and is, of all other means, the most objective ground that can be provided to back up one's claim. It is a second best solution and has several limitations but it is still the most valid.

SirChristof,

I think your hypothetical situation bares more truth than you realize. However, it is localized truth.

If the user finds that on his laptop w/ crappy sound chip and crappy built-in speakers or headphones that he can only hear a difference up to xxxx kbps with yyyy codec, then he has succeeded in finding the sweet spot for that particular hardware. If that will be his primary listening platform, then the test worked and served him well.

Since he is using a sub-par sound setup, he probably is not an audiophile, and he will not be concerned too much with future-proofing or cross-platform portability.

It can. What it can't do is disproving any difference.

But say that one people hears a clear difference between two sources. He runs the ABX test, hears the difference, and fails the test. Then the test proves that the difference that he heard during the test did not come from the source.



You can't assume that better equipment is more neutral in general. Take for example two high end headphones : the Sennheiser HD600 and the Beyer DT990. According to headphones.com, the second one output treble 15 dB louder than the first one ! I've listened to the Beyer once, and immediately noticed some clicks in the CD mastering that I had never noticed with my HD600. I bet that ABXing lossy codecs should be much easier with the Beyer than with the Sehhneiser. By the way, it seems to me that Guruboolez uses a Beyer.
I have already succeeded (in the last multiformat test run by Roberto) in ABXing two samples with my Dynaudio Gemini speakers, that has a peak in treble response, while I was unable to do so with the HD600.

About ABX in audio in general, I wrote something in french : http://www.homecinema-fr.com/forum/viewtop...apter#170073190
It would be interesting to translate it into english, but I have not got the time to do so these days.

Nope. A failed ABX test proves nothing about the audibility of differences. I can fail to ABX any samples you can give me, no matter how big the difference.

I can not pass any ABX test, though, and this is the important point. A passed, properly done ABX test cannot be the result of placebo or malice. A failed one can.

About laptop speakers, at least mine can reveal some artifacts in high and low frequencies that are succesfully masked on higher quality speakers. My guess is that since better speakers have a better response, they can reproduce maskee components quite well, while poor speakers might not reach the desired power at that frequency and "unmasking" the noise.

Sighted listening has shortcomings that ABX does not. ABX has no shortcomings that aren't also present in sighted listening...unless you consider the effort involved in doing the ABX, a shortcoming.

Btw, doesn't this thread belong in either 'Listening Tests' or 'Scientific/R&D Discussion'?

If you are listening to A and claim that you are recognizing B with no mistake, then the test proves that you are mistaken.

Wouldn't it be better to say: the test results failed to prove an audible difference ?
I agree with Pio that when someone claims a "day and night" difference, and the ABX results don't confirm this, the difference (if any) was probably not as big as presumed.

Both are true. The listener made a mistake, and the test failed to prove an audible difference.

The negative hypothesis is proven as soon as we can tell that the difference under test is completely and clearly heard during the test, with no possible doubt for any answer given by the listener. If the test is a failure, then that difference is disproven.
The more surely the heard difference during the test can be said to be the same as the difference heard everyday, the more generalizable is the negative result.

The conclusion that one can draw from some results depends on the way the test is setup. If you always remain in the 0.05 % type I error, the statistically representative listeners etc, you will always get the same kind of answers.
But when you want to study a given phenomenon, you may setup any suitable experiment.

For example, in standard studies, we need as much as possible average listeners, and they should not know what they are testing. But in standard studies, we look for the average abilities of the human ear.
If you are wondering if a given codec can produce any audible distortion, you should on the other hand choose trained listeners, and they should first learn to recognize the artifacts under test.

Different question, different setup.

Your statement and mine are logically identical, since 'a' and 'any' have the same meaning in this context.

The topic title is wrong. The OP is suggesting that the conclusions drawn from an ABX test can be wrong, not the ABX test itself.


Of course this is simple human error.

If I made a thread title:

"Stopwatches, Where they fail and why. One viewpoint."

and then started explaining that timing measurements from people who do not know how to handle stopwatches are not to be trusted, I'm obviously kicking in open doors. I'm not saying there stopwatches are failing at all, am I?

There is nothing wrong with ABX. Unfortunately, there is with humans.


Not necessarily.

A difference could be a specific difference spotted at time X.
Any difference is every possible difference imaginable.

The chance of failing a proper ABX goes against 100% if you want to fail, no matter if you hear a difference or not. Thus a failed ABX test cannot prove anything at all, unless you first prove that you wanted to succeed. Good luck with that.

Or I could just encode my CDs to OGG Q4 (which I've ABXed many times and find it transparent) and fit, instead of 500 CDs, 5000 CDs. Your logic is a little strange, and right now I'm wondering what kind of car you have in your driveway. You might only need to drive to work and to the store, but who knows when you might have to haul 1000 lbs of dirt, tow a car, drive 20 people around, run a road race, or do all kinds of other ridiculous things. But I've never had to do those things, which is why I drive a regular car. If I actually needed to do those things with my car, I'd rent/buy another one. Same with music. I will never ever need any sort of quality higher than ogg Q4, but on the chance I do, I can always rerip it. No sense in keeping all your music in (useless) lossless and wasting space.

You use the argument that more storage space is very cheap; I use the argument that smaller filesizes are free (and transparent).


I think it should be just the opposite. Current quality should stay the same with filesizes dwindling. I don't need anything sounding better than ogg q4, so if they found a way for it to take up less space, I'd be very happy. There is no reason that consumers need DVDA and SACD, as far as quality reasons. The only reason I support those formats is for extra information like 5.1 mixes, pictures, videos, etc. But that's a whole other story

Following this logic, a successful ABX test cannot prove anything unless you first prove that the author is not lying.

This is completely possible by watching the test being performed.

Sure there is, since reripping an entire collection will take more time than transcoding lossless files to another format. Moreover, if you ever lose your CD(s), you will still have a 'perfect' archival copy.

I understand hushypushy's point, and I agree. True, transcoding from lossless is easier than reripping, but the point is: if you can't abx a lossy file at a given bitrate, and you're using a living codec that has decent hardware support, and you have your files backed up, then the chances of having to rerip for any reason are absurdly low. In several years of digitizing music to mp3 (and now AAC), I've still not had to rerip anything for any reason.

Before you continue inaccurately in this direction, please look up the term "control" as it is used in things like ABX tests. You will find that the test will detect that you're not trying very quickly.

Now, suppose another codec comes along that allows you to make even smaller files, that are still 'transparent' to you. Chances are the 'transparency' will only pertain if you start from a lossless format, as transcoding from one lossy format to another usually means audible losses. So *if* you want to take advantage of that new , smaller lossy format -- and by hushypushy's reasoning he would, since he is interested in optimizing storage space -- you and hushypushy are going to have to rerip, rather than simply transcode.

Well sure, "suppose" being the key word. In that supposition there's the huge assumtion that 64kbps (or something similar) will be transparent. I'm skeptical about that. Does anyone really think it will be? The mp3 and aac I have now will always be transparent for me (and probably for most others as well).

But to concede your point, yes, if 64kbps becomes transparent, I'll be wasting some considerable space, and by not keeping my CDs as lossless, will have to rerip. Which still wouldn't be the end of the world

When transcoding using a lossy format that gives transparent results when encoding lossless tracks, I seriously doubt this.

EDIT: Added bolding to krabapple's quote.

Why is it that you doubt this? It is easy to see how adding quantization noise twice can push noise from under threshold to well over it.

I think the risks of transcoding using transparent settings have been grossly exaggerated, or maybe my experience is based on overly conservative settings.

Is it unreasonable to ask for some ABX results?

I don't think so, because he'd be able to proove it by passing the test .
No, more seriously, I'm still looking for the "control" thing Woodinville told me about, but I didn't have much time yet. Maybe you could give me a pointer where I could look, because "control" is not a very defining search term.

I honestly don't know what is in play to make sure someone actually tries to discern a diffence, though I want to believe that people try their best and put forth an honest effort. Why else would they choose to spend their time doing it?

As for the claim made by krabapple, it, like any other claim regarding sound quality, is subject to TOS #8; even if it is based on strong theoretical grounds which Woodinville appears to be suggesting.

Woodinville is right - a control can be used in properly designed tests to demonstrate the validity of the result. I can't see how it can work for user designed tests (at least with all ABX software I have tried), but in a well controlled test a proper control can go a long way to demonstrating that the testers aren't throwing the result. With ABX you will only need a positive control - the test methodology itself makes a negative control unnecessary.

A control would be something that can reasonably expected to be heard as different by someone with normal listening o music -- RMS average level difference of a couple of dB for example. Or better yet, one can perform a set of preliminary tests, where for example, the difference in levels (or some other parameter, like distortion) of A are B are gradully increased until one can reliable tell them apart in an ABX. That becomes a control.




No more so than this one of yours



Time and again HA posts recommend against serial lossy compression. Has this all been unfounded in fact?

I've used bash scripts to encode MP3s 100 times at 320kbps, and the results are a lot better than I had expected. No discernible volume loss, but the length keeps getting longer by 2 frames. I'll try again with LAME 3.98a7 and try to upload some 20 second samples.

Er... Never mind, 3.98a7 seems to have the same bug with MP3 input...

EDIT: No need to explain this to me. Sorry about the earlier comment, even if I know this already, it is actually worthwhile information. I've never participated in any public listening tests so I don't know if a control test is done or not. For personal tests I do not use a control, but I did at first in order to understand how the ABX test worked. Anyway, I pretty much agree with that Pio was saying; it boils down to honesty.

If you notice, I am not the one making claims about quality. I'm merely suggesting that you back yours up. It should also be obvious that a much more meaningful case would be made if you were to prove that you hear a difference rather than for me to prove that I can't.

Did you prove it for yourself, or are you just repeating what you've read?

Again, I've made no claim other than to challenge yours.

Wow, am I now required to personally test all HA collected wisdom before referencing it on HA? Transcoding from lossy to lossy is disrecommended in the forum FAQ (see "Transcoding/Re-encoding 128kbps to higher bitrate"). The technical reasons are sound. That doesn't mean EVERYONE under EVERY condition will hear an audible hit to quality, but the FAQ doesn't point that out either. Shall we call the FAQ a TOS violation then?

If one has to reinvent the wheel every time, what is the point of HA? In any case, if you or anyone wants to archive in a lossy format, knock yourself out. No one's stopping you. I'm just pointing out a potential pitfall that may manifest itself down the line.

Citing someone else's test would have sufficed.

Anyway, I'll take that as a no, you haven't tried to back your claim with an ABX test.

EDIT: Responding to your edit...
Yet earlier you said, "usually means audible losses."

I don't dispute this. Had you said "might mean audible losses" or had you said (in accordance with the wiki), "will always mean worse quality," I would not have taken issue.

Sorry, no, there are a variety, for instance, of MPEG tests that tested tandeming. And, yeppers, it's generally a bad thing to do unless you use very high rates.




What's your point? The two quotes are entirely consonant.

Feel free to do a little victory dance.



Right, and the two statements are not contradictory.




I suggest you take up the issue with the forum moderators. You might want to start by complaining about the forum FAQ and wiki, e.g.,

http://wiki.hydrogenaudio.org/index.php?title=FAQ

where clicking Transcoding leads you to

http://wiki.hydrogenaudio.org/index.php?title=Transcoding

wherein one reads

"Transcoding between lossy formats is therefore generally not recommended. The sound quality of the result will always be worse than the (lossy) source file."

Also includes links to listening tests of lossy-to-lossy transcoding, which USUALLY show an audible hit.

I think I see how control works with testing medication, you could give a "proven" medication to reference group A and see if the reaction is similar to group B which is testing the new medication. When the results of groups A and B are both bad, you have a test which proves nothing, since there was obviously some kind of problem (e.g. negative placebo, if that term exists). If group B shows good results and group A doesn't, you have shown medication A is not very good.

However, this technique is not usually used in listening tests. When I was talking about a failed ABX test earlier, meant a test consisting only of the standard procedure, done by one person who wants to see if the artifact in question is audible to him. The use of control makes this test more than just a usual abx. Also, I don't know how you would apply control in this situation. But I agree that a controlled abx test could probably disprove an audible difference for most people.

Until then, I'll be awaiting your test results.

As a simple test, you could take a Lame 3.97 -V 4 track that you cannot ABX from the original and re-encode it again with the exact same settings to see if you can ABX it from the original. Maybe try -V 5, but again, only if you can't ABX the first generation mp3 from the original.

And no, it doesn't have to be mp3, nor does it have to be the same format or codec. The only criteria is that you use always use settings that give transparent results upon first generation. This was the claim you were making, no?

Give it a shot, what have you got to lose?

Yes, it is. And it's part of any, and I do mean ANY proper test procedure. I don't care if it's self-administered or not (and any test writeup should include the control details, of course).

Btw, unlike what somebody said above, yes, you do have to include a negative control. You can not a-priori assume that the hardware is neutral. You should always, ALWAYS test it.

It works fine. You just have to do it, like you should do in any good test.




Why? There's test results that show the answers from literally 100's of hours of work by MPEG, BBC, CRC, etc, that show exactly what he said.

He has referred you to the test data. It's you who lack test data as far as I can see.

It is not I who has made a claim which I refuse to back up.

As I eluded earlier, if I were to provide ABX tests that show that I couldn't tell the difference, what would that prove?

...that I don't have the ability to identify artifacts, perhaps?

It isn't my responsibility to disprove a very specific claim that krabapple is unwilling to even prove to himself let alone anyone else except by waving his hands.

Oh, erm... now that I think of it... is that where the low anchor comes in?