Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: What does ABX really prove? (Read 19196 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

What does ABX really prove?

Reply #25
I wouldn't even want to see any lossless abx logs - assuming all samples were bit identical.

What does ABX really prove?

Reply #26
audioslut512:

What is your definition of ABX? What is your testing method?


What does ABX really prove?

Reply #28
Just run the bit-comparator to check you are in fact comparing like for like... it will tell you if there is any difference already...

On 2nd thoughts, why is this nonsense being allowed on HA? I thought I was on Head-Fi for a moment...

What does ABX really prove?

Reply #29
An ABX test cannot prove that there isn't a difference.  It only has the ability to demonstrate that the specific person taking the test was able to determine that there was a difference.  These are not the same thing.

Well in actuality, it can never even prove that the person was able to determine a difference, it can only show how likely it is that the listener can tell a difference. That may seem academic but it's important for the next point:



All I did was ABX Radiohead's Kid A, encoded in FLAC level 4 and level 8, both directly from the CD.  I matched the correct tracks to one another 9 of the 10 times I ran it.  That's all.  Maybe I screwed something up while encoding them that caused one to sound different from the other.  Ultimately I had no idea which one was which, I just managed to notice the differences and match the tracks accordingly.  Hence my post...

I can't remember exactly how the output of fb2k looks (or if it's even the same as last time I used it) but it should have had a percentage at the end which says how likely it is you were just guessing. 9/10 is quite high but if you conduct enough ABX tests clicking randomly you will get every result from 0/10 to 10/10. If you can get the same result again even with ten tests then it becomes much more likely that you have a problem somewhere in your encoding. Otherwise it's probably just chance.

What does ABX really prove?

Reply #30
I would if I had them.  All I did was ABX Radiohead's Kid A, encoded in FLAC level 4 and level 8, both directly from the CD.  I matched the correct tracks to one another 9 of the 10 times I ran it.  That's all.  Maybe I screwed something up while encoding them that caused one to sound different from the other.  Ultimately I had no idea which one was which, I just managed to notice the differences and match the tracks accordingly.  Hence my post...

You sure must have done something wrong somewhere between putting the CD in drive and clicking the ABX utility in foobar2000. As said before in an other post, bitcompare the decoded FLAC files and you'll see the resulting wav files are different.

What does ABX really prove?

Reply #31
"More often than not" does not equal "statistically significant".

Let's say you got 6 out of 10 correct: this is usually written 6/10 at HA. 6/10 has a 37.6% chance of being sheer luck. This is no good.

2/3, 3/4, 3/5, 4/6, 4/7, 5/8, 5/9, 6/10, 6/11... these are all no good.

If you're saying something like 60/100 then that might mean something, but unless you give us those numbers, you're just being vague and might well be fooling yourself.

What does ABX really prove?

Reply #32
All I did was ABX Radiohead's Kid A, encoded in FLAC level 4 and level 8, both directly from the CD.
Do I get you right, you ripped it twice from the CD? And what does the difference you hear sound like? Some popping noise, some click?

What does ABX really prove?

Reply #33
All I did was ABX Radiohead's Kid A, encoded in FLAC level 4 and level 8, both directly from the CD.
Do I get you right, you ripped it twice from the CD? And what does the difference you hear sound like? Some popping noise, some click?



It was actually quite interesting...

When I listened to substantial chunks of each track I couldn't tell the difference at all.

I then came up with a little method, of switching between A,X,Y,B, in rapid succession, with about 1-2 seconds in between each switch.

I then noticed that if when I did this, cycling through the four buttons clockwise for example, I would hear shifts in the overall sound (im not sure how to desribe those shifts, I just referred to them in my mind as up and down).

So in doing so, if I heard upshift, downshift, upshift, downshift, I would then answer accordingly, and got it right 9 out of 10 times.

Like I said before though, I was never challenging ABX, and admitted that I might have screwed up the encoding processes of the tracks, and maybe that was what caused me to be able to discern the difference.

The tracks were both ripped from the same CD however, and both were ripped under the same conditions (i.e, no other programs running at the time).

I suppose I could and do it again to see if its repeatable, and to maybe stop so many people from bashing me, but my post was never about the ABX test I did on those tracks to begin with, it was about a question that those tests led me to...

What does ABX really prove?

Reply #34
The point is that you cannot make a proper conclusion from an ABX test if you don't know how conduct one correctly.

eevan really hammered the point home if you ask me:
Quote
Why should this be of any interest if you say that it's irrelevant to pinpoint the cause of your test result.


What does ABX really prove?

Reply #36
You chose not to accept the fact that you couldn't ABX substantial chunks and you did not take the proper steps to ensure that there wasn't an error in the encodings or that they used the exact same source.

Some good may come out of this discussion after all since you may have identified a problem with the way foobar2000's ABX test.  Too bad it took all the way to the 34th post for some useful albeit still qualitative information to come out.

You are aware that foobar2000 decodes the files you wish to compare to wave and places them in a temporary directory?  Being that these were all lossless encodes, you could have done a binary comparison as a simple sanity check.

It's quite possible that subsequent rips of the same track can produce different results, even under seemingly similar circumstances; and yes, these would have been revealed with a binary comparison.  A ripping error is the only logical conclusion if a binary comparison revealed differences.  Lossless is lossless after all!

What does ABX really prove?

Reply #37
There is just no point in ABXing lossless files. If you suspect an encoding error you should do a bit comparison. Either there is no difference or there is. No point in listening for a difference in either case.

What does ABX really prove?

Reply #38
There is just no point in ABXing lossless files.
It all depends on what you're trying to compare.  What if a track gives me ripping errors and I want to see if I can hear them?

What does ABX really prove?

Reply #39
He doesn't even know if there are errors. It's the first thing to do. Run the bit comparator already!


What does ABX really prove?

Reply #41
You chose not to accept the fact that you couldn't ABX substantial chunks and you did not take the proper steps to ensure that there wasn't an error in the encodings or that they used the exact same source.

Some good may come out of this discussion after all since you may have identified a problem with the way foobar2000's ABX test.  Too bad it took all the way to the 34th post for some useful albeit still qualitative information to come out.

You are aware that foobar2000 decodes the files you wish to compare to wave and places them in a temporary directory?  Being that these were all lossless encodes, you could have done a binary comparison as a simple sanity check.

It's quite possible that subsequent rips of the same track can produce different results, even under seemingly similar circumstances; and yes, these would have been revealed with a binary comparison.  A ripping error is the only logical conclusion if a binary comparison revealed differences.  Lossless is lossless after all!



It's not that I chose not to accept that I couldn't ABX substantial chunks, it's just that I didn't know any better, and for that matter still don't.

Now that you mention binary comparison, I will look into it.  That is yet another thing I am trying to understand...


What does ABX really prove?

Reply #43

I still don't really understand what you did.  Could you just post the logs?



I would if I had them.  All I did was ABX Radiohead's Kid A, encoded in FLAC level 4 and level 8, both directly from the CD.  I matched the correct tracks to one another 9 of the 10 times I ran it.  That's all.  Maybe I screwed something up while encoding them that caused one to sound different from the other.  Ultimately I had no idea which one was which, I just managed to notice the differences and match the tracks accordingly.  Hence my post...


You more likely screwed up when testing and not when encoding. You are saying that you perceived a difference between two things that are EXACTLY the same.

What does ABX really prove?

Reply #44
Statistically significant is that which is unlikely to occur by chance.

A statistically significant difference in this case, merely implies that statistical evidence indicates that a discernible difference has a greater probability of existing than it does a probability of not existing.

Whether that difference is due to errors in encoding, playback, or whatever, is entirely irrelevant.



If you repeatedly, 'successfully' (>16 trials each, p<0.05) ABXd flac from flac, or flac from .wav, all using the same track, then your flac encoding/decoding is broken, or your ABX test was crap.  Seriously. If you understood what you are claiming, you'd know this must be true.

What does ABX really prove?

Reply #45
Not to mention what author is doing with his ABXing but to note people who says that ABX-ing lossless is pointless in any cases - I have one example (theoretical though): for testing decoder part. Can it happen in portable player that lack of processing power will lead to produce decoding errors in higer compression levels? Or software bugs will allow one lossless format sound better than another? You can't easily bit-compare DAP output...

What does ABX really prove?

Reply #46
Can it happen in portable player that lack of processing power will lead to produce decoding errors in higer compression levels?

There's a heated argument that occasionally flares up over on the Slim Devices forum that's related to this speculation:

Some people claim that if they decode their FLACs on the server and stream uncompressed WAV to a Squeezebox, it sounds different than if they stream FLAC and have it decoded in the Squeezebox.

The hypothesis is that the additional CPU activity required to decode the FLAC in the Squeezebox might possibly generate some RFI that affects the analogue circuitry. (Note: there is absolutely no suggestion that the two processes result in different digital data). If you have an open mind, you have to concede that this is at least possible, however unlikely. (On the other hand, those who have taken the trouble to do a blind comparison failed to distinguish streaming FLAC and WAV).

I'm sure the same people who believe FLAC and WAV sound different in the above scenario would have no trouble speculating that the different CPU activity required to decode different levels of FLAC compression might result in different RFI which would in turn affect surrounding analogue circuitry in different ways.

My view? It seems unlikely in the extreme that any such effect (if it exists) will be audible. Life is certainly too short to bother testing it rigorously.

What does ABX really prove?

Reply #47
Trolling for whom?

I am just trying to understand how this stuff works and get crucified for it.

Thanks again.

I'm sorry, then.what.was.the.quote.for?

just forget it kids.


Wasn't it on the part of your closed mind to refuse to accept the logical explanations provided by the more knowledgeable individuals in this forum? And you blame others for rightly 'crucifying' your obstinacy?

The entire forum is still waiting for your bit-comparisons and ABX logs with p values < 0.1%.

What does ABX really prove?

Reply #48
Well, I just noticed that even I *might* be able to ABX FLAC -4 from -8, given the CPU usage pattern explanation... you see, my CPU emits a (very low-volume) high-pitched whine when it's idle, that changes when the CPU is used. The problem is well known for the Core2-based MacBook IIRC, but also occurs on other systems including my laptop, apparently.

Well, if decoding the two files causes different CPU usage, it might very well be possible to hear a small difference because of this phenomenon. Because of its high pitch, I do hear the noise over music I play back when running Linux (it's worse there, don't ask me why), but I usually have something running in the background (Seti@Home, Prime95) to keep the processor occupied, which stops the whine.

If both files are decoded completely before playback however, I don't think it should be possible.

 

What does ABX really prove?

Reply #49
@MedO-
-When you run anythingABX test, I always found the testing program to create two WAVs of the inputted files, eliminating any decoding lags or "hiccups." I have seen this with a stand alone ABX utility, and I believe FB2K does this. There is no way, to my understanding, that this ABX ws performed properly. EDIT: It is unlikely that the processor lag or hardware strain noise could give one an advantage when performing an ABX test on two lossless sources.
OP can't edit initial post when a solution is determined  :'-(