I've tryed the Mad vs FhG challenge at 16 bits. (quite hard with this bad integrated sound chip):

0 of 1, p = 1.000
1 of 2, p = 0.750
2 of 3, p = 0.500
3 of 4, p = 0.313
4 of 5, p = 0.188
4 of 6, p = 0.344
5 of 7, p = 0.227
6 of 8, p = 0.145
7 of 9, p = 0.090
8 of 10, p = 0.055
8 of 11, p = 0.113
8 of 12, p = 0.194
8 of 13, p = 0.291
8 of 14, p = 0.395
9 of 15, p = 0.304
10 of 16, p = 0.227
10 of 17, p = 0.315
11 of 18, p = 0.240
11 of 19, p = 0.324
12 of 20, p = 0.252
13 of 21, p = 0.192
13 of 22, p = 0.262
14 of 23, p = 0.202
15 of 24, p = 0.154
16 of 25, p = 0.115
17 of 26, p = 0.084

sorry gabriel. what do these values tell me?

I believe this means that he was successfully able to ABX FhG's MP3 decoder from the MAD decoder 17 out of 26 times.

Edit: Changed quote date and time to ISO 8601 format.

...or MAD from FhG... maybe you could clear things up Gabriel

Usually, a p <= 0.05 is considered a significant result. This is pretty close though, which to me indicates that there is a very good chance that more testing would result in a statistically significant result.

Gabriel, Which Version is The MAD?

Downloads?

yes, it would be intersting to see some further results.
AFAIK a p-value of 0,01 is more generally accepted, 0,05 is little more like 'bending' the rules (though it is also widly used in tests).
care to do some further testing Gabriel?

http://ff123.net/madchallenge.html

I do not see any difference between discerning FhG from Mad or Mad from FhG.



Mad 0.14.2b
FhG decoder 2.91


I think that it will be hard for me to do better with this sound card. When you push up the volume, there is a lot of noise.

there's no differnece the way you say it. however, I just wanted to know which of the two represents the numbers in the end. did you mange to put your finger on MAD or Fhg 17 times of 26? sorry if I didn't make that clear.

You are missing the point in an ABX-test. An unknown sample X (which is either A or B, randomly chosen at each trial) is compared to two references (A and B ). The tester has to decide if "X is the same as A" or if "X is the same as B" is true.

Out of 26 trials taken, 17 times the answer was correct. This is not connected to whether A is MAD or Fhg.

right.
and in order to tell the difference, one file have to be worse quality (or sound somehow differnet to you) than the other, am I right? otherwise you could not tell a difference...

Yes, but the difference relation is symmetric.
(A /= B is equivalent to B /= A)

Is your question, what sounded better to him?

exactly

None sounded better, just slightly different.

You have to keep in mind that I did not achieved 99% confidence, but only 91.6% (94.5 at max).
It means that you can not really considere it as a definitive answer to the Mad vs FhG challenge. It gives a positive indication, that is all.

I just did a compliance test for both MAD 0.15.0 and the decoder in Winamp (latest). Result is that for the comliance sine sweep MAD (without dithering) reaches 23 bit accuracy and Winamp 16 (winamp output was 16 bit, so that's the maximum). However on the non-sine sweep reference files the Winamp decoder was a lot less accurate ("only" 15 bit accuracy), while MAD kept 21 or 22 bit accuracy. Except for 1 file where both decoders reached an accuracy of 0 bit (1kHz 0dB sine).

However (!!!), when I tried the MAD plugin for winamp (0.14.2), with 16 bit output, it reached only 14 bits accuracy on all files except the one 0 bit file (and 16 bit on the compliance sine sweep).

So apparantly the difference is quite big. FhG screws up but MAD screws up even more. Does this MAD Winamp plugin do dithering? And yes, my guess is that for some people this difference is big enough to be audible.

Menno

edit:
Note: command line MAD with 16 bit output and no dithering produced 15 to 16 bit accuracy except for the 1kHz sine.

BTW: Foobar did not reach more than 18 bit accuracy on 24 bit undithered output (however still fully 16 bit compliant), but when using 16 bit undithered output it gave a result of 15 bit accuracy for most files (including the compliance sine sweep) and 16 bit on 2 files. On that 0dB 1kHz sine wave it got 14 bit (!!).

Menno

Yes, the winamp plugin should do dithering. This is why I think that it should only reach 15bits accuracy, as the lowest bit is dithered. However, there is still 1 bit missing.

Oh wait, I made a mistake with the FhG delay on that 1kHz file, it produces 15 bit accuracy there.

Menno

Hmm, so it is 15 bit like the undithered output, but loses 1 because of the dithering -> 14.

FhG and MAD only have 16 bit accurate output on the compliance sine sweep. But then again, this is where the compliance is based on, but it is "recommended" that the other files also produce 16 bit accurate output.

I find it strange that MAD produces 22/23 bit accurate results when decoding to 24 bit but produces only 15 bit results on those files when decoding to 16 bit. Must be something with the rounding.

Menno

Also, you didn't seem to have fixed the number of trials before starting. This way (testing until the score is good enough) leads to an overestimation of the confidence, because the result is biased towards a good score.
For example if you had chosen to stop at 16 to begin with, your result would have been 77.3 % only. This must be taken into account in the final estimation.
But it is so complicated that we prefer to fix the number of trials before.
Note that you can still change the number of trials during the test as long as you have not seen your previous results.

In theory, using flat dither with as much as 1 bit of amplitude, dither would only have an effect on the 16th bit, so the 15th bit wout not be touched.

Edit: But if some kind of noiseshaping dither is used, it could have an effect on higher-level bits too (bits < 16).

A bit off topic:

Is the MAD Winamp plugin developed further these days? It's still 0.14.2b and is lacking updates for some time. AFAIK it doesn't support the extended metadata interface used by Winamp Library yet (so working with current 2.9+ Winamp versions isn't very convenient).

Who has any infos?


(sorry for the bit off topic, amano)

OT discussion about the validity of sequential ABX tests splitted there :

http://www.hydrogenaudio.org/forums/index....topic=15151&hl=