IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Mad challenge - my result
Gabriel
post Nov 6 2003, 12:42
Post #1


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



I've tryed the Mad vs FhG challenge at 16 bits. (quite hard with this bad integrated sound chip):

0 of 1, p = 1.000
1 of 2, p = 0.750
2 of 3, p = 0.500
3 of 4, p = 0.313
4 of 5, p = 0.188
4 of 6, p = 0.344
5 of 7, p = 0.227
6 of 8, p = 0.145
7 of 9, p = 0.090
8 of 10, p = 0.055
8 of 11, p = 0.113
8 of 12, p = 0.194
8 of 13, p = 0.291
8 of 14, p = 0.395
9 of 15, p = 0.304
10 of 16, p = 0.227
10 of 17, p = 0.315
11 of 18, p = 0.240
11 of 19, p = 0.324
12 of 20, p = 0.252
13 of 21, p = 0.192
13 of 22, p = 0.262
14 of 23, p = 0.202
15 of 24, p = 0.154
16 of 25, p = 0.115
17 of 26, p = 0.084
Go to the top of the page
+Quote Post
amano
post Nov 7 2003, 03:02
Post #2





Group: Members
Posts: 483
Joined: 1-December 02
Member No.: 3949



sorry gabriel. what do these values tell me?
Go to the top of the page
+Quote Post
Compact Dick
post Nov 7 2003, 03:14
Post #3





Group: Members
Posts: 156
Joined: 6-April 02
Member No.: 1707



QUOTE (amano @ 2003-11-07T02:02Z)
sorry gabriel. what do these values tell me?

I believe this means that he was successfully able to ABX FhG's MP3 decoder from the MAD decoder 17 out of 26 times.

Edit: Changed quote date and time to ISO 8601 format.

This post has been edited by Compact Dick: Nov 20 2003, 22:42
Go to the top of the page
+Quote Post
Digga
post Nov 7 2003, 05:07
Post #4





Group: Members
Posts: 1046
Joined: 28-June 03
From: on the dock of the bay
Member No.: 7423



QUOTE (Compact Dick @ Nov 7 2003, 03:14 AM)
QUOTE (amano @ Nov 7 2003, 02:02 AM)
sorry gabriel. what do these values tell me?

I believe this means that he was successfully able to ABX FhG's MP3 decoder from the MAD decoder 17 out of 26 times.

...or MAD from FhG... maybe you could clear things up Gabriel smile.gif

This post has been edited by Digga: Nov 7 2003, 05:08


--------------------
Nothing but a Heartache - Since I found my Baby ;)
Go to the top of the page
+Quote Post
phong
post Nov 7 2003, 05:11
Post #5





Group: Members
Posts: 346
Joined: 7-July 03
From: 15 & Ryan
Member No.: 7619



Usually, a p <= 0.05 is considered a significant result. This is pretty close though, which to me indicates that there is a very good chance that more testing would result in a statistically significant result.


--------------------
I am *expanding!* It is so much *squishy* to *smell* you! *Campers* are the best! I have *anticipation* and then what? Better parties in *the middle* for sure.
http://www.phong.org/
Go to the top of the page
+Quote Post
Hyperion
post Nov 7 2003, 05:18
Post #6





Group: Members
Posts: 115
Joined: 8-January 03
Member No.: 4467



Gabriel, Which Version is The MAD? ohmy.gif

Downloads? dry.gif


--------------------
It's My Life,
It's Now Or Never,
I Ain't Gonna Live Forever,
I Just Want To Live While I'm Alive.
Go to the top of the page
+Quote Post
Digga
post Nov 7 2003, 05:21
Post #7





Group: Members
Posts: 1046
Joined: 28-June 03
From: on the dock of the bay
Member No.: 7423



QUOTE (phong @ Nov 7 2003, 05:11 AM)
Usually, a p <= 0.05 is considered a significant result.  This is pretty close though

yes, it would be intersting to see some further results.
AFAIK a p-value of 0,01 is more generally accepted, 0,05 is little more like 'bending' the rules (though it is also widly used in tests).
care to do some further testing Gabriel?

This post has been edited by Digga: Nov 7 2003, 05:27


--------------------
Nothing but a Heartache - Since I found my Baby ;)
Go to the top of the page
+Quote Post
rjamorim
post Nov 7 2003, 05:37
Post #8


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Hyperion @ Nov 7 2003, 01:18 AM)
Downloads?  dry.gif

http://ff123.net/madchallenge.html


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Gabriel
post Nov 7 2003, 10:13
Post #9


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



QUOTE
...or MAD from FhG... maybe you could clear things up Gabriel

I do not see any difference between discerning FhG from Mad or Mad from FhG.


QUOTE
Which Version is The MAD?

Mad 0.14.2b
FhG decoder 2.91


I think that it will be hard for me to do better with this sound card. When you push up the volume, there is a lot of noise.
Go to the top of the page
+Quote Post
Digga
post Nov 7 2003, 10:21
Post #10





Group: Members
Posts: 1046
Joined: 28-June 03
From: on the dock of the bay
Member No.: 7423



QUOTE (Gabriel @ Nov 7 2003, 10:13 AM)
QUOTE
...or MAD from FhG... maybe you could clear things up Gabriel

I do not see any difference between discerning FhG from Mad or Mad from FhG.

there's no differnece the way you say it. however, I just wanted to know which of the two represents the numbers in the end. did you mange to put your finger on MAD or Fhg 17 times of 26? sorry if I didn't make that clear.


--------------------
Nothing but a Heartache - Since I found my Baby ;)
Go to the top of the page
+Quote Post
Continuum
post Nov 7 2003, 10:31
Post #11





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



QUOTE (Digga @ Nov 7 2003, 11:21 AM)
QUOTE (Gabriel @ Nov 7 2003, 10:13 AM)
QUOTE
...or MAD from FhG... maybe you could clear things up Gabriel

I do not see any difference between discerning FhG from Mad or Mad from FhG.

there's no differnece the way you say it. however, I just wanted to know which of the two represents the numbers in the end. did you mange to put your finger on MAD or Fhg 17 times of 26? sorry if I didn't make that clear.

You are missing the point in an ABX-test. An unknown sample X (which is either A or B, randomly chosen at each trial) is compared to two references (A and B ). The tester has to decide if "X is the same as A" or if "X is the same as B" is true.

Out of 26 trials taken, 17 times the answer was correct. This is not connected to whether A is MAD or Fhg.

This post has been edited by Continuum: Nov 7 2003, 10:32
Go to the top of the page
+Quote Post
Digga
post Nov 7 2003, 10:40
Post #12





Group: Members
Posts: 1046
Joined: 28-June 03
From: on the dock of the bay
Member No.: 7423



right.
and in order to tell the difference, one file have to be worse quality (or sound somehow differnet to you) than the other, am I right? otherwise you could not tell a difference...


--------------------
Nothing but a Heartache - Since I found my Baby ;)
Go to the top of the page
+Quote Post
Continuum
post Nov 7 2003, 10:45
Post #13





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



Yes, but the difference relation is symmetric.
(A /= B is equivalent to B /= A)

Is your question, what sounded better to him?
Go to the top of the page
+Quote Post
Digga
post Nov 7 2003, 10:53
Post #14





Group: Members
Posts: 1046
Joined: 28-June 03
From: on the dock of the bay
Member No.: 7423



exactly smile.gif


--------------------
Nothing but a Heartache - Since I found my Baby ;)
Go to the top of the page
+Quote Post
Gabriel
post Nov 7 2003, 11:04
Post #15


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



QUOTE
Is your question, what sounded better to him?

None sounded better, just slightly different.

You have to keep in mind that I did not achieved 99% confidence, but only 91.6% (94.5 at max).
It means that you can not really considere it as a definitive answer to the Mad vs FhG challenge. It gives a positive indication, that is all.

This post has been edited by Gabriel: Nov 7 2003, 11:07
Go to the top of the page
+Quote Post
menno
post Nov 7 2003, 12:55
Post #16


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



I just did a compliance test for both MAD 0.15.0 and the decoder in Winamp (latest). Result is that for the comliance sine sweep MAD (without dithering) reaches 23 bit accuracy and Winamp 16 (winamp output was 16 bit, so that's the maximum). However on the non-sine sweep reference files the Winamp decoder was a lot less accurate ("only" 15 bit accuracy), while MAD kept 21 or 22 bit accuracy. Except for 1 file where both decoders reached an accuracy of 0 bit (1kHz 0dB sine).

However (!!!), when I tried the MAD plugin for winamp (0.14.2), with 16 bit output, it reached only 14 bits accuracy on all files except the one 0 bit file (and 16 bit on the compliance sine sweep).

So apparantly the difference is quite big. FhG screws up but MAD screws up even more. Does this MAD Winamp plugin do dithering? And yes, my guess is that for some people this difference is big enough to be audible.

Menno

edit:
Note: command line MAD with 16 bit output and no dithering produced 15 to 16 bit accuracy except for the 1kHz sine.

This post has been edited by menno: Nov 7 2003, 13:05
Go to the top of the page
+Quote Post
menno
post Nov 7 2003, 13:03
Post #17


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



BTW: Foobar did not reach more than 18 bit accuracy on 24 bit undithered output (however still fully 16 bit compliant), but when using 16 bit undithered output it gave a result of 15 bit accuracy for most files (including the compliance sine sweep) and 16 bit on 2 files. On that 0dB 1kHz sine wave it got 14 bit (!!).

Menno
Go to the top of the page
+Quote Post
Gabriel
post Nov 7 2003, 14:01
Post #18


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



QUOTE
FhG screws up but MAD screws up even more. Does this MAD Winamp plugin do dithering?


Yes, the winamp plugin should do dithering. This is why I think that it should only reach 15bits accuracy, as the lowest bit is dithered. However, there is still 1 bit missing.
Go to the top of the page
+Quote Post
menno
post Nov 7 2003, 14:01
Post #19


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



Oh wait, I made a mistake with the FhG delay on that 1kHz file, it produces 15 bit accuracy there.

Menno
Go to the top of the page
+Quote Post
menno
post Nov 7 2003, 14:12
Post #20


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



QUOTE (Gabriel @ Nov 7 2003, 02:01 PM)
QUOTE
FhG screws up but MAD screws up even more. Does this MAD Winamp plugin do dithering?


Yes, the winamp plugin should do dithering. This is why I think that it should only reach 15bits accuracy, as the lowest bit is dithered. However, there is still 1 bit missing.

Hmm, so it is 15 bit like the undithered output, but loses 1 because of the dithering -> 14.

FhG and MAD only have 16 bit accurate output on the compliance sine sweep. But then again, this is where the compliance is based on, but it is "recommended" that the other files also produce 16 bit accurate output.

I find it strange that MAD produces 22/23 bit accurate results when decoding to 24 bit but produces only 15 bit results on those files when decoding to 16 bit. Must be something with the rounding.

Menno
Go to the top of the page
+Quote Post
Pio2001
post Nov 9 2003, 23:21
Post #21


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



QUOTE (Gabriel @ Nov 7 2003, 11:04 AM)
You have to keep in mind that I did not achieved 99% confidence, but only 91.6% (94.5 at max).

Also, you didn't seem to have fixed the number of trials before starting. This way (testing until the score is good enough) leads to an overestimation of the confidence, because the result is biased towards a good score.
For example if you had chosen to stop at 16 to begin with, your result would have been 77.3 % only. This must be taken into account in the final estimation.
But it is so complicated that we prefer to fix the number of trials before.
Note that you can still change the number of trials during the test as long as you have not seen your previous results.
Go to the top of the page
+Quote Post
KikeG
post Nov 10 2003, 14:47
Post #22


WinABX developer


Group: Developer
Posts: 1578
Joined: 1-October 01
Member No.: 137



QUOTE (menno @ Nov 7 2003, 02:12 PM)
Hmm, so it is 15 bit like the undithered output, but loses 1 because of the dithering -> 14.

In theory, using flat dither with as much as 1 bit of amplitude, dither would only have an effect on the 16th bit, so the 15th bit wout not be touched.

Edit: But if some kind of noiseshaping dither is used, it could have an effect on higher-level bits too (bits < 16).

This post has been edited by KikeG: Nov 10 2003, 14:51
Go to the top of the page
+Quote Post
amano
post Nov 11 2003, 18:32
Post #23





Group: Members
Posts: 483
Joined: 1-December 02
Member No.: 3949



A bit off topic:

Is the MAD Winamp plugin developed further these days? It's still 0.14.2b and is lacking updates for some time. AFAIK it doesn't support the extended metadata interface used by Winamp Library yet (so working with current 2.9+ Winamp versions isn't very convenient).

Who has any infos?


(sorry for the bit off topic, amano)
Go to the top of the page
+Quote Post
Pio2001
post Nov 11 2003, 20:33
Post #24


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



OT discussion about the validity of sequential ABX tests splitted there :

http://www.hydrogenaudio.org/forums/index....topic=15151&hl=
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 18th April 2014 - 02:09