Blind test challenge, Find the SB64 recording ! Options

[Pio2001]

Can anyone ABX any of these samples vs the original (that is resampled and leveled, but in the digital domain, all others are analog copies) ?

Please, tell me I'm either tired or deaf.

KikeG, there are soundcards analog recordings, you might be interested.

Right click to download original.flac
Right click to download 2.flac
Right click to download 3.flac
Right click to download 4.flac
Right click to download 5.flac

Less than 900 kB each

[KikeG]

Nice, I'll give a try when I have some time.

[Pio2001]

I was unable to ABX any of these samples with the original, though I can hear the SB64 one. Isn't it strange ? I can hear the coloration added by the SB64, and a spectral analysis showed exactly the defects I hear in the frequency, but I got only 4/8 on ABX.

I think that ABXing the sound of such samples, that show continuous difference in the sound itself, rather than localized artifacts audible at a given place, is more difficult. The difference I hear seems to vanish after listening 4 or 5 times to the samples.

I'll try slower, maybe one session per day, to see if there is an improvement in my scores. It is possible that some differences in the sound are very difficult to get right in an ABX test with short samples played over and over.

I still can use another sample for the test (audiophiles would maybe prefer a Rebecca Pidgeon recording), and / or resample and level all analog recording to 44.1 kHz so as to have a bit perfect original to compare with various distorded files.
The later would be interesting if tested on audiophile CD players, and if the ABX is impossible (would prove that some device don't add distortion to the sound), while the current setup is better at comparing the tested devices between them (analog samples are closer to the analog source).

[Patsoe]

I'm not really understanding what this is about. But since you sent me to this thread, I guess its about using the analog out of your soundcard with different cables?
Should we listen to the samples before knowing what they are made of? I mean, if I put them in abx-software it will still be blind even if you tell us what you did to the samples, right?

Anyway, I'm not very familiar with ABX yet, so I'll first practice with some example files before trying these, probably hardly distinguishable recordings. I'll report back in a few days, then.

...while the current setup is better at comparing the tested devices between them (analog samples are closer to the analog source).

I really don't get this. Could you explain some more?

[KikeG]

I was unable to ABX any of these samples with the original, though I can hear the SB64 one. Isn't it strange ? I can hear the coloration added by the SB64, and a spectral analysis showed exactly the defects I hear in the frequency, but I got only 4/8 on ABX.

I think that ABXing the sound of such samples, that show continuous difference in the sound itself, rather than localized artifacts audible at a given place, is more difficult. The difference I hear seems to vanish after listening 4 or 5 times to the samples.

If this is what is happening to you (you get good scores at first, and bad at last, differences vanish as the test goes on) you need to take the test in a more relaxed manner, without stressing too much, taking your time after every trial. Sometimes listening tireness causes this effect. On the other side, concentrating a lot can help overcoming tireness... I guess the best procedure dependes on everyone.

I'll try slower, maybe one session per day, to see if there is an improvement in my scores.

That could be a posible way of avoiding tireness.

It is possible that some differences in the sound are very difficult to get right in an ABX test with short samples played over and over.

Mmm... the important thing is if you can tell them appart reliabily in a blind manner. "Usual" ABX is only a standard procedure to achieve this, but the precise method can be "flexibilized" to suit your preferences, as long as blindness and statistical validity is assured.

The later would be interesting if tested on audiophile CD players, and if the ABX is impossible (would prove that some device don't add distortion to the sound)...

All analog input/output devices add distortion, the thing is how big is this distortion and if it's audible or not. This depends also on the type of music used, I'd say noisy and already distorted music is worse to detect further distortion.

This post has been edited by KikeG: Feb 19 2003, 17:15

[Pio2001]

I'm not really understanding what this is about. But since you sent me to this thread, I guess its about using the analog out of your soundcard with different cables?

I sent you there because among these samples, there is a recording of the analog output of the Marian Marc 2 soundcard, about which you asked. I can tell you in private message which one it is. This way, listening to it, you may see if there is an obvious problem with the sound for you.
The other samples are, and the subject shows, a recording of the SoundBlaster 64V PCI output, and two other ones, that are not recorded from soundcards.

Should we listen to the samples before knowing what they are made of? I mean, if I put them in abx-software it will still be blind even if you tell us what you did to the samples, right?

Yes, it would still be blind. But knowing what they are, you may try to find some expected sound characteristics in the samples. It's better to let people give a unbiased opinion rather than to ask them which sample sounds rather like this or like that, especially if we want to test, blindly, if a device really has the sound we expect it to have.
Example : let's say I hid a vinyl recording among them. If you know it, you are going to search for clicks and noise, maybe mistake a CD click or MP3 noise for them, and tell that a CD or MP3 file "clicks like a vinyl", while otherwise, you would maybe not have thought about it this way.

I'll report back in a few days, then.

Thanks in advance for anyone willing to participate, but no one is forced to do so, if you have not got the time, but are interested, you can ask me in a private message which sample is what (but I'll ask you not to reveal them until I do it myself).

...while the current setup is better at comparing the tested devices between them (analog samples are closer to the analog source).

I really don't get this. Could you explain some more?

I've got a problem : the external ADC I use for recording analog sources is a Sony DTC 55ES, and it doesn't support 44100 Hz sample rate. Only 32 and 48 kHz, as most early consumer DAT decks. However, the original is 44.1 kHz, since it is a CD.
So to compare original vs copies, we end up comparing 44.1 kHz and 48 kHz. This is not good, since soundcards can behave differently at different sample rates (think about resampling).

There are two solutions :
1) Resample the 48 kHz recordings to 44.1 kHz
2) Resample the original to 48 kHz.

I did the second choice, so as to provide test samples as close as possible to the analog source. This way, comparing samples between them, the differences in quality are not attenuated by a common resampling process.
On the other hand, since some samples can be difficult to ABX, the first option would have allowed to compare a soundcard recording, for example, to the unmodified original. In the case a difference is audible, it will be difficult to know if it is the soundcard's fault or the resampling to 44.1 kHz fault, but imagine that someone uses a listening system superior to the devices tested, eg a high end audiophile CD Player, reading a CDR with the samples burned on it. If he doesn't hear the difference between the bit exact original and the analog resampled copy, it would mean the the soundcard tested (for example) is really very good.

I can upload 44.1 kHz versions if someone wants to burn the files on CDR and perform the tests on a high end hifi system.

[Patsoe]

Thanks for the thorough explanation. I'm very interested in doing this as an abx comparison, so I won't ask for the answers just now.

Just a last question though: do you think I should be able to detect a difference, since I have no equipment like a Marian Marc (not yet, that is )? I'm using a SB128 with Academy monitoring headphones. I could borrow a SB Live from my room mate and install it with kX drivers, but I guess that wouldn't help too much...?

[Pio2001]

You may detect the SB64 sample. The other ones will be very difficult. It would be better to perform the test on a CD player, if you've got a good one, but in this case, I should upload a 44.1 kHz version.

EDIT : I wonder if an MP3 compatible standalone DVD Player can play wav files as well (48 kHz).

[Patsoe]

You may detect the SB64 sample. The other ones will be very difficult. It would be better to perform the test on a CD player, if you've got a good one...

I don't have a very high-end unit. It's a Denon, type DCD-6..something (screaming front saying 20bit labda dac or so). Sorry, I would have enjoyed participating. Perhaps I'll just go out and get the Marian card

[KikeG]

I've been able to ABX one of the samples conclusively.

I have to admit I did a little trick to save time and effort. I did a quick and not accurate comparative spectral analysis in order to know which is the file that seemed to have worse frequency response, and found one with quite many attenuation of frequencies above 20 KHz, so I assumed this was the worst one, and the one I tried to ABX.

The computer I used for the test has a not very good SoundMax built-in soundcard, and I did tests in a quite noisy environment, with headphones connected to the soundcard.

First I tried with my somehow boomy Sony MDR-7506, and seemed to hear some diferences, but I was unable to get any good ABX scores. Then I tried plugging my old Seenheiser HD560, that gave a quite more detailed picture, and started hearing some subtle but clear differences.

On two different parts of the song, I got 7/7 (p<0.8) and 14/17 (p<0.6%). I didn't try the other samples.

Pio has more details, that I will post when he thinks it's ok to do so.

[KikeG]

More:

I guess I couldn't abx the file with the Sony headphones also because I was looking for a treble reduction, or something similar. Now that I know where the difference lies (I've also performed a detailed frequency response analysis), I've been able to ABX without much difficulty the file too with the Sony headphones (10/11 p<0.6%, 12/14 p<0.6%).

[Continuum]

I have only tried original><file 5 yet. I was quiet confident that I heard a difference in the first 4 trials (4/4) but then the impression vanished and the results dropped.

[Bedeox]

I got 14/15 result after half hour meditation.
Without, I had only 7/15... weird, isn't it?

<edit>
I'm using headphones too...

Of course it might have been a lucky shot... I'll try to do it again.

Heh. Forgot to mention I was testing against file 5.
</edit>

This post has been edited by Bedeox: Feb 21 2003, 22:07

[voltron]

Here are my ABX results. I never want to hear those samples ever again!






I am not sure if I even did this test correctly, but my best guess is that the SB64 is sample 2. Can someone message me the correct answer? Thank you.

System specs: Turtlebeach SantaCruz with Sony MDR-v250 Headphones.

voltron

This post has been edited by voltron: Feb 22 2003, 02:39

[Pio2001]

Thank you people !

Well, the results are not going in the direction I expected ! You have better ears than me. This test would have been conclusive if some of the samples were not ABXable at all, but maybe I'll have to ask for two more ABX (between two samples each, that I yet have to record) to draw a conclusion.

If you've got extra time, you can already try to ABX 3 vs 5. According to what is usually said by the hardcore ABXers in HA, this should be impossible.

[Continuum]

Here are my ABX results. I never want to hear those samples ever again! 

Uhm.. wouldn't it be easier to use the text written to the log-file?

[KikeG]

Voltron, in order to ABX succesfully a sample you must get a minimum probability of guessing of < 5%, and <1% is better to make totally sure it's not by chance. So, the only sample you could say to have abx'ed is 4.wav.

This post has been edited by KikeG: Feb 23 2003, 23:03

[KikeG]

By the way, the sample I abxed was 4.wav.

Bedeox, 14/15 is below 0.1% of guessing, I doubt you did it by chance.

[Bedeox]

It might be chance as well (this means, 0.1% hit, very rare, but possible)
I did the test again, and it IS different. (6/7) (5.wav)

<edit>
Forgot to say that I can't ABX 3.wav vs 5.wav.
</edit>

This post has been edited by Bedeox: Feb 24 2003, 11:20

[KikeG]

I ABXed yesterday 5.wav (17/22 p= 0.8%). At first I couldn't really notice any differences, just my imagination, so I didn't get any good scores. Then, I compared spectrums of the files, and saw some differences. After that, I repeated the test focusing on the theorical differences, and started hearing them.

This post has been edited by KikeG: Feb 25 2003, 09:50

[Pio2001]

It's going to be time to tell the results, and propose further samples (the current results are little informative, due to the limited choice I made).
Patsoe, must we wait for you, or do you give up ?

[Patsoe]

It's going to be time to tell the results, and propose further samples (the current results are little informative, due to the limited choice I made).
Patsoe, must we wait for you, or do you give up ?

I give up I guess. I couldn't get hold of a decent soundcard this week. Thanks for waiting

[KikeG]

I couldn't get hold of a decent soundcard this week. Thanks for waiting 

I don't think you need a very good card to detect some of the differences found here. I did it with a mediocre one, I'd say it could be more important what speakers or headphones you use.

[Garf]

Bedeox, 14/15 is below 0.1% of guessing, I doubt you did it by chance.

It's 7/15 + 14/15 what makes 21/30, and I don't know how singificant that is.

If you do multiple tries, add them up, or the probabilities become meaningless.

[Moneo]

It's 7/15 + 14/15 what makes 21/30, and I don't know how singificant that is.

About 1,2%.