Training for listening tests, from the 0 Options

[mephiron]

Hi everyone!

Ok so I've had a lot of different portable music players during the last years. I can't got to the ancient stuff, mainly casette players and cd players. Anyways my first better player was Sony MZ-R 70, I pretty much liked it, but the recording part sucked and those days you didnt have any HI-MD things or anything so you had to carry a lot of discs with you. After that I had couple different MP3 players the latest being some Packard Bell 512megs.

Ok, so I was on Canada trip at Calgary around 5months a go and a few months before coming back here to Finland I bought a 4Gb iPod nano black. Without any further explaining I wasnt and probably am not yet any "audiophile" or HI-FI enthusiast. That being said I came to be intrested in using flac with my iPod and Rockbox. After a short while of finding some comparisons btw. lame, vorbis and flac I stumbled here and have been quite obsessed with reading your forums. Well enough with history and onto the business.

So I decided to make a little test for me, I crammed some old cd of mine and tried to think of some good track. Which I like and could maybe be able to make some differentiations in quality. I found some old Dance Trance cd from 1995 and it has 37 tracks (small tracks). There's this track 15 that I really like, so I decided to stick with that. Ok, I'll go through some stats and my test system and off we go.


Test track> The Sphinx: What Hope Have I, it has this incredible woman singer on it at the start and quality and really be differentiated on the mids.

Test System> Logitech Z-2200 computer speakers and Koss "The Plug" modded. (I know it's crappy and I didnt think I could make any real comparisons between file formats.)

Test Formats and quality> I made 6 different encodings from the cd with dbpoweramp;

1) .waw, PCM 44.1kHz 16 bits 2 channels, 1411 CBR
2) .flac, medium quality, ~1025 VBR
3) .ogg, aoTuV b4 SSE2 -q6, ~196 VBR
4) .mp3, lame 3.97b3, 192 CBR
5) .ogg, aoTuV b4 SSE2 -q4, ~125 VBR
6) .mp3, lame 3.97b3, 128 CBR

Test Op> I just slammed all the files to Winamp playlist and started to play the beginning of the track on random. Trying to use .waw and .flac as an anchor and dropp away the file with least quality on my hearing.

Ok, so I wasn't expecting any real results since this is my first time doing anything like this and I didn't even know how to differentiate those tracks. At first I had some Iron Maiden track in .flac and mp3 format and couldn't even differentiate them. Then I made some 56 Kb version of the track and ofz noticed the difference. After that I tried some 128 Kb version and I couldn't make the difference again. For a couple days I was like wth. should I just stick with my mp3's as all of my collection (almost) is already in mp3 format.

Then I made this test and just started to press next track button (repeat all and random) on winamp and only listened to the first 5-10 seconds of the track (has a good start for this). After a while I started to notice the differences. The track has a really high and crisp singing on it and I started noticing differences after the cut of the .. shouts. I noticed that some of the files didn't have any distort after the cut-off and some files were really muddy on the cut-off. So I continued and it almost seemed like I could clearly differentiate the bad from the better ones. (notice I did this test while not watching to my screen)

After a while I decided to break on the one file that I though was of the worst quality and tadaa, it was the 128 Kb. So I removed it from the playlist and continued. One by one I could remove the wirst quality imo from the playlist. Until I only had vorbis -q6, flac and waw left. This was really hard and at first I couldn't tell the difference, although I could all the time make out the absolute best of the files, though I didn't bother to check what it was. Then after some 30 mins of listening I was able to make a choice and it was correct, the ogg drobbed from the playlist. I was really excited I could differentiate the files and only had the lossless ones with me.

So after this I thought that I already have the waw and flac and really didn't know if I could make the difference with them, but all the time I had this feeling that one of them was really a lot better than the other. It didn't have any distortion at all, well I thought it would be the flac since it does have some compression and war is uncompressed. Well this time I played them and tried to find the clearer one and then broke the play-back. It was the flac, I was like; "hmm.. this can't be". I recreated the test couple more times and always I found the flac "better", well I dunno which is better in you guys opinion, but for me the flac sounded better and clearer. Right now I dunno why it is so and maybe the flac compression filters some distortion away from the original stream and waw just leaves it there, that would mean that the waw would in reality be of better quality and would match the original better, but still the vorbis would sound crisper.

Results> In descending order; (better at the top)

2) .flac, medium quality, ~1025 VBR
1) .waw, PCM 44.1kHz 16 bits 2 channels, 1411 CBR
3) .ogg, aoTuV b4 SSE2 -q6, ~196 VBR
4) .mp3, lame 3.97b3, 192 CBR
5) .ogg, aoTuV b4 SSE2 -q4, ~125 VBR
6) .mp3, lame 3.97b3, 128 CBR

So, it pretty much came to resemble the original order in which I thought they would go.

Now I just probably need to continue practising and need to think of someway to differentiate some track in low and mid section too, since this differentiation came from the cut of the highest peak.

- Mephiron

[evereux]

What Is A Blind ABX test?.

[mephiron]

What Is A Blind ABX test?.

Yeah, tried to figure out how that program works and your link really helps. Thank you.

- Mephiron

This post has been edited by mephiron: Sep 2 2006, 10:13

[pest]

I recreated the test couple more times and always I found the flac "better", well I dunno which is better in you guys opinion, but for me the flac sounded better and clearer.

you are probably imaging things or your player-setup is not correctly configured.
lossless files are bit-perfect to the orginal pcm-source.
the word "quality" in conjunction with a lossless audio codec is misleading.

[mephiron]

I recreated the test couple more times and always I found the flac "better", well I dunno which is better in you guys opinion, but for me the flac sounded better and clearer.

you are probably imaging things or your player-setup is not correctly configured.
lossless files are bit-perfect to the orginal pcm-source.
the word "quality" in conjunction with a lossless audio codec is misleading.

Yeah, but still.. taking out flac from the 2 for a few times in a row. Currently I don't have any explanations and I'm going to test it with abx.

[kjoonlee]

Or you could try to calculate the p-value. I bet the p-value will be high for your previous test, making it statistically insignificant.

This post has been edited by kjoonlee: Sep 2 2006, 10:30

[kjoonlee]

The null hypothesis in your case would be that the two files sound the same. The alternative hypothesis is that you can tell the difference. You want to reject the null hypothesis and "prove" your alternative hypothesis.

If the p-value is too high, you can't reject the null hypothesis, because the p-value is the chance that you got your results by chance.

You want your success rate to be high and the p-value to be low.

Choosing FLAC over .wav 4 times in a row is not good, because the p-value is approx. 0.062; a six-percent chance of luck, with only 93.8% certainty is not good enough. 95% or 99% is what you should be going after.

Besides, suppose you got the answer wrong on the fifth try. Now the p-value jumps to 0.188: even worse. You want the number of trials to be big, and you want the p-value to be small.

[mephiron]

The null hypothesis in your case would be that the two files sound the same. The alternative hypothesis is that you can tell the difference. You want to reject the null hypothesis and "prove" your alternative hypothesis.

If the p-value is too high, you can't reject the null hypothesis, because the p-value is the chance that you got your results by chance.

You want your success rate to be high and the p-value to be low.

Choosing FLAC over .wav 4 times in a row is not good, because the p-value is approx. 0.062; a six-percent chance of luck, with only 93.8% certainty is not good enough. 95% or 99% is what you should be going after.

Besides, suppose you got the answer wrong on the fifth try. Now the p-value jumps to 0.188: even worse. You want the number of trials to be big, and you want the p-value to be small.

I'm sure to have more trials soon.

[kwanbis]

3) .ogg, aoTuV b4 SSE2 -q6, ~196 VBR
5) .ogg, aoTuV b4 SSE2 -q4, ~125 VBR

You should try the aoTuV Release 1.

4) .mp3, lame 3.97b3, 192 CBR
6) .mp3, lame 3.97b3, 128 CBR

Tried VBR? -V5 and -V2 should be equivalent to 128 and 192 CBR

Now I just probably need to continue practising and need to think of someway to differentiate some track in low and mid section too, since this differentiation came from the cut of the highest peak.

Unless you have a special need, i think that training to find problem samples is a bad thing. It would take the fun out of normal listening.

[Canar]

Being completely untrained can be a bit of an embarrassment though. I have difficulty abxing 48kbps Nero AAC sometimes. (!!)

[stephanV]

That's not embarrassing. It's a blessing.