results of lossless vs lossy listening test on reddit Options

[krabapple]

results out today

http://www.reddit.com/r/audiophile/comment...ssless_results/

[db1989]

So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.

I don’t like this.

I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that there’s a good amount of scientific discussion and scepticism there.

I’m sure there are lots of delightful comments there if I were to read far enough, though. I don’t really do Reddit. And I try to avoid comment sections in general.

Anyway, I’ll wait with interest to see what other users here think about this.

[Kohlrabi]

So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.

I don’t like this.

3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?

This post has been edited by Kohlrabi: Feb 22 2013, 20:15

[krabapple]

So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.

I don’t like this.

I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that there’s a good amount of scientific discussion and scepticism there.

I’m sure there are lots of delightful comments there if I were to read far enough, though. I don’t really do Reddit. And I try to avoid comment sections in general.

Anyway, I’ll wait with interest to see what other users here think about this.

There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.

edit: btw, it occurs to me that maybe this should have bene in the Listening Tests subforum...sorry, mods, please move as needed.

This post has been edited by krabapple: Feb 22 2013, 20:19

[db1989]

There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.

Sure, and that’s what I meant:

I had a sinking feeling […] but I was pleasantly surprised […] I’m sure there are lots of delightful comments there if I were to read far enough, though.

I was referring to the same uppermost comments as you and my reluctance to read much further down.

[Arnold B. Kruege...]

results out today

http://www.reddit.com/r/audiophile/comment...ssless_results/

In as few words as possible - how were the evaluations done? I looked at the site and nothing about that popped out at me.

[mjb2006]

In as few words as possible - how were the evaluations done? I looked at the site and nothing about that popped out at me.

See http://www.reddit.com/r/audiophile/comment...st_details_and/

[Garf]

3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?

No, the test didn't show that. If you squash all listeners together, and hence pretend only a single person took the test, then yes. You can also squash the samples together, and then say FLAC won only 1 test That's not the right way to analyze this.

If we instead consider there are 62 listeners, times 5 samples, each of which did a block of 4 codecs, then we get this:

CODE

./bootstrap.py --compare-all --blocked reddit2.csv
bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto <gcp@sjeng.org>
License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html>

Reading from: reddit2.csv
Read 4 treatments, 310 samples => 6 comparisons
Means:
MP3_320      FLAC   AAC_192   MP3_128
   2.406     2.165     2.694     2.761

Unadjusted p-values:
          FLAC      AAC_192   MP3_128  
MP3_320   0.013*    0.005*    0.001*  
FLAC      -         0.000*    0.000*  
AAC_192   -         -         0.520    

FLAC is worse than MP3_320 (p=0.013)
AAC_192 is better than MP3_320 (p=0.005)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.001)
MP3_128 is better than FLAC (p=0.000)

p-values adjusted for multiple comparison:
          FLAC      AAC_192   MP3_128  
MP3_320   0.028*    0.014*    0.004*  
FLAC      -         0.000*    0.000*  
AAC_192   -         -         0.521    

FLAC is worse than MP3_320 (p=0.028)
AAC_192 is better than MP3_320 (p=0.014)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.004)
MP3_128 is better than FLAC (p=0.000)

Note that better really means worse here - Reddit had good=1 and bad=4. The test result is basically:

FLAC > MP3_320 > AAC_192 = MP3_128

I'm surprised that they're apparently able to tell 320kbps MP3 from a FLAC, but fail to tell an 192kbps AAC from an 128kbps MP3. (Unless 192kbps iTunes has the same lowpass as 128kbps LAME, of course )

[Garf]

Three of these sections were then re-encoded from that lossless WAV to MP3-320 Joint Stereo, AAC-192 and MP3-128 Non-joint Stereo (via Goldwave) while the last section remains lossless.

What encoders goes Goldwave use? Also why force full stereo on an 128kbps MP3 encode, that does not make sense.

[LithosZA]

I hope they didn't use something like FAAC for AAC encoding...

Also why force full stereo on an 128kbps MP3 encode, that does not make sense.

I agree, that doesn't make sense at all.

[Garf]

I hope they didn't use something like FAAC for AAC encoding...

Seems to be QuickTime (which should be fine?) and some version of LAME (potentially an alpha).