High Bitrate ~320kbps MP3 vs. Lossless audio, Spoiler: mp3s were TRANSCODED from Lame free-format. Options

[Archimago]

Happy New Year and greetings everyone! I know the folks here at Hydrogen audio have been around and back with regards to these tests... Nonetheless, I've been collecting data from around the audio forums since early Dec 2012.

Since early December 2012, I've opened up a survey to see if music lovers & "audiophiles" around the world can tell the difference between high bitrate ~320kbps MP3 against the original un-lossy-compressed CD audio. I'm only planning to collect data until the end of January 2013.

So far, I have >70 responses from around the world with 5 continents represented. Folks have been using everything from inexpensive (but good) headphones to megabuck separates >$50K for this. There's 2 weeks left so if you've ever been curious about participating in a blind test or just want to add to the dataset, here's your chance :-).

The "test" is relatively simple and consists of 3 musical passages encoded as "Set A" vs. the same songs "Set B"; one of which was MP3 encoded.

Come to my blog to download the music and fill out the survey:
Archimago's Musings

Thanks and have a wonderful 2013!

OOps - the title should have read 'ends Jan 31'... [fixed]

This post has been edited by db1989: Jan 15 2013, 21:16

[pdq]

Just out of curiosity, which forums have you been posting to, and what kind of responses are you getting?

[krabapple]

Just out of curiosity, which forums have you been posting to, and what kind of responses are you getting?

From his blog one can see that he's posted it to Audio Asylum too. I can well imagine at least one species of response there: "bah, CD audio is lossy too!"

[Destroid]

I've opened up a survey to see if music lovers & "audiophiles" around the world can tell the difference between high bitrate ~320kbps MP3 against the original un-lossy-compressed CD audio.

Meaning no disrespect, but comparing those options are not indicative of a solution. I wonder what is the objective. To clarify:

1. To use such high bit-rates is to achieve transparency/archival.

2. Any user using either of those options is likely dealing with higher space confines.

Putting all cards on the table, I would not entrust to archive with 320kpbs MP3, nor I would put those massive MP3's on my player (i.e. LAME -V3 thru -V5 is "good enough" on my portable). Currently, I am content with AAC ~125kbps without any qualms.

If I misunderstood that this was a listening challenge, then I apologize for my interruption. I probably could not ABX MP3 320kpbs and lossless.

[db1989]

1. To use such high bit-rates is to achieve transparency/archival a warm and fuzzy feeling.

FTFY

I hope Moore’s Law has taken us past the point where people feel a need to connect any lossy configuration to archival. IMO, the middle ground makes no sense: either choose a bitrate that (aside from in possible highly rare cases) is transparent, or use lossless. Space confines, obviously possible, but then don’t waste space on 320 kbps for listening! If someone lacks the space to archive, then that’s a bit of a shame, but there’s no point in settling for anything other than lossless if one wants music in a reliable and future-proof form.

[Destroid]

Ok, thanks for catching my rash assumption.

So, I guess it will be interesting in reading about why a music lover would go after 320kbps, and, whether that audio aficionado could detect a difference to the lossless source.

My best guess why this is issue is of concern: buying music online. Except, I doubt the major vendors would have knowledge of LAME/halb27's cutting-edge contributions.

[Kohlrabi]

Reading the test description one flaw seems to be that you encoded one set with a lossless encoder (let's call it "A"), and the other with a lossy MP3 encoder ("B"). Since you don't enforce the use of a software which ensures the test is being conducted in a double-blind fashion, so that the experimenter himself doesn't know which file is which until the test is finished, the following chain of events might occur:

Someone might analyse the spectra of the files and look for MP3 signatures beforehand, or conduct a successful test, and thus be able to determine correctly that file "B" is the lossy one. He then might publish it, which will affect any listeners who don't use a DBT software.

To me, the only way to ensure a proper test is to ensure that no cheating is possible in the first place, i.e. the test is done by enforcing the use of a DBT software, and ideally the result cannot be tampered with. Similar to how the ABC/HR tests have been done here in the past.

The questionnaire reduces the test to a single question about whether "A" or "B" is the inferior source. If the listener did not perform a full ABX test with a high enough number of trials, but rather a single listening or guess, this is the same as conducting only a single ABX test, where a person who guessed has a 50% chance to guess correctly. How will you be able to exclude guesses from your results, if you only have this single question at the beginning, and not a full test result with a number of trials that reduce your likelihood of the result occurring by chance to below 0.05?

This post has been edited by Kohlrabi: Jan 16 2013, 14:20

[pdq]

There is a further problem. Suppose that a listener is able to reliably differentiate between the two versions, but he prefers the sound of the lossy encoded file. This will then skew the results toward listeners NOT being able to hear a difference.

This post has been edited by pdq: Jan 16 2013, 14:41

[DonP]

My best guess why this is issue is of concern: buying music online. Except, I doubt the major vendors would have knowledge of LAME/halb27's cutting-edge contributions.

Properties in foobar shows all my emusic files as encoded by Lame. Older ones (3.96) just show an average bit rate ~170-190kb. Newer ones (3.98r) are listed as -V0 or V2. I don't know if Lame started explicitly tagging the encode setting?

[2Bdecided]

I'm sure there are examples of 320kbps mp3s being ABXed right here on HA.

I don't think anyone here would claim a common/universal night-and-day difference with all tracks though.

Cheers,
David.

[DonP]

Someone might analyse the spectra of the files and look for MP3 signatures beforehand, or conduct a successful test, and thus be able to determine correctly that file "B" is the lossy one. He then might publish it, which will affect any listeners who don't use a DBT software.

Even if someone guesses wrongly, publishing that guess would introduce bias.

ANother interesting quirk, compared to ABX, is that there's no reference. Even if you can detect a difference (likely to be pretty subtle), you are also tasked with figuring out which one is "better."

[testyou]

I'm going to decline your invitation Archimago.
You don't want to actually post at a reasonable place like HA, you're just trying to get hits on your blog.

[Archimago]

I'm going to decline your invitation Archimago.
You don't want to actually post at a reasonable place like HA, you're just trying to get hits on your blog.

Ain't that a bit cynical?

I know this place is "reasonable" but just the same, I am interested in an honest evaluation for anyone who hasn't already tried a blind test or just wants to be charitable in devoting some time and let me borrow the use of their ears/brain and equipment. Although I love audio gear, I know most of the world doesn't care for the bizarre/cultish/tweaky nature of much of it (frankly neither do I!).

You'll note that my blog has nothing on it other than this. Also, I have no ads and I care not about the # of hits, haven't even signed up for Google Analytics or anything like that...

As of today, there are just >100 responses from around the world! There's about 1.5 weeks left so I would love to get more responses to beef up the stats analysis.

[Archimago]

Someone might analyse the spectra of the files and look for MP3 signatures beforehand, or conduct a successful test, and thus be able to determine correctly that file "B" is the lossy one. He then might publish it, which will affect any listeners who don't use a DBT software.

Even if someone guesses wrongly, publishing that guess would introduce bias.

ANother interesting quirk, compared to ABX, is that there's no reference. Even if you can detect a difference (likely to be pretty subtle), you are also tasked with figuring out which one is "better."

Yup, no doubt there will be criticisms of the methodology. My hope is that there will be enough responses to statistically differentiate any bias towards A or B. Also, I would love to see if there's a significant bias for those who thought it was easy to tell the difference (ie. can the confident audiophile pick out the MP3?). Also, I can look at results vs. cost and presumably quality of the equipment used.

Ideally, an ABX methodology would be great. However practically, this would mean a 30% increase in the download size. Furthermore, given the nature of the test; it would be WAY TOO EASY for anyone to compare X with A / B without listening :-). Impossible that I would trust that people would not "cheat" when it's so easy.

[Archimago]

Reading the test description one flaw seems to be that you encoded one set with a lossless encoder (let's call it "A"), and the other with a lossy MP3 encoder ("B"). Since you don't enforce the use of a software which ensures the test is being conducted in a double-blind fashion, so that the experimenter himself doesn't know which file is which until the test is finished, the following chain of events might occur:

Someone might analyse the spectra of the files and look for MP3 signatures beforehand, or conduct a successful test, and thus be able to determine correctly that file "B" is the lossy one. He then might publish it, which will affect any listeners who don't use a DBT software.

To me, the only way to ensure a proper test is to ensure that no cheating is possible in the first place, i.e. the test is done by enforcing the use of a DBT software, and ideally the result cannot be tampered with. Similar to how the ABC/HR tests have been done here in the past.

The questionnaire reduces the test to a single question about whether "A" or "B" is the inferior source. If the listener did not perform a full ABX test with a high enough number of trials, but rather a single listening or guess, this is the same as conducting only a single ABX test, where a person who guessed has a 50% chance to guess correctly. How will you be able to exclude guesses from your results, if you only have this single question at the beginning, and not a full test result with a number of trials that reduce your likelihood of the result occurring by chance to below 0.05?

Good points. Again, I think such is the limitation of open tests like this over the 'Net.

Why don't you analyze the files and tell me which is the MP3 by PM? I've tried my best to make it not easy to "cheat" the usual way :-)

[Archimago]

FINAL CALL!

Closes next week. So far 124 respondents to the test / survey.

Thanks to all participants.

[Archimago]

Study complete! Total respondents - 151.

Follow on my blog as I post the procedure and analysis in the days ahead. Thanks for all the input.

[C.R.Helmrich]

From your blog:

This utilizes LAME's "free format" to create initially a 400kbps MP3 without the usual lowpass filter in place, then runs the resulting file through the MP3 encoder again but at a lower 350kbps bitrate (again with low-pass turned off) which closer approximates the 320kbps target bitrate for the test. By doing this, even though the resulting MP3 size is slightly larger by 30kbps, the degradation in sound quality by objective measures is in fact approximately the same or slightly worse than if the audio were processed directly through 320kbps but without the tell-tale sign of the strong low-pass filter.

Wait... so:

• You're not comparing lossless to the claimed 320-kbps MP3 (free-format LAME is not MP3, and 400-kbps/350-kbps tandem coding certainly isn't!)
• You don't do an ABX test (like pdq/DonP said; A and B might not be ranked significantly different even though a significant number of listeners heard a difference between A and B).
• People can look at the spectra of A and B before listening, and it's trivial for them to do so. How about HA's method of providing test software which randomly assigns A and B and encrypts the results to prevent tampering with them? You'd still be able to watch the spectra while the samples are being played back, but I claim that far fewer people would know how to do that. Edit: And providing such software, like ABC-HR (latest version here?), would not increase the download size by 30%, as you claimed.
• You tried to make it harder for people to identify MP3, and by doing so you crippled LAME's psychoacoustic model. An RMS error measure averaged over a test item does not tell you if two items (here true 320-MP3 vs. your 350-tandem-coding) sound the same. You need to do that measure in the perceptual (masking) domain and also look at instantaneous worst-cases. And since how to exactly replicate that perceptual domain in software is partly unknown, you shouldn't do this at all, I would say.

Sorry, you idea was great, but I can't see how you'll be able to draw any conclusion from your test which cannot be torn into pieces in a subsequent discussion.

Chris

This post has been edited by C.R.Helmrich: Feb 2 2013, 18:28

[greynol]

Wow, I can't believe I once had this topic in validated news.

Two words for you, Archimago: temporal smearing

You should have studied up on the block-size limitations of mp3 and how that makes it particularly vulnerable to the pitfalls of lossy-lossy transcoding before making such a bone-headed mistake.

Any sort of statistical results are going to determine the audibility of transcoded mp3 against lossless long before they can even begin to apply to high-bitrate mp3 against lossless.

This post has been edited by greynol: Feb 5 2013, 02:22

[Kohlrabi]

I (and others) did question the approach before the test was conducted, too bad not more regulars decide to chime in. Now the damage is done. It's a pity with all the effort that went into organizing this. I mean, this after all shows that high-priced gear "audiophiles" might have worse hearing than the general populace, so this would have been valuable.

This post has been edited by Kohlrabi: Feb 2 2013, 20:42

[greynol]

It's pretty obvious the OP registered here only to solicit participants.

[Archimago]

Hang on guys, you're perhaps misinterpreting the test and jumping to conclusions...

This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

Of course people could look at the spectrum while testing, there's no way I can control that EXCEPT by masking it with turning off the lowpass filter, but I could not do that and have the spectrum "look right" unless I went slightly above 320kbps - hence the double pass so I could get the "error" within the 320kbps level.

Even with the double LAME compression passes, people still thought the MP3 sounded better than lossless! As the report shows, those using >$6000 gear got it even more wrong than those using <$500.

Is it because those owning >$6000 gear were older and had bad hearing? Who knows, but the point was that I wondered if MP3 was really that atrocious compared to lossless as many claim (I know you guys here generally do not believe this). Clearly even running the music through 2 psychoacoustic passes with all the horrors of smearing, and hindering the algorithm further with turning off the lowpass filter, it was undetectable and in fact more respondents PREFERRED it.

An ABX methodology with a test like this would not be practical on a large scale IMO and doomed for multiple reasons.

No, I did not register just to solicit subjects... Been reading here for years in fact, so wanted to have some input from some of the folks here whom I honestly highly respect. If you haven't seen the results I suggest having a gander.

This post has been edited by Archimago: Feb 3 2013, 08:42

[Canar]

An ABX methodology with a test like this would not be practical on a large scale IMO and doomed for multiple reasons.

Been reading here for years in fact, so wanted to have some input from some of the folks here whom I honestly highly respect.

I cannot rationalize the existence of these two claims with respect to each other. "An ABX methodology" is the foundational concept of this forum, for heaven's sake. This is a flagrant breach of TOS8. You're showing one of the most massive disrespects possible.

[Archimago]

An ABX methodology with a test like this would not be practical on a large scale IMO and doomed for multiple reasons.

Been reading here for years in fact, so wanted to have some input from some of the folks here whom I honestly highly respect.

I cannot rationalize the existence of these two claims with respect to each other. "An ABX methodology" is the foundational concept of this forum, for heaven's sake. This is a flagrant breach of TOS8. You're showing one of the most massive disrespects possible.

Really? I'm not criticizing or questioning the ABX methodology, am I? I think the ABC/HR idea earlier looks very interesting and maybe if I had been aware of some software that could have approached it from that angle, it would have been the way to go.

My goal was to have an easily accessible test to reach the "audiophile" crowd (as can be seen from most of the forums solicited and the test introduction) since the idea and disagreement which started it all began there. In order to get input from the cohort, the test needed to be flexible enough so some could burn to CD, others could upload to music servers, etc... Not sit around a computer to flip between A & B. Sure, a proper ABX would be significantly more reliable but looking at the equipment people used to test (remember, many of these guys use stripped down computers, insist on Amarra/JPlay/Decibel, etc.), a large proportion unfortunately would not have accepted this type of methodology.

This post has been edited by Archimago: Feb 3 2013, 08:56

[C.R.Helmrich]

Fair enough, given that you apparently didn't know about ABC-HR and given that the "result says no difference" issue pdq first mentioned did not occur. But the bottom line is: your listeners heard a difference between A and B. Proponents of lossy (but transparent) coding can now argue, "of course they heard a difference, the MP3 encoding was badly done", while opponents might argue, "this test used some enhanced (free-format, not low-passed) 350-kbps MP3 and doesn't tell you anything about whether listeners would have preferred real-life non-tandem-coded MP3". See what I mean?

This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

I think we all understood this perfectly. The problem is that when you have to modify a lossy encoder to hide what it's doing from your test participants, your test is flawed.

Chris

This post has been edited by C.R.Helmrich: Feb 3 2013, 16:11