Hi All,

I've been corrected before on this site for making the statement: "I can tell the difference between an MP3 and lossless". I've never ABX'ed myself but I was wondering if there has been a statistically significant (with regard to population) ABX study done with (say) Lame insane vs. Lossless?

Clearly for myself it makes more sense to do a personal ABX but I was wondering in general about the population.

I searched manually through the posts in this forum (since search seems useless for the term 'mp3') and didn't see anything.

Thanks for help,

ss.

You couldn't do an accurate statistically significant test with average users because they'd all be using different (and much of the time lousy) equipment.

try searching for lossless lossy abx

And that is a perfectly accurate reflection of the real world. If your statement was true, any of the group blind listening tests conducted here would be invalid.

No, you're not listening to what he is saying. He wants to test the limits of people's hearing, not their equipment. If you did a blind test with $50 headphones and onboard sound, and the people taking the test were unable to determine a difference, does that mean they cannot detect the difference between mp3 and lossless? Not necessarily, it means they cannot on that equipment.

Group tests conducted here are different, they are set up to determine whether people can tell the difference on their equipment because that's what they will be listening on. They do not determine whether the person is capable of percieving a difference.

"lossy vs. lossless ABX" doesn't make any sense. You simply ABX lossy codecs against the original, be it a wav file or a losslessly encoded file. The term "lossless" here is irrelevant. ABX tests of various lossy codecs at various settings have been made several times, search the forum (just don't search "lossy vs. lossless" specifically, because of the reason I just gave).

Come on. I think every single person who read my post understood what I intended here. You are arguing semantics instead of giving information.



As noted in my original post, this doesn't work because the (in my opinion insanely dumb) search restrictions on the board prevent searching for any term shorter than 4 characters.



I guess the study that I was looking for would use standardized equipment for all participants thus eliminating this factor.

Thanks for all the information... but I still haven't seen a link to an actual study.

If indeed there is no study like this, I'm a little shocked. With all the ABX tests that have been done before (on this board and other sites) why has no one used a lossless (ok ok "orginal") sample as reference? Maybe I'm missing something here... I'm by no means a statistical or testing expert.

ss.

Why make it complicated:

Comparing lossy VS lossless is common practive when ABXing. This is because a lossy sample is usually compared against the original(which is lossless). So, what you are asking about is not an exception but just normal standard practice in almost every ABX-test.

Concerning numbers: taking high-bitrate ABXes on only this forum into account, they would total in hundreds if not come near thousand.

So yes, statistical evidence(for non-transparency in everyday-listening) after dozens of tests is quite low.

Of course, it is always possible to find single fragments of music, where a difference can be heard. However, most people dont care if of 10 full CDs there may be 10seconds where a difference can be noticed when listening to the 10sec sample in a loop a dozen times! The background noisefloor in your room in that case is a much bigger hazard.

So, the evidence currently is low enough that one could ask "Is the background-noise in your room(a room is almost never absolutely silent) a problem to you? If no, then properly encoded high-bitrate lossy audio should be transparent to you during everyday-listening."

- Lyx

Edit: notice that you are asking the question the wrong way around: It is not possible to prove that something doesn't exist with absolute certainity. It is only possible to prove that something DOES exist. So, the question should not be if there are enough negative ABXes, but instead if there are enough positive ABXes under conditions which can be applied to normal listening. Thus, you cannot ask others to prove non-existence - the burden is on the one who claims that there IS a difference.

I've done the search I referred to you, and found several topics compating lame preset insane to wav. I suggest you do the same.

Why do you not post links to the studies you found? Saying that you found them does not benifit anyone. I have done similar searches are found individual results but no -studies- as I asked for at the beginning of this thread.


Thank you for the explanation. I agree with you completely. My quote from the original post was merely a summary of a viewpoint I expressed in another post not really the question I was trying to answer. I agree that you can not prove non-existence.

I guess I was hoping that there was some summarized data along these lines instead of having to go through a bunch of individual ABXs in order to determine how significant the differences were between the lossy and the lossless/original sample(s).


This seems to be the case and indeed it is noted as a key part of the ABX by pab in his introduction to ABXing. However, all the studies I've found (including the extremely well constructed studies by rjamorim), these never show the results for the original sample in their results. The results always compare different formats but never give us a control to compare against... or perhaps I am reading the results incorrectly...?

ss.

I believe that all of the listening tests compare the lossy file to the lossless original. It is then given a rating on how well it did.

Then, the ratings of each lossy codec are compared to each other so as to see which one most accurately reproduced the lossless file.

Just to be sure that everyone understands...

Comparing the lossy file to a lossless file is no different than comparing the lossy file to the actual CD.

Basically, any codec that is transparent versus the original sample, will be un-abx-able. The results will be regarded as unsignificant and discarded, because most of the time, abx results are to compare version evolutions, or different encoders.

To summarize:

a) A "lossless to high-bitrate MP3 ABX" test = a "high-bitrate MP3 ABX" test. No need to specify lossless, all ABX tests are done against a lossless source by definition. A test comparing two codecs (again, against a lossless reference) would be an ABC test.

b) Even the 128kbps tests usually come up with statistically insignificant results because the public just isn't able to distinguish. So the only way you'll ever get interesting results at high-bitrates is to either do it yourself, or read the results of someone who did. Guruboolez is about the best person here for this sort of thing... he can tell the difference on some problem samples, but not must samples.

No
mp3 ABX test does mean ABX vs original source (or lossless, it's the same), but nothing prevents you from ABXing mp3 vs mpc, mp3@128 vs mp3@256 and so on...

True.... Even in this forum, which probably is the place where most motivated and skilled people can be found, 128> listening tests don't involve many people, because all codecs are nowadays excellent (I didn't say perfect, or CD quality...) at any bitrate >= 128...

Thats certainly true for recent times(i think the fact that most lossy codecs are "too good" is also a factor which seems demotivating to even try ABXes for many people. Multiple people mentioned that they got the feeling that "there isn't much going on in the lossy department anymore, because it is already so efficient"). However, i've heard so often that the tests on lame 3.90.3 were massive in numbers - did those also include many high-bitrate tests? I dont know, because i wasn't there, but i would guess yes, considering the reputation of preset standard? What about the pre 1.0 tests on vorbis? Were there many high-bitrate tests? And MPC? I think it would be interesting to get some info from long-time board-members, because i have the feeling the time where most of the majority of tests were done is a few years back in time. And during this "peak-time" codecs just became so good that ABXing them at high bitrates became very difficult? Insight from someone more knowledgeable would be greatly appreciated.

- Lyx

Okay, fair enough - around here, the ACCEPTIBLE process is to ABX vs the original, since ABXing two different lossy codecs doesn't tell you much of anything, just which one sounds better to you (which could be the OPPOSITE of which one is closer to the source. Some people like the sound of certain codecs).

Related to the first posts...
Without the intention to violate the AUP 8 with this assesment, I've heard artifacts using the onboard audio and onboard speakers of my laptop that were not audible using better equipment.

The same applies for the car, sometimes on the puter I can't hear some artifacts but on the car it is a different story, and some artifacts I hear on the car I can not perceive on the puter (I'm talking about the same file, but just decoded to CD-audio for play in the car).

I believe that the explanation behind this situation is that different equipment will create different masks, poor quality equipment might not have the fidelity needed at certain frequencies making a worse mask than that obtained with hi-fi equipment. Likewise, poor quality equipment might produce a higher mask within certain frequencies making some artifacts inaudible.

Just a thought thou.

No this is not correct... ABX tells you *nothing* about your personal preference. It only tells you can hear a difference between the two files you are ABXing. While you can ABX two lossy files against one another, after doing this you can only show people that you can hear a difference between the two files, not which one you prefer.

[edit] removed some unclarities

I'm new, hi everyone, i want to make a suggestion on this subject.

This is my slant:

In a few years ill be buying either an sacd or dvd-a player depending on the climate, which of course will be a wireless device and send the digital data straight to the amplifier, which will also be wifi connected to my media storage device, containing my music collection, very likely with storage in the realm of terabytes.

When i play my SACDs, and then flip to my mp3 encoded tracks, they are going to sound terrible. Even cds will begin to sound a little lifeless. I'm putting all my CDs on my HD with Flac, and thats that - it makes sense for the future, sooner or later we'll all have media devices instead of dvd players.

Don't spoil your expensive systems you'll buy in the future with a poor substitute for lossless when hard drives are getting so cheap and so big.

Soonish, I'm going to mirror raid myself up a terabyte archive beast (current costs = 8*250 gig HDs - Im waiting till i can run 4*500 gig hds without breaking the bank) and once ive got that there is no excuse for worrying about the space my music collection takes up. Some of you may have enormous collections, which causes a problem, but i still like the idea of wifi music throughout my house at CDDA quality with no cds in sight!!

This applies in this thread because I think these days we should be trying move away from lossy over processed codecs, to take advantage of this cheap storage with the highest quality media. What you pump out to portable devices could very easily be done in realtime, and set to encode different bitrates dependant on the device. So your ipod, your car and your house can sync up the audio library, but in whatever bitrates is best for that device. Maybe, 320kbit in the car, 192 on the ipod, reflecting available storage on media devices.

When this sort of system becomes available, you'll want a good quality source.

Just my perspective, in respect of yours

yes yes, but what is the point of that? Okay, MP3 "a" sounds different than MPC "b". This means.... nothing.

So why would such a test be performed in the first place? Because the tester wants to prove that one of the two tracks sounds better. But of course, as you said, this just proves that they are DIFFERENT, not which is closer to the original. Tester however concludes that this difference supports their preference. A preference they are entitled, to, I might add, but it doesn't prove much else.

Give me another example of why you'd ABX two lossy copies of a file. I can't think of one.

*sigh*. No one has yet been able to prove they can hear a difference between a PROPERLY MASTERED CD (that's a hard thing to come by, unfortunately) and a DVD-A or SACD disc. If you can, please do so. If not, don't spout such misinformation. TOS #8 violation and whatnot.

I'm really getting sick of saying this. I think this is like the 5th time in 2 days.

God bless placebo - because otherwise, imagine the pain of paying for all that expensive equipment, and then noticing that it doesn't sound any better than normal audio-cds :-) (except of maybe multichannel tracks).

Did this post mean, that i fed a troll? :)
- Lyx

Hmm, there's a bit of confusion in this thread.

All of the MPEG verification tests were made against the original, which is the same thing to compare to as a bitwise-exact coding method.

All of the MPEG verification tests I'm aware of, which would include MPEG-1, MPEG-2 BC, MPEG-2 NBC, various and sundry MPEG-4 verification tests, have shown that at least some listeners appear to distinguish between the original and the coded at the test rate, even for rates like 256 and 192 for layers 1 and 2, and for 192 as Layer 3, if I recall correctly.

The CRC test for the american DAR project, the one done a long time ago when the audio quality was of the 160kb/s variety, also showed that 160kb/s PAC and some higher rate of (I think it was AC3, but I frankly don't remember) could be distinguished from the original.

So, I think it's confirmed via some very sophisticated tests, done by world experts, that the usual rate of most codecs is not "transparent" in the ABX sense, although MPEG has used a standard wherein anything above the 4.0 CCIR scale mark is considered 'acceptable'.

Well, not only the bitrate average matters, but also the encoder and method. Is Fhg CBR competitive against LAME VBR at high-bitrates? How high was the amount of positive ABX-tests compared to total trials? What kind of samples were used?

The most sophisticated tests done by world-experts can only tell something about "what" they tested. If the tested encoder/settings were inferior compared to high-bitrate LAME VBR, then the tests only concluded just that: That with an inferior encoder some people were able to tell the difference.

- Lyx

Well, in those tests, the encoders were built by the algorithm proponents, with everything on the line, based primarily on encoded quality.

I don't think it's possible to get evidence at this point to show exactly what was tested, but the people doing the encoding were the inventors of the coding methods, and the ones with the most to lose if the test results weren't good. The encoders were generally not restricted to real time, not restricted to simple, and were supposed to show the "ultimate" quality.

That's all I can offer you.

None the less, I don't think there is a 128kb/s MP3 encoder that meets the "ABX transparent" standard, or at least I haven't meant one, even in tests where all of the degradations were reported as being over 4.0 on the scale, the ID for the 'reference' signal in an ABC/hr test wasn't even close to any reasonable random hypothesis. The MPEG test results, at least, as well as the CRC results that are on much, much newer encoders, have been published. I don't know the citations offhand, but the results were not ambiguous in any real fashion.

Well, that 128kbit MP3 (especially CBR) is not transparent to everyone is nothing new. I doubt anyone here would argue on that(although some people would probably point out, that LAME VBR around 110-140kbit is very difficult to most people, but certainly not to almost everyone).
Newer doesn't automatically mean better. If i remember correctly, then Fhg is mostly tuned for low-mid bitrates, and its VBR-implementation is not on par with current LAME). The situation at low-bitrates is the opposite way around: There, fhg is more optimized and has intensity-stereo (which lame lacks). So, simplified one could say that below 112kbit fhg tends to be better, while above LAME usually is better.

And, i highly doubt that they did test something which "ain't an mp3 encoder" ;-) So, i suppose they did test fhg. So, the problem is that we have lots of listening tests on this board with encoder A, and listening tests from the MPEG-folks on encoder B. But both can only be compared against each other, not merged.

Thanks for the infos,
- Lyx

Woodinville,

I'm not quite sure what you're trying to say about the official tests.

It's true that mp2 and mp3 "failed" at the target bitrates.

However, AAC Main Profile at 128kbps did rather well (if not a 100% perfect pass) here:

http://www.mp3-tech.org/programmer/docs/w2006.zip

I haven't seen any such test for "high bitrate" mp3, though we know some specific samples where Lame mp3 has difficulty, while commonly high bitrate Lame passes ABX tests for most samples - though some listeners find problems on certain types of sound.

Cheers,
David.

The AAC Main Profile, with what the developers of the codec believed was the best available encoder, still provided a non-random (to many orders of magnitude, I've forgotten how many) ID rate in reference identification in the final AAC verification test, which I suspect is the best test run with AAC.

The impairments that were listed were quite small, but many (not all) listeners were easily able to ID the reference signal, which shows that there is "indistinguishable" according to MPEG (all items over 4.0), and "transparent" according to ABX id rate, and that while MPEG's definition is very good, it's not perfect.

In that test, some listeners were excluded for having negative impairment scores, this was used as a screening method. None the less, there were far, far too many people who ID'ed beyond the significance required by the number of trials and subjects, especially on the most difficult tracks.

I think this says it all, quoting the report:



Note that this is a weaker test than asking what the ID rate for the reference is, as well.

A few things to note about Roberto's tests:

1. People who pull the hidden reference slider down on any codec are typically excluded from the analysis (although I think if there are only a few listeners, he may decide to rely on ABX results, if there are any, to make a final decision about whether or not to exclude a listener's results). This is a much tougher standard than just excluding somebody who has a negative impairment score. However, it does mean that the results will always be less than 5.0 for any codec. Remember that Roberto isn't trying to find out whether or not a codec is statistically distinguishable from the reference, only what the relative ratings are of the codecs under test.

2. Although there may be a few "problem" samples mixed in with Roberto's tests, the prevailing idea has been to be representative of "real" music. I don't know if the MPEG tests are tilted towards problem samples or not.

3. The group of people who participate in Internet tests are probably skewed towards more discriminating listeners.

All this is saying, which should be rather obvious, is that Roberto's tests don't necessarily compare with the MPEG tests, although I think they certainly serve their purpose. But that purpose does not include determining if a codec is statistically the same as the reference.

ff123

For goodness sake read my post first. I believe you just violated one of your own TOS.

I was saying that SACDs will make an MP3 collection sound bad, as opposed to say - an uncompressed CDDA library. If I was saying what you suggested then it wouldnt belong in this thread now would it? Also I'm sure SACDs will make reissues from the master vinyls of old records sound a lot better than a cd equivalent reissue, since it will support the full frequency ranges of audibility, and also has the dynamic range to cope without any processing whatsoever. Im sure you're all aware that they have to compress the audio samples to fit the dynamic range of AudioCDs, since you're all so very knowledgable.

So far youve totally misunderstood a perfectly clear post, been completely arsey and called me a troll, and accused me of spreading misinformation. Its a great welcome to the forum. You know what you can do with your TOS. Cya

My emphasis on "will" and "sure".

ToS#8
"Hydrogenaudio is supposed to be an objectively minded community that relies on double-blind testing and relevant methods of comparison in discussion about sound quality..."

That is the point I believe he was trying to get across. You cannot say without proof that SACD will make an mp3 file sound bad, unless you've done testing or cite testing that proves this. This has been discussed over and over on HA. If you aren't going to go about comparisons based on scientific merit, and instead focus on purported capabilities then you are in violation of ToS#8.

Maybe I misunderstood too, but I found your post to be BS.


It doesn't matter. You post confuses a lot of us anyways.


How "sure" are you?


I don't think your post is "perfectly clear", as this:


Prove it with ABX results.

Okay, my post was clear, it was just that you got confused because you didnt read much of it at all. Then your buddy just started dickriding you. That was the only confusion.

One small problem with ABXing that stuff:
1) I have no SACDs
2) I cant be sure any SACD sample I get hold of will be well produced and mastered, taking full advantage of the frequency response, from a master press, which of course would have to be analogue recorded, mixed and mastered all the way. Supply me with this and I'll abx it.


But again youve missed my point.

I was comparing the change between a well mastered SACD, and a 192kb MP3. I dont need to ABX it to know that - in my car I can tell EVERY cd that has been burnt from MP3s by ear, although I admit it gets harder with 320kbps ones. If I put an original CD in there the difference is phenomenal between it and 192kb mp3s. Are you really suggesting that I should waste hours of my time abxing it when I - and everyone else with half a brain on this site knows what the result is going to be?

I think you're just being awkward, perhaps my post was a little out of place but it certainly wasnt BS, you misread it, and what you misread it to be could have been interpreted as BS.

The bottom line is: my statement was based on the fact that 128-192kb mp3s, dont sound as good as CDDA, or SACD. IF you think that is BS without ABX testing then I'm glad you have that much spare time to waste.

I dont. Good day.

Edit:
One addition I should mention is that the main issue i have with mp3s, is listener fatigue, especially at high volums.

ABX testing cannot possibly give any decent results which indicate anything about fatiguing. ABX is not the one ring to rule them all, and it does not sort everything from everything else. These are textured sound waves, you cant use a binary "is it better" system like ABX to say if it sounds GOOD. I think you are all in a world of your own. Try abxing some heavy sections of NIN- The Fragile @ 320kb and @ CDDA. MP3 encoding falls apart with kind of guitar texture.

An artifact is an artifact. If it exists, and your ear can hear it, then you can hear it. It doesn't matter about mastering.


You have already violated TOS #8.


Yes, because it has been proven before that can simply be placebo.


I really hope so, but it seems it is not.


Again, prove it. Don't you know that you just showed me you dare not do so?


I believe I need to call moderator now. You are BS and troll.

In case you don't browse the forum using the sections, you can find some ABX tests in the "listening test" forum : http://www.hydrogenaudio.org/forums/index.php?showforum=40

Here are some old ABX tests at high bitrate :

http://www.hydrogenaudio.org/forums/index....wtopic=7783&hl=
http://www.hydrogenaudio.org/forums/index....howtopic=12031&
http://www.hydrogenaudio.org/forums/index....topic=12534&hl=

I've been here since september 2001, and never saw a study about lossy compression at high bitrate, exept Guruboolez' one, below in this forum section.

128 kbps tests run by Roberto Amorim and Sebastan Mares are the hardest that a large number of people can undergo. For higher bitrates, only some poeple can perform some ABX tests with some samples.

I've heard many times about a professionnal german study that tested MP3 at 256 kbps, before 2001, and that concluded that it was transparent to the listeners, but it seems that the paper is not on the Internet anymore. I've read that they listened with Senheiser Orpheus headphones.
However, many problem samples are known, see the above links.

The alt-presets for Lame were developed around 2001. As far as I remember, no ABX tests were published during the development. I think that the dm-presets have undergo some ABX tests, but it was in a forum that has now been taken offline (r3mix.net)

The article you reference is here:

http://www.heise.de/ct/00/06/092/default.shtml

ff123