I know most of you are audiophiles, but does anyone think it's worthwhile doing some testing for people who don't have super equipment or wonderful hearing?

I don't know if this has already been taken into account, but maybe things like -alt-preset standard are overkill for those with "average" setups - the majority of people. The results might be a useful thing to implement in recommended settings for general users.

Your thoughts?

How do you decide whether a given person is an "average listener"?

You seem to imply that this would be people that don't have "super equipment" or "wonderful hearing," but what exactly do these things mean?

What is defined as "super equipment"? (It may be worth noting right here in fact that the majority of users on HA, AFAIK, do not have "super equipment," at least in comparison to the equipment that often shows up in "audiophile" magazines in the in price ranges of thousands to tens of thousands of dollars -- regardless of whether that equipment is actually of high performance or not.)

And how do you define "wonderful hearing"? This is completely subjective, because you need a baseline to establish a comparison against. So you arbitrarily choose either 1) some specific listener that you measure others against, and assume that if they can hear what he/she can, or more, that they are "wonderful listeners," or 2) you arbitrarily pick some sort of characteristics from a signal that you feel (for whatever reason) that a listener must be able to discern to be considered part of the "wonderful listener" group.

So, in the end, it's almost completely arbitrary. This makes such an effort, IMO, pretty useless. But for a little bit of history, this is just about exactly what Roel tried with the --r3mix switches: he aimed for "average listener" performance, but the big problem with that was that "average listener" was defined solely by his own whims. As it turned out, many so-called "average listeners" found that his assumptions were wrong when it actually came time for people to do the listening tests.

Really, it makes much more sense to aim for either "transparency" or "least offensive," both quite simply defined according to a consensus among the largest group of listeners that you can manage to assemble for a given test. Of course this may not be fully representative for certain groups of listeners (ALL listeners for example, or "average listeners" even), but it's about the best you can do.

I think it would be useful

I should have guessed it would have too many counfounders.

Nevermind then.

Not everyone has "super equipment". My setup (in my sig) cost about $750 AUD ($575 USD).

The equipment is largely irrelevant in determining what codec flaws you can hear.

At least much more irrelevant than most people think.

The same is true for "wonderful hearing". It has more to do with training and experience. That's why your idea (which, as Dibrom explained, has been tried before) is harmful in the long run.

Some time ago and just out of curiosity, I tried to recognize the flaws in some MP3 testsamples (from listening tests) using cheap DAP's. Most of them were clearly noticable. My conclusion was: Even cheap Players need well encoded music.

I think blue57's idea has some merit, it's just his reasoning that is flawed.

A better explanation might be that it has become very difficult to spot encoder artifacts these days without training yourself to hear them. I know that I can't ABX Lame at anything over 130-140 kbps, even on many of the difficult samples. Maybe I have worse hearing than average, but I think that people without artifact training wouldn't do much better.

Personally I think that --preset medium doesn't get nearly enough love here on HA. It is what I use for my DAP, for encodes I do to send to friends, and other such. (On my main computer I use musepack.) It may fail more often than standard, but in a large majority of samples for a large majority of people it will be transparent.

In some ways, this is a fantastic compliment to the devs of Lame, Vorbis, and AAC. It helps that people are willing to spend more kbps these days than they were years ago, but great improvements have been made even for the peolpe who don't know about anything other than 128.

Usually, detecting artifacts is mostly a matter of "training" - equipment and hearing-abilities are secondary.

Many large-scale tests which have been done(especially roberto's listening tests) were done around 128kbit, so if that is what you're looking for, it does already exist. Even more so, since medium-bitrate tests are more common because they are easier to do.

If what you ask about is using "normal samples" instead of "killer samples", then that may be difficult to do. The problem is that all modern lossy codecs are already so good that they are indistinguishable to the original most of the time(i.e. with "normal samples"). So, current listening tests and tunings are mostly about "problem cases" - not about how an encoder performs most of the time.

I was meaning the people with >$50 setups.. mobo with integrated sound, cheapo speakers, $15 headphones (like myself), and have music on for enjoyment or as a sonic decoration, rather than those who focus on the audio.

Anyway, don't worry about it now. -aps may be overkill for me right now, but at least there's some room for growth in the future.

I dont understand - why would someone with such a setup care about "minor" differences in mp3-encoding? Its like going cheap with 90% of what is significant in audioquality, and then being picky with the remaining 10%.

- Lyx

Esentially, to save some disk space.

If you choose the right pair of $20 headphones you can get plenty enough quality to appreciete better encodes. Integrated sound mobos are more of a problem, but there are plenty of cheap but useable sound cards out there. In fact, for $50 you could have the above headphones and a Chaintech AV710. Plus sound cards can easily be recycled from older computers.

As for only using music for sonic decoration... The problem with a bad encode is that once you notice it, it can be difficult to ignore afterwards.

First of all, don't feel pressured to do anything you don't want to do. Nobody here will disrespect you if you use preset medium instead of standard. If file size is important to you because you have limited space, that should be part of your decision.

Second, I think insurance is a big thing with the HA group. People use EAC not because it is better all the time, but because it is better in the worst case scenerio. They use lossless not because they can tell the difference from a hq lossy, but because it provides a perfect backup in case something happens to their cd collection. Many here have invested large amounts of money and time in their music collections, so putting extra effort into the computer version thereof is only natural.

You hit the nail on the head right there. Room for growth. Maybe you have what some would consider 'low end' equipment, but say you get a few extra dimes and upugrade your soundcard and speakers? You would want the quality of your encodes to hold up without having to re-encode everyting.

What's wrong w/ someone trying to get the best sounding audio out of his/her 'tin cans' all the while saving some space on the h/d?
I think that's what most, if not all, people are trying to achieve by extensively testing out LAME or any other encoders.
It doesn't matter whether it's cheap or expensive setups; it's what compressed audio's all about.

I'm just seeing it from a different perspective.
If I put myself in a position where I can't spare a dime for audio h/w & h/d space, I know I'll try to get the best of both worlds.
Getting it as small as possible, all the while sounding not too bad on the speakers.

To quote it right off HA's slogan :

the audio technology enthusiast's resource

If blue57 is enthusiasted about the 'best' sounding compression level on a 'tin cans' setup, let's just help him.
Or at least show him on how he could help the others, and even himself, out.
Who knows, he might produce his own blind test and become something of a guruboolez calibre at 'tin cans' setup.

I don't think the issues with such a test are too insurmountable. As others have mentioned, all that is really necessary is for untrained listeners to do the tests. Along with this we might want to consider relaxing the ABX protocol a bit, because the focus here is on casual listening.

128kbps encoder tests are somewhere around what is requested here, but really, the quality of 128k encodes nowadays is extraordinarily high, and we're not really sure what it means for the average joe. I would not be surprised if lame -V7 turns out to be transparent to some people.

You are scaring me.

Come on! All I'm saying is that if you take a random person off the street and throw them in an ABX test on halfway decent hardware, I would expect them to hear a lot more things than if they just listened to the music casually, even if the test was relatively short. And that's exactly what we're testing, casual listening. In that context an ABX test is testing the wrong thing.

Now, I admit to having absolutely no idea how such a test would operate and still have any power whatsoever, but it's important enough not to discount out of hand.

So here we go again...

What is an "untrained listener"? Is this someone who hasn't spent much time listening to lossy audio codecs? Or is this someone who doesn't listen to music much in general? Or is this someone who ... ?

I can think of quite a few people who, in other contexts, people would consider "trained" (musicians for example, or stereotypical "audiophiles"), but where such a label doesn't necessarily translate well into this domain.

The problem with that right away is that as soon as you label such a group and use them to make a statement regarding quality derived from the results of their listening test, you're going to have some group complaining about the representativeness of it all. Ultimately, it'd take a lot of effort to make the arbitrary distinctions necessary to setup the test, and you'd only end up with questionable results. It's simply not worthwhile.



Why?

Again, you have the same sort of problem. One person's "casual listening" is going to be completely different from another's. What you need to do to get a representative result is control this situation as much as possible, and that's what ABX provides for us.

But let's say that somehow, even given these problems, you find a correct way to carry out such a test and to rely on such relaxed conditions. Why would any developer in their right mind want to waste tuning for such results? It's a waste of effort because, in the end, people are going to ultimately compare their efforts based on some sort of benchmark, and "casual listening" is hardly a good metric to use for that. And furthermore, the results themselves are questionable because they lack a level of objectivity that is necessary to really nail down problems in quality and fix them. If you relax ABX and tune for the "average listener," you'll spend all your time chasing phantoms.

No, the way to do it is the same way I listed earlier. Then, if you need lower quality, you provide some sort of smooth quality scaling. Most codecs do that these days anyway (LAME with -V, Vorbis and MPC with --q). From here, a particular individual can determine what meets their own "casual" needs through a few simple listening tests.



And how do you know this?

Yes, that is the wrong thing in such a scenario. But so is trying to use such an uncontrolled situation to make a general conclusion, or to use the results of such a situation to perform the type of precision work that is involved with codec tuning.



I don't think anyone is discounting it out of hand. It has been explained why it's not a very workable idea, and furthermore, why it's a bad thing from a quality tuning point of view. This is an issue that has been discussed in a lot of different contexts in HA's early history.

An untrained listener is exactly what the adjective implies: Somebody with no experience in detecting encoder artifacts. While I can't offer any evidence that such a listener is particularly fungible (ie musicians and regular listeners and audiophiles would make equally fine untrained listeners), I see no evidence to the contrary either. If such a listener is fungible, then I would argue that the power of a test involving them is going to be OK - not good, not great, but acceptable for the target audience.

In other words, your point about representativeness is cogent, but to the best of my knowledge, not actually validated. In this respect this situation only differs from transparent encoding tests by degrees. Before HA and before ff123 and before the whole listening test era, was it obvious that there is a well-defined boundary of transparency for properly designed encoders? (Actually that isn't a very rhetorical question as I don't know the answer; if it was obvious, then this comparison isn't that valid.)



I agree that tuning for this sort of thing is mostly useless. ie, when you tune an encoder, it would be far more effective to tune based on transparency and based on 1-5 ranking results with trained ears rather than based on casual listening. For transparency, there is a well-defined and psychoacoustically sound boundary to tune for. For casual listening there isn't, and you'd have some people who can't hear anything and some who are just naturals at telling differences, which makes drawing that sort of a boundary impossible.

If I were to ad-hoc this a bit further, I would argue that this could be worked around by making the result for such a test statistically determined from the distribution of listeners rather than the ABX results themselves. That is, given 20 or so listeners, the final "tuning" is going to be the one that yields "casual listening transparency", whatever that is, for, say, 70% of the listeners (I pulled that number out of a hat).



In all reality these are probably sufficient for most people. The only realistic thing that a casual listening tuning would turn into is a note saying "start at -Vsomething for background music, car music, casusl listening or workout music etc", and that is so close to the current recommendations that it's probably not worth pursuing.


I admit, I don't, I was just guessing. I think I may have confused "casual listening" with "listening at acceptable levels of distortion", which is what the 128 tests have been doing.

This thread is getting a bit out of hand, and the issue itself is kind of moot for me because I would never use such a "casual listening" encode, and Dibrom has made enough good points about the efficacy of all of it, so I'll bow out at this post.

Axon, there is another reason why this would be a waste of resources...... you talk about a "relaxed/quick test".... which basically just means "half-assed" - to "simulate" the attention-strength with which average joe usually listens to music....... guess what, i'll tell you what will probably be the result, asuming that proper blind-testing-methodology is preserved: At -v5 almost no one of your "untrained casual listeners" will notice a difference.... heck, they probably will even fail at -v6!

So what does that tell you? Nothing, because people will turn the "goal" of the test into a flaw: They will begin to ask "oh, but i want to have some safety-buffer in case i listen more carefully to music. And i dont want to reencode my mp3s when i get better equipment.". So, the logical step for those people is to encode at -v4 (preset medium). HOWEVER, exactly this setting was already considered the "safe setting for average joe" BEFORE you did this test - so in the end, you revealed nothing new and the test was a waste of resources.

Its like with many other things: The more effort you spent on a test, the more useful the results. The less effort you spent on a test, the more useless the results. With this, i dont mean that a test with "average joe" should not be done - but you cannot lower the quality of the test without as well lowering its usefulness.

OK now, saving disk space is actually a practical and sensible goal. Lowering quality just because it won't be missed (due to poor hearing or equipment) is not. If file size is not a matter of concern, then the "excess" audio quality does no harm.

Achieving the best possible quality at given lower bitrates is already a subject that gets a fair amount of attention. Listening tests and encoder (+ settings) recommendations are out there. An individual only needs to do a relatively small amount of personal testing to choose what's acceptable for their particular needs.

I think what is really needed isn't a "casual listening test", but instead V4/preset medium getting a more prominent presentation and attention. I suspect that the high attention which V2 and V0 (standard and extreme) get here is the reason why "casual listeners" feel that their needs aren't covered enough. So, IMHO it is not a lack of knowledge/data which is the problem but instead the "marginalized" presentation and popularity of V4/medium.

I'm with Lyx here. I can recognize artifacts pretty well by now, but on the vast majority of stuff I can't tell -V4 apart, and I encode most stuff at -V4 or -V3, going with -V2 on something that I'm more likely to be archiving (e.g., when I rip a friend's cd).

Dibrom, I can understand that you've seen this sort of thing before, and I agree that relaxing the ABX standard wouldn't make any sense, but I think that an "untrained listener" isn't so much of a definitional problem....

Overall, I think the best thing is to give more attention to -V4 and -V5.

On a semi-related note, I had a friend over on Wednesday night, a physicist who's an audiophile and really into classical music and obscure metal, and apparently has really good hearing. He thinks he can tell -V4 encodes apart pretty eaily because he's familir with the music and can sense that it's lacking some high-frequency. Then it turns out he can't ABX a typical metal sample (opening of Metallica's sad but true) on -V5. Boy, was he squirming, and not because I was pressuring or making fun of him, but just because he realized that he had no idea and was failing a double-blind test on relatively lower-quality encoded music.

It's not so much that "untrained listener" is a definitional problem (though it is in it's own right, and I think I explained why) as it is the case that practically the entire constraints for such a test are completely arbitrary, and practically meaningless.

I agree with Cosmo here, and this is what I tried to convey earlier: The entire premise for such a test as this is flawed. Instead, you need to simply shoot for the best possible level of tuning and quality that you can for a given resource target (i.e., bitrate, whatever). Then, it's up to the actual listener himself to choose a point along the scale of different possible resource configurations to find what works for him. If he wants to find something that works for "casual listening," then great -- he determines what "casual listening" means for him, and can discover what works best for him to achieve that by performing his own private listening test.

You can't properly test for the "average listener" in the fashion that most tests around here are performed (in other words, you can't really do it in a highly relevant and meaningful way). And trying to tune for such a target is not only just about impossible, but a very bad thing to even attempt at that.

This discussion is very interesting. I joined on this forum because of a recent purchase, my most advanced piece of audioequip, an ihp-120. After researching portable devices for some time, I concluded the decision after comparing the firepower of everything on the market. Specifically, the ability to record (highest quality available) on the go, while maintaining a decent file compatability, drive size, and good quality output.
It was some time ago that computer audio tech was hitting the consumer market, and I asked a friend what an mp3 was. A filetype that "cuts out" all the noises that we are incapable of hearing on track. There are few people who can hear one instrument in a 50 musician arrangement, but numerous people who could spot a change from the original. The difference between 128k/s & 192k/s wavs I found to be huge. I avoided electronic audio for some time because I know nothing about the mechanics and theories behind audio/sound, and know nothing of hacking. I do not believe it is possible to remove the inaudible alone, and would go almost as far as to say those sounds are just as important as the rest of the track. As one could imagine, I am having difficulty on deciding how to encode my lib. Lossless will very quickly kick my hardspace's ass, and I'll end up having to grab an extra 200$ worth (a matter of time rather than neccessity). And then I wonder, are lossless formats lossless? And then after much reading.... I ran into this topic, and yet feel nowhere closer to what I'm looking for. I hope some of these words will provoke discussion towards comfortable resolution, and give me some clarity on what to do now.

It's possible in theory if you have a perfect psychoacoustic model and know the various hearing thresholds of everyone that ever existed or will exist

But in practice, it's impossible, yes. There will always been an exception out there -- someone who is able to hear a change in a file that 99% of the other people will not. That's why people perform these listening tests (to get the best idea possible), and why codec developers are always looking for ways to improve things.



If you can hear what an encoder is changing, and it bothers you, then yes it's just as important as the other stuff. In this case, the encoder's model is wrong.

On the other hand, if you can't hear it, then it's NOT as important as the other stuff. This is at least if you are talking about subjective perceptual quality. If you're implying something else that isn't related to perceptual quality, then that's a different story. But that's also outside of the scope of this discussion really.



Yes. Assuming that the encoder is not buggy (this sort of bug is easy to test for usually), and that your computer components are working properly (meaning you don't have bad RAM or a bad HD which could both end up corrupting data), then a lossless file should decode to a 1-to-1 bit identical clone of the input signal.

If that doesn't answer things, you probably should start another thread

Lots of interesting comments in here, especially Axon's idea of distributed listener results. But I can see both sides of the argument.

I guess I have to learn how to do ABX tests (without trying too hard) for my own conditions, I get the general idea of them, but probably need to read more on how to do it.
Is 10 short samples of various music styles be enough?

oh well... I've done one for myself and headphones, 12 songs. Anyone care to hear the outcome?

sure!

OK

The executive summary:

My first ABX.. It was so much easier to run than I thought it would be, it just got tedious.

Testing: Lame 3.96.1 -V switches. Budget Headphones.
The goal was to find my transparency threshold with my current listening equipment.


V9 - very easy to pick audio defects. I think even the dodgiest audio set-ups would benefit from settings >V9.

V8 - Mostly transparent for me, but some artifacts weren't too hard to pick up at all, so that's obviously not going to be enough to optimize all my music.

V7 - I can rarely make out any differences between this and wav, unless I try. Good enough for me and my sample of songs. I'd probably suggest this to people with low-end hardware who care more about the music than the audio (but still care enough to optimize size & sound).



I was surprised that I only got this far. I was thinking I'd get to V5 before it got tricky... Ah well, it only has raised my respect bar ridiculously high for those who can differentiate V2 from wav.


Here's my crude, completely non-standard issue graph. It's just the overall results. This test was for me, so I'm not going to try and waste too much more of your time with logs files, etc..

edit: look down

it stretched my post making it hard to read...
here it is

10$ speakers, right? I can barely make out 128 from aps on those (though i still can.. actually, i can diff. 192 cbr from aps, pretty easily...; I've only found a difference between aps and wav on few samples.)

(on a sidenote, try lame 3.97b1, for your abx test, again, if you can -- it might be transparent at v8 for you! )

LOL!

I juts think that "relaxing" the ABX is just what who doesnt know what ABX is already doing nowadays: encode the music, listen in the speakers and say "perfect" or "it's really good", changing between the original and the encoded. I think that the great majority of people dont care about small artifacts at all - so they test by themselves in their proper ways. The problem is when someone testing samples using this method tries to clame to everyone that nothing can beat his test.

Or it is testing without really testing anything. They could as well encoded at v5. Besides of "minor artifacts", those users will probably notice no serious problems. And there is even a good chance that they wont even notice the minor artefacts. The question is: if we know this already with high probability, then why do we need multi-listener tests? Just to ease the "good feeling in they belly"?



Because it is not representative and unreliable - what was the purpose of a test again?

Tests like the one which blue did are a useful addition. But relaxing the requirements which are needed for reliability will make the those tests weak and meaningless for other users. People would basically just test for their own placebo and not for really knowing whats going on. As mentioned earlier, you cannot relax the fundemental requirements without loosing something important(relevance).

I dont want to stop people from testing for placebo. But ha.org is the wrong place for this kind of "research". Even the dreaded TOS8 would be in conflict with conducting this kind of tests here.

- Lyx

Hey, I'm not into relaxing the tests. I wasnt saying that as an argument to relax the ABX tests: i was just saying we dont need tests for the great majority of people because they dont really care about that.

Okay, sorry - my misunderstanding of your words.

What placebo?
It's blue57's own ears; or are they not as 'qualified' as guruboolez'?
Even guruboolez stated that his test results are from his own ears, and that it might not represent every audiences.
And most people here seem to take it (the results) for granted.

But if what u meant by 'testing for placebo' is 'a relaxed ABX test', then I agree.

Yes, i was adressing what would happen if blind-testing conditions are taken less serious - the results would be useless for others, and the tester as well - so, the tester would only do the test to satisfy his own "warm feeling of asumed safety".

Just the other day, my wife and I are both in our office working while listening to mp3's. All at once I audibly shutter as I shout, Dam!%. There was a slight artifact in the track! At once I began preparations to re-rip the track while making sure everything is kosher with my rig. Through all this my wife just looks at me like I am crazy. Even after replaying the offending section to her multiple times she could not understand what the problem was and why on earth it would get me that agitated. That is just the difference between us, what sounds acceptable to her would keep me up at night. But I love her anyway =) Because like love, music quality and acceptability is subjective. We all know this.

My wife would never come to a forum like this and ask a question about audio setup or formats. But I think that if a person DOES comes here to ask what the forums opinions are, that person is demonstrating that while they may not be an audiophile, they do want a decent listening experience and are trying to get input from a group whose opinion is valued. I don't see why it should not be ok to do so without all the rigamaroll.

How many times has a close friend told you, 'here try this, you'll love it!' So you try it and it it's terrible! Your physiology is the same, your interpretations are not. But the opinions of others have value in so far as that it gives you a place to start figuring out what works for you. As long as it’s stated as opinion, then I have no problem with it, it’s when a putz extols their opinion as fact that the ship runs a ground.

Disclaimer:
The opinions of rockdog are solely his and may not be reproduced in whole or in part w/o the express written consent of the national football league. Said opinions are in no way legally binding and the author reserves the right to amend the opinion as new information comes to light. Any violation of TOS #8 either real or imagined is understood to be bad form but should not result in ridicule or death.

That's ok, I've been writing ambiguous enligsh sentences lately.