Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.

Poll

96 or 128 kbps public test?

~96 kbps test on "average music" samples with little amount of difficult samples
[ 55 ] (39.3%)
~128 kbps test with a higher amount of difficult samples
[ 85 ] (60.7%)

Total Members Voted: 160

Topic: 96 or 128 kbps public AAC test (Read 39632 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

96 or 128 kbps public AAC test

Reply #25 –
Quote from: greynol on
That's great and all, but what about the codecs that are actually going to be tested?


WTF? I realized just now that LAME is not planned to be included.  A second bell rings, duh, the whole things seems to be called "AAC test" for a while now and not "multiformat", anymore.

From my subjective perspective, we have accumulated a lot more unfinished AAC vs. MP3 business recently than which AAC encoder was the best. No chance to slip LAME in there with the special role as venerable touchstone?

96 or 128 kbps public AAC test

Reply #26 –
Quote from: IgorC on
greynol
I invite you to read the whole discussion's topic http://www.hydrogenaudio.org/forums/index....showtopic=77272

What makes you think I haven't done this already?

96 or 128 kbps public AAC test

Reply #27 –
My opinion is that despite being very democratic, this pool was biased from the start, there is more than 100 voters but there is not more than 100 HA users doing regulary ABXing & posting their findings.
What this means is that many testers will be the average Joe testing their ABXing skill. But IMHO on a daily basis the average Joe is likely to use a bitrate higher than 128Kbps because for his daily music he likes transparency.
Now what this means for this pool is that the average Joe, is very likely to be willing to test a setting that is the closest possible to the setting that he uses on a daily basis, that's why the average Joe is very likely to vote for 128Kbps.
Some people said that testing 96Kbps would not be a great idea because nobody would use 96Kbps to encode his music ... this is exactly the behavior that I was expecting from an average Joe.
An advanced ABXer knows that a codec that his bad at 128Kbps isn't suddenly good at 96Kbps ... so it doesn't matter ... what matters is the choice of samples.

You can have all encoders put to shame at 128Kbps or all encoders looks shining at 96Kbps, all this just by changing the set of samples you use.

The choice of samples will be harder at 128Kbps than at 96Kbps ... at 96kbps you cannot really do a mistake in the choice of samples ... at 128Kbps lots of samples will be hard to ABX for beginners.

Not all killer samples are equal, some old killer samples have become less critical with recent encoders, but some people still use them for listening tests ... if you chose one of those at 128Kbps it will likely be a sample tested for nothing.

96 or 128 kbps public AAC test

Reply #28 –
I voted 96.

Based on my experience of all Sebastian's tests a 128 kbps AAC public test would be unpleasant and unrewarding for the testers.

In some of the past tests 96 kbps iTunes VBR was used as a high anchor. It was never transparent to me, but I didn't find it annoying at all. In fact, it was the only encoder in the 64 kbps multiformat test that was not annoying to me. In general, I was disappointed to the quality of 64 kbps AAC-HE. However, when I listened through a playlist of the 64 kbps HE-AAC test samples in a casual situation (using my big hifi speakers instead of headphones) I was surprised about the apparently qood quality. Perhaps this explains why so many testers had problems with hearing the HE-AAC problems in the test. You really need to have lots of experience before the minor artifacts can be detected.

I don't think the samples in the previous tests can be considered "easy". They were intended to represent various typical encoding problems and were chosen after careful consideration.

In general I don't think the so called killer samples are very deadly for 128 kbps AAC. Obviously they are far from transparent when the tester is experienced and knows what to look for, but detecting the problems might be difficult for a tester who has no previous knowledge of the typical artifacts.

The two most recent 128 kbps tests clearly showed that many testers had problems with finding any differences between the encoders.


It would be good to start a new round of HA tests with a test that is easier for the testers and for the test organizers. It would already be difficult to find a good sample set for a 96 kbps test. After a successful 96 kbps test it would be easier to organize the more demanding 128 kbps test when all the experience of the 96 kbps test is available.

And there should not be a long break between the tests. Once the new generation of participants has been summoned and trained it should be actively used. 

P.S.

If the test turns out to be a multiformat test the only addition to a 96 kbps test besides AAC should be Vorbis. MP3 is really not competitive @ 96 kbps (I believe many HA users can agree that or do I need to post ABX results?).

A 128 multiformat test would be different. MP3 should be included. However, I am not sure if it should be LAME. There was no winner in the last public 128 kbps MP3 test.

96 or 128 kbps public AAC test

Reply #29 –
Quote from: sauvage78 on
An advanced ABXer knows that a codec that his bad at 128Kbps isn't suddenly good at 96Kbps ...


But a codec that is bad at 96kbps may suddenly be better than its peers at 128kbps.

PS

But I think you still have a point regarding average Joe voting behavior and its roots.

96 or 128 kbps public AAC test

Reply #30 –
I'm an average joe, and I voted for 96.

96 or 128 kbps public AAC test

Reply #31 –
I can only speak for myself & I already posted my logs but for me Harlem & Autechre are deadly for AAC 128Kbps, I don't use lossy but I wouldn't use AAC below 192Kbps VBR actually.

The so-called quality of AAC 128Kbps is more due to poor average listening skill & bad samples choices, than to the real efficiency of AAC 128Kbps.

AAC 128Kbps is only usually transparent, if you randomly listen to average music. I hope average music is not what will be selected for the test.

rpp3po
Quote
But a codec that is bad at 96kbps may suddenly be better than its peers at 128kbps.

This is not my experience. I think that becoming "suddenly better" is more an exception than the rule.
What I mean is that from my experience the ranking between codecs didn't change, it does happen that a codec which was bad at 96Kbps is suddenly transparent at 128Kbps when other codecs are still not transparent ... but within all the encoders at 96kbps it is very likely that it was already the best result. (There is the exception of some critical point within the encoder like suddenly no more SBR, but these are exceptions)
It never happened to me that codec X would beat codec Y at 96Kbps & that suddenly codec Y would beat codec X at 128Kbps, all this within the same sample. Different codecs have different strenght & weakness on different samples, but on a target same sample the results are usually very linear.

96 or 128 kbps public AAC test

Reply #32 –
Quote from: Alex B on
If the test turns out to be a multiformat test...

How is that coming? Magic?
oh, yeah I see: The choise of  AAC competitor for multiformat will be based on....  on... on....
on what?

96 or 128 kbps public AAC test

Reply #33 –
Quote from: greynol on
Quote from: IgorC on
greynol
I invite you to read the whole discussion's topic http://www.hydrogenaudio.org/forums/index....showtopic=77272

What makes you think I haven't done this already?


Ok, discussion around higher bitrate was started from actually very reasonable rpp3po's jump into it.

Actually Sebastian's test were some kind of average music + some difficult (not so killer test). It's important.

Untill today there is no any public killer sample test. How can somebody be sure if it's usefull or not when it was never performed?
Excuse me my naivety but when Einstein has spoken for the first time about his theory everybody thought he was retarded.

Let's think a little bit differnt just for one test. If we wrong about 128 then that's ok. Good conclusion would be that 128 for average listener will be more than enough even for very killer samples.
Untill now we can't say that.
Can we?

96 or 128 kbps public AAC test

Reply #34 –
Another 128kb/s test has no value to me.  It just doesn't give conclusive results about what encoder is doing universally better.  96kb/s should do that much better.

A fully-AAC test makes even less sense.  Why are we treating AAC like a more major format than it is or ever will be?  If you want to do a multi-encoder AAC test, then it doesn't require all the build-up; just do a quiet side-test with the other 12 people on Earth using it as their primary lossy format.

96 or 128 kbps public AAC test

Reply #35 –
Quote from: sauvage78 on
You can have all encoders put to shame at 128Kbps or all encoders looks shining at 96Kbps, all this just by changing the set of samples you use.

The choice of samples will be harder at 128Kbps than at 96Kbps ... at 96kbps you cannot really do a mistake in the choice of samples ... at 128Kbps lots of samples will be hard to ABX for beginners.

Not all killer samples are equal, some old killer samples have become less critical with recent encoders, but some people still use them for listening tests ... if you chose one of those at 128Kbps it will likely be a sample tested for nothing.

Which is why I propose not to randomly select items from a set of "maybe-killer" samples for the 128-kb test, but to do careful pre-screening of listening material and to select only those items for which at least one of the codecs under test is clearly not transparent.

Regarding the "will be hard to ABX for beginners": IMHO, this should not be a test for untrained listeners. No matter which bitrate! This is exactly the mixture which gives inconclusive results: experienced and unexperienced listeners. At such high bitrates, only the former should participate. This is HA, so I expect quite a number of members belonging to that group.

Quote from: Curtor link=msg=0 date=
Another 128kb/s test has no value to me. It just doesn't give conclusive results about what encoder is doing universally better.

A 128-kb as I proposed it above is not trying to answer that question. It will answer the question "which AAC encoder does best given very critical audio material". The thing is, universally, all popular modern AAC codecs do very well at this bitrate.

Every sentence of your post shows that you are not interested in AAC. Believe it or not, but AAC is a major format, and more than 12 people are using it (not necessarily only people ripping their CD collection... think further: iTunes, YouTube, lastFm, ...). For me, a 128-kb test as above will have a lot of value, because it will show me how close to transparency you can get at that bitrate with the most critical material available.

Chris
If I don't reply to your reply, it means I agree with you.

96 or 128 kbps public AAC test

Reply #36 –
Quote from: C.R.Helmrich on
A 128-kb as I proposed it above is not trying to answer that question. It will answer the question "which AAC encoder does best given very critical audio material". The thing is, universally, all popular modern AAC codecs do very well at this bitrate.


But how do pick "critical" material in an unbiased manner?  If you just pick problem samples for AAC from this forum, you'll mostly get things that break Nero and maybe a few that break iTunes.  Thats not really a fair test since theres much fewer people who would identify problem samples in other encoders.  Would you pick problem samples identified for other formats then?  Perhaps this is less biased but these files may be less "critical" then you'd like.

96 or 128 kbps public AAC test

Reply #37 –
Quote from: Mike Giacomelli on
But how do pick "critical" material in an unbiased manner?  If you just pick problem samples for AAC from this forum, ...

I don't rely on posts in this forum exclusively. I'm in the process of finding my own additional samples, as well as going through about 100 MP3-critical samples posted or linked to in this forum.

Quote
Would you pick problem samples identified for other formats then?

No. This is supposed to be an AAC test, so I only search for AAC-specific ones. However, AAC critical items should also be MP3-critical, since the underlying codec technologies are similar. If, however, I'll find a sample which is AAC- but not MP3-critical, I'll report.

Chris
If I don't reply to your reply, it means I agree with you.
 

96 or 128 kbps public AAC test

Reply #38 –
Quote from: greynol on
So does that mean I can put you down as one person who can routinely distinguish 128kbit AAC from lossless as part of your regular music listening experience?

@greynol

If this was directed at me, of course not. I would not claim to be regularly able to detect 96kbit from lossless either. But, on the other hand, I never said or implied that I or any body else, come to that, could.

What I actually said was that we expect all AAC encoders to be good at 128, but that performance on really different examples might be a possibility for differentiation. Of course, it could be that 96 + difficult samples would be more informative.

I only chipped in because you seemed to be getting angry because some people were voting for 128. You still seem to be too angry to actually read what people are saying; which, in my case, was that there could be a reason for a different choice from the one you favour (though not denying that your preference might, ultimately, be the better).

I notice that some people with more knowledge and experience and better ears than me see a point in a test at 128 with difficult samples; I also note that there are problems with that. Further, there seem to be several different ideas as to what a test now is meant to achieve; some people seem to want an exciting ABX test, with good sport in the field; some people want a general rank ordering; some people want something that illuminates the really difficult areas and fine points of encoders that are, in general terms, already good enough.

The only dog I have in this fight is help in choosing settings that won't hurt other people's ears. If this poll is only for participants in the test, and not for consumers, then I would certainly respect that decision, but it ought to be made clear. I really think things would be a lot clearer, and there'd be more light and less heat, if the purpose of tests were articulated.

96 or 128 kbps public AAC test

Reply #39 –
I'm not angry, I really don't even care about the outcome.  I just think there is a serious risk that the test is not going to reveal the results people are hoping for.  I think more weight should be given to those who have actually conducted these tests and/or those who have demonstrated an ability to detect artifacts than to those who will likely not participate let alone provide data that will be useful.

Your saying that you don't feel that you would be able to contribute simply reinforces my point.  Thank you.

Perhaps I'm wrong in that people aren't interested in results that are routine and regular for them, but I don't think so.  I may also be wrong in expecting that people are going to make unfounded suggestions that codec X is better than codec Y because of the results of this test, like what was done with the ~128kbit mp3 test, assuming that the results are not particularly interesting except for the killer samples.  To this day, there are still people who can't seem to understand that the result of that test was a statistical tie.

96 or 128 kbps public AAC test

Reply #40 –
Quote from: C.R.Helmrich on
Quote from: Mike Giacomelli on
But how do pick "critical" material in an unbiased manner?  If you just pick problem samples for AAC from this forum, ...

I don't rely on posts in this forum exclusively. I'm in the process of finding my own additional samples, as well as going through about 100 MP3-critical samples posted or linked to in this forum.

Quote
Would you pick problem samples identified for other formats then?

No. This is supposed to be an AAC test, so I only search for AAC-specific ones. However, AAC critical items should also be MP3-critical, since the underlying codec technologies are similar. If, however, I'll find a sample which is AAC- but not MP3-critical, I'll report.

Chris


If you pick AAC specific problem samples, then you bias the test results against whichever codec the samples were identified from, and your results become less useful.  For instance, if you pick a lot of samples identified from iTunes encoded files, then your conclusion becomes "For AAC files that iTunes has trouble with, iTunes does poorly . . ."

96 or 128 kbps public AAC test

Reply #41 –
Didn't somebody (muaddib?) come up with a sample choosing methodology that would be fair to use? When I read about that it seemed to make a lot of sense to me. Or we could just classify each type of encoder problem (preecho, tonal, etc.) and make sure there is an even distribution of each type of problem, and within that the usual even distribution of genres, recording era, etc.

About this poll, I don't really mind either bitrate but I would like a high proportion of difficult samples for the reason others have said. Because without them most AAC encoders are so good and results might not be as strong statistically.

96 or 128 kbps public AAC test

Reply #42 –
Quote from: Mike Giacomelli link=msg=0 date=
If you pick AAC specific problem samples, then you bias the test results against whichever codec the samples were identified from, and your results become less useful.  For instance, if you pick a lot of samples identified from iTunes encoded files, then your conclusion becomes "For AAC files that iTunes has trouble with, iTunes does poorly . . ."


Not really following you here.  If we're talking about an AAC shootout (which determines the one AAC codec to be used in the multiformat test), then the sources would have to be wav/flac samples, and every AAC encoder is on equal footing.

96 or 128 kbps public AAC test

Reply #43 –
Quote from: greynol on
Your saying that you don't feel that you would be able to contribute simply reinforces my point.  Thank you.


It is not clear that this thread is addressed only to people who can contribute useful results. It is not perverse to think it might be asking for a general expression of opinion from people who will read the results (and who will, by the way, value the effort and skills of the people who conduct them). I accept that, in at least some people's view, this is in fact a poll of potential participants. That's fine, and if there were a way of retracting my vote for 128, I would. But it would be better if the target audience of the poll were made clear, rather than being snarky to people who guessed wrong. The internet is a very public place, and if the title of the thread has the word "public" in it, it's good to indicate that it's not open slather. I really can see the point of asking just the people who are going to test.

Quote
Perhaps I'm wrong in that people aren't interested in results that are routine and regular for them, but I don't think so.  I may also be wrong in expecting that people are going to make unfounded suggestions that codec X is better than codec Y because of the results of this test, like what was done with the ~128kbit mp3 test, assuming that the results are not particularly interesting except for the killer samples.  To this day, there are still people who can't seem to understand that the result of that test was a statistical tie.


Well, that test would, in some people's eyes, have been "uninteresting", because it resulted in a statistical tie. But from another point of view, that was a most interesting fact, in that it showed that some other MP3 encoders had caught up with LAME. In other words, it all depends on what the point of the test is; and the only interest I have is to try to help to clarify that issue, in the midst of everybody making their points.


96 or 128 kbps public AAC test

Reply #44 –
The problem with public listening tests is that they are
- very useful on one hand (because they give some important information about encoders)
- and of restricted use on the other hand (because they give limited information about encoders mainly due to sample selection).

Not many people like such a situation. Most people prefer sporting for 'winners'. That's why listening test results are misinterpreted so often.

As sauvage said sample choice has an essential influence on the test (no matter whether @ 96 or 128 kbps).
But it's something we can't get a perfect solution for, so we shouldn't worry too much about it and be content with a set of samples which not obviously gives some bias.

We are lucky to have C.R. Helmrich here, and as FhG AAC doesn't participate in the test (unfortunately) I suggest to let him with his AAC dev experience build up a proposal set of samples (in case he is willing to do so). He is thinking already about such a set.
Of course in the time of this proposal list building new proposals can be made here.
The proposal list should be rather big, and for the final choice a random selection as proposed by IgorC can be used.
lame3995o -Q1.7 --lowpass 17

96 or 128 kbps public AAC test

Reply #45 –
Quote from: C.R.Helmrich on
Which is why I propose not to randomly select items from a set of "maybe-killer" samples for the 128-kb test, but to do careful pre-screening of listening material and to select only those items for which at least one of the codecs under test is clearly not transparent.

Sounds good to me.
Future discussion of samples will help to avoid samples which were submitted as issues only for one particular encoder.

Quote from: C.R.Helmrich on
Regarding the "will be hard to ABX for beginners": IMHO, this should not be a test for untrained listeners. No matter which bitrate! This is exactly the mixture which gives inconclusive results: experienced and unexperienced listeners. At such high bitrates, only the former should participate. This is HA, so I expect quite a number of members belonging to that group.

It sounds optimal at least for me.
I propose to add 4th rule.
Code: [Select]
Remove all listeners from analysis who
1. graded the reference lower than 4.5,
2. graded the low anchor higher than all competitors.
3. didn't grade the low anchor.
4. didn't grade any of competitors.

There is no any value of results when all competitors were ranked at 5.0 and low anchor at 2.0.

Quote from: halb27 on
We are lucky to have C.R. Helmrich here, and as FhG AAC doesn't participate in the test (unfortunately) I suggest to let him with his AAC dev experience build up a proposal set of samples. He is thinking already about such a set.

Yes, he is co-organizer of test.

Thank you Chris for you great contribution.

96 or 128 kbps public AAC test

Reply #46 –
Quote from: sauvage78 on
My opinion is that despite being very democratic, this pool was biased from the start, there is more than 100 voters but there is not more than 100 HA users doing regulary ABXing & posting their findings.

Honestly I should admit you're right.

The gived situation when some well know HA members prefer 96 kbps while other 128 kbps we can test at intermediate bitrate.

Apple TVBR q55 gives ~115 kbps but  there isn't such bitrate for CVBR (only 96 or 128).
115 was never tested so nobody can make claims of difficulty 
It's an option to drop CVBR.
Opinions are welcomed.

96 or 128 kbps public AAC test

Reply #47 –
Thank you, IgorC and halb27, for the flowers  If some expert listener wants to help me with the pre-selection of items (which would be great), please send me or Igor a message.

Some more comments:

Mike:
I don't understand what you're trying to say, either. When selecting items for the test, I will check, for a given item, the performance of every codec under test. Since all codecs will be AAC, that sounds fair to me. If I happen to find more critical samples for codec X than for the other codecs (of course I won't tell anyone before the test), doesn't that tell us something about codec X?

IgorC:
Code: [Select]
Remove all listeners from analysis who 4. didn't grade any of competitors.

Given that at least one competitor will be clearly non-transparent (to me at least), yes, that would make a lot of sense. But I think then we also have to modify rule 1 to:
Code: [Select]
1. graded the reference lower than 5.0

As I noted earlier, this will greatly reduce the number of listeners left for analysis, but given that we only want experienced ones, so be it.

I prefer not to use an intermediate bitrate because Fraunhofer's encoder doesn't have 112-kb VBR either  I'm sure you understand that, after the test, I will want to compare Fraunhofer's encoder against the per-item winner. If not on this forum, then at work.

Note about this poll: What's the problem with non-listeners voting? If someone votes but is not going to participate (or be kicked out before analysis), that's fine! It still shows us which bitrate people are interested in.

Note about the analysis: I'm planning not to publish an average grade for this test, but only to show the results for each item separately, if on average, all codecs will be tied. That will prevent some people of falsely assigning the word "winner" to a codec. If of course, one codec will be significantly better on average, I will show that. What do you think?

Chris
If I don't reply to your reply, it means I agree with you.

96 or 128 kbps public AAC test

Reply #48 –
Quote from: C.R.Helmrich on
... But I think then we also have to modify rule 1 to:
Code: [Select]
1. graded the reference lower than 5.0

As I noted earlier, this will greatly reduce the number of listeners left for analysis, but given that we only want experienced ones, so be it. ...

Good idea IMO.

Quote from: C.R.Helmrich on
...I'm planning not to publish an average grade for this test, but only to show the results for each item separately, if on average, all codecs will be tied. That will prevent some people of falsely assigning the word "winner" to a codec. ...

Wonderful.
If it were me I would not consider averaging the results at all. The average itself has no real meaning, it is just a formal way to transform n results into one (loosing important information), in order to satisfy the 'sporting for winners' attitude. Sure I know not many people share my way of seeing this. But practical conclusions for encoder selection can be drawn without an average, focussing on results for samples of individual practical relevance.
lame3995o -Q1.7 --lowpass 17

96 or 128 kbps public AAC test

Reply #49 –
All this discussion about usefulness of testing killer samples is coming through important fact that today the focus of lossy encoders has changed.
Nowdays it's not enough anymore for lossy encoders to be "good enough". As HDD space and internet band are enough large  that today people see lossy encoders more as replacement of lossless without bitrate trade off.

If the approach of lossy encoders has changed then testing methodology should changed also.
Today I see more sense in testing only difficult samples at least at 128kbps.