I agree with these contenders.


Actually I prefer the 30 second length. In the last test, there were a few samples where I focused on one area near the sample beginning, gave grades, but then discovered an area near the end where an artifact was worse, so revised my score on all contenders by checking any regression at that point as well. If I had not access to that portion of the sample, my grade would have been artificially high on some encoders.

I do agree also that we should remove samples that scored high on all encoders.

I disagree somewhat. Although I do see you point, of using the best setting for ~128kbps LAME, there have been claims by certain vendors that their audio codec performs as well or even better at 64 kbps than mp3 @ 128 kbps. AFAIK these claims have been based on 128 kbps cbr encodings and even though cbr should deliver a bit worse quality than the --vbr-new -V5 setting, I think the comparison between the contenders and lame @ 128 kbps cbr would be of higher interest.

/Kef

This is exactly the reason why samples should be shorter. 5-10 second part of a sample with strongest distortion should be used, so that listeners can concentrate on that part. Many listeners will not do what you did for that 30 second sample and also somebody might miss that part since he concentrated on another one.


Do we really care to show in this test that these claims are wrong or correct?

I personally also dislike 30 second samples cause my acoustical memory doesn't last that long (except for obvious samples). With unknown 30 second samples I always search for the most critical spots first.
Usually a shortened sample would make life a lot easier for people like me.
Of course if there are several critical spots of different character not close to each other the shortened sample should contain them all, but this happens rarely (and even in this case it might be the better alternative to consider this case in several samples).

OK, I did some tests with Nero's LC-AAC implementation and have to say that it's not really worse than WMA Standard in case we want to test that (what I would like). Therefore, it's not really good as low anchor.

That's flawed and dangerous method. How can you sure that the distortion always exsist with every encoder? We all know that short sample is easy to test, but each encoders have different weakness. That's why we have to test many long samples to look for unknown artifacts.

Such a short but difficult sample is maybe killer sample for a certain encoder. You shouln't use that in listening test.

You are right that killer samples shouldn't be used. Shorter parts should be carefuly chosen to account for all known weaknesses that exist in any of encoders to be used, not just to impose a weaknesses of one of encoders. Very problematic samples like fatboy shouldn't be used.
But I don't agree with unknown artifacts. Encoders to be tested are used already for a long time. This test is to judge and compare quality of already known encoders and not for searching unkown artifacts (which I don't expect to be found in this test). Instead it might be good to include some new samples with artifacts that don't exist in samples used in previous tests and that were found in meantime (if there are any). And during this discussion searching for new artifacts can be done.
And using longer samples would probably not help to find unknown artifact. What is more expected is that most part of a sample will be of no significance or that distortion will be repeated. And if there are two different kind of distortions in a long file, then grades will be very vague for the reason that halb27 mentioned - acoustical memory is usualy very short. If you have those distortions splited into 2 samples then it would be easier for people to judge them consistently.

The artifacts of lossy encoding is hard to describe in word. You will find it almost impossible to define very subtle artifacts of every samples on ahead. If you know all existing artifacts of encoder, why should we test again? To make things worse, at middle to high bitrate, that's barely audible for most of listener.


That's just a bad sample or a listener couldn't detect artifacts. Short sample means little chance of telling an artifact that is unknown for a listener.

Well, I agree that the samples should be somwhat shorter - Locomotive Breath with its 43 seconds is really too long for example. However, I have nothing against 20 seconds samples.

It's possible to delay the test by a few weeks since a user here would like to conduct a quick speech test in order to present the results during a speech.

20 seconds sounds reasonable to me. What about choice of samples?
When do you expect that test will start?

I'm on vacation until Easter, so feel free to delay it as long as you like, Sebastian.

RE sample length: I tend to use the 3-5 'worst' seconds of the low anchor in any given sample and judge the other samples accordingly. Like halb27, my acoustical memory isn't that long for subtle differences.

Well, it would be cool if the guy with the test could contact me - I only got the info from Roberto.

What do you people think about not using the results where low anchor was given 5? IMO it means that the person does not hear well enough and that his results are not valid. It also may mean that low anchor was not chosen correctly, but I hope that can be avoided while choosing low anchor.

Yeah, possible.



Why not? A public listening test is meant to be public and address a large mass. If some persons really don't have problems with the low anchor, it means that in their special case, even the low anchor gives acceptable results.



Now this is going too far. If you ask a 90% deaf guy if he thinks that LAME at 48 kbps stereo sounds well enough for him and he agrees, it doesn't mean that the low anchor was not choosen correctly.

Also, the low anchor is not only meant to sound bad, it is meant to be the worst of the tested codecs. Otherwise we would all be using Shine at 32 kbps or something.

The problem is that we settled on a MUSHRA test, which is more appropriate for a vocodec test. But there are exactly 0 suitable MUSHRA comparers for Windows, so I'm not sure if the test will be feasible after all...

You wrote on your site that you try to adhere to specification ITU-R BS.1116 as much as possible.

From ITU-R BS.1116:
What I propose is to use second method of "Post-screening of subject" from ITU-R BS.1116: method which "relies on the ability of the subject to make correct identifications". Pre-screening is not possible in your listening tests, first post-screening method is not justifiable, so only "correct identification method" can be used.And if no method for pre/post-screening is used than small number of subjects (20) is not sufficient for drawing appropriate conclusions from the test.

this does not make sense to me. to have broad statistically valid results you should reach most of the listeners and do not discard any, except for ranked reference (IMO).
ITU full compliance is not really needed (assuming you are interpreting the recommendation correctly).

In recent 80kbps AAC listening test, menno discarded results from whom a listener didn't tell contenders except low anchor to get more accurate conclusion. IMO low anchor itself isn't necessary. It never helped me do ABC/HR.

Take the following scenario: we do a public listening test at 320 kbps. As low anchor, we choose something like AAC at 160 kbps. After the test is over, I have 100 results. 10 of these results have a rated low anchor, the remaining 90 don't. If I screen only the results of the 10 testers who managed to rank the low anchor, we get the impression that 160 kbps is not transparent. Is this true? It is, but only for 10% of the testers - for 90% of the testers it was, so we can assume that the codec did a pretty good job.

Anyways, I doubt that anyone should have problems finding the low anchor in an 80 kbps test where the low anchor will have 64 kbps or less.

Edit: Corrected spelling and grammar.

One more quote from ITU-R BS.1116:
One more quote:

Of course post-screening process should be done carefuly. If unappropriate low anchor is chosen then problems might occure. But as you say it should be easy to find low anchor in this test.

Oh God.

Dude, here we are using 1116-1 for its METHODOLOGY, not for for its PURPOSE

1116-1 was created to help developers find bugs and inaccuracies in their psychoacoustic models. Therefore, right, they don't need to extrapolate to the general public - they just need to detect the goddamned artifacts and report them to the devs. It was also used to help standards groups like the MPEG to discern among several competing technologies (yeah, right, as if the MPEG standards were selected on purely technical grounds).

The tests ff123, I and Sebastian conducted were not done to help developers. They were done to help users find out which codec sounds best in average. Therefore, yes, we need to at least try to extrapolate to the general public.


As for the second quote: sure, it would be great if we could have hundreds of test participants to obtain a good resolution. Will you find them for us?


I would suggest you stop quoting 1116-1 until you figure out what is going on here.

Ok. I give up trying to ask for post screening.
Acutaly your tests do help developers. Me in particular, because I am working on objective audio measurement.

I know they do. But it's not what they are meant for, unlike the ITU/EBU/ISO tests.

The interest in a 64 kbps test wasn't that high a few months ago so I hope things've changed a bit.

Last thing I would like to know before I can start the test is what samples to use. As mentioned earlier, I would like to use only 18 samples and not 20. This means that at least two samples used in the last test have to be removed - which ones? If more than two should be removed, what alternatives do you have in mind?

The contenders are:

Vorbis AoTuV 5 Beta
Nero HE-AAC
WMA Professional 10
iTunes LC-AAC @ 48 and @ 96 kbps for low and high anchor

WMA Pro is going to use CBR as used in the WMA vs. HE-AAC test Microsoft paid for.

why not using lame 3.88b3 at V5 (128kbits) as high anchor?

i think mp3 is the most used format and this also for the next years.

I didn't see the list of samples posted, but if there are any quasi-non-music samples, e.g. contrived problem samples or tricky electronic/synth, I would drop them first. It's useful information for perfecting and testing compression techniques, but I would guess it's misleading for the purpose of general mainstream use. Also, I assume there is at least one choral type. I also vote in favor of over-representing classical music (relative to popularity), on the snobbish grounds it's richer, more demanding, more emotionally complex, and otherwise just plain good for you.

p.s. I second the use of lame mp3 128kps -v5 as high anchor, unless there's some fear that will not be high enough for an anchor.

Sorry, the choice of contenders and anchors is over. 96 kbps iTunes should sound better than contenders at 64 kbps and therefore serves well as high anchor. MP3 at 128 kbps was already transparent. This is a good opportunity to also see how well iTunes performs at 96 kbps.

Well, I don't mean to spoil your enthusiasm, which I honestly appreciate, but I pass by the HA listening test forum now and again, curious of anyone coining a public multiformat 96k test.

Judging by the wiki, the 96k-ish bitrate range has never been thoroughly tested, apart from Guruboolez' fantastic private effort, albeit almost two years ago.

128 kbps has proven to be as good as transparent to most people. And imho, the difference between 64 and 48 kbps, the latter having been tested quite recently, cannot be expected to be spectacular, while near-to-transparent ranges such as 96 kbps are largely unexplored, and might therefore be far more interesting to test in the short term.

Just my 2 cents.

Edit: how about IgorC's recent personal multiformat 96k test? To his ears, iTunes and Nero AAC, WMA10pro and Ogg Vorbis Aotuv at 96 kbps are on par with or even marginally outperform LAME -V5. At over 30 kbps lower bitrate (i.e. 3/4), mind you. If that's not worth double-checking

IMHO, the difference between 48 kbps and 64 kbps is significant. For me, the 48 kbps samples in the last test were not good enough to be enjoyable (including Nero), but at 64 kbps Nero HE-AAC and Vorbis start to be usable (= easy to detect, but generally not too annoying). I have no experience of WMA Pro at 64 kbps.

However, I don't have personal interest in these bitrates at the moment. The situation could be different if I had a small mobile gadget with HE-AAC or Vorbis support.

@ Polar: if I recall correctly, Sebastian (or maybe someone else) did briefly mention doing a 80 or 96 kbps listening test after this one; however I could be wrong & I'm afraid I can't look in any depth now (I'm at work & it's getting late in my time-zone, sorry!).

Yes, the next tests will be 80 kbps multi-format which is going to feature both HE-AAC and LC-AAC along with some other popular codecs like Vorbis, WMA Professional 10 and maybe MP3, and MP3 @ 128 kbps.

If the next final Lame is out by then (3.98?), it would be good to check to see how it fares.

Basing on my personal 96k test http://www.hydrogenaudio.org/forums/index....showtopic=54967
I can conclude that many samples from previous public test are very well tuned by Nero devs for their AAC encoder as they are HA members. But when I continued to add more samples I saw the situation has changed.

I think it's logical to include only new and randomly choiced samples. The forum members will provide them without any delay.

I don't see how you can conclude this from your results. I think you saw the other encoders improve while Nero stayed at the same constant average?

I also don't see how this is even possible.

I'd like to exclude sample10 bibilolo and 13 aquatisme and change the length of sample 1 (43sec). The rest are fine to me.

menno told me that they are working on a new encoder and will probably release an unofficial version optimized for 64 kbps within the next days / weeks. I am going to wait until the beginning of the next month with the test and if the encoder is not out, I will use the currently available version. This doesn't mean that the samples discussion has to stop, so please give me feedback.

This intermediate version will be exactly the same as our next complete release for 64kbps. We will just disable other bitrates because it's simply not possible to do all tunings in the next 2 weeks. But for 64kbps we will not change the tuning anymore after putting up the binary for this test.
We did a lot of work on the new encoder the last few months, that require us to redo or check the tuning of all current psychoacoustic parameters.

The average perfomance of Nero on samples from previous tests is better than Itunes in my test but it's far from that for another part of my test. Here is excel table http://rapidshare.com/files/34474605/96kbps.xls.html

But the Nero results are pretty much constant and stable. You can pick almost any set of 15 samples from your test and Nero will have something like 4.4/4.3 on average. If you want to draw conclusions from this maybe they should go about iTunes and not about Nero.

For us it's no problem if other samples get selected for the test

IMHO those samples are interesting because they seem to be hard to encode. Why removing them?

Simply because it's not fun to listen that pattern of sound (at least for me).

I agree those 2 samples and sample like eig are good to measure some performance of lossy. But I think there are too much sampels from unusual music in recent test.

Like what? I think the three ones mentioned are the only unusual samples at all.

ALL BOLLOCKS!

This is the only listening test worth of your time:
http://www.audioasylum.com/cgi/vt.mpl?f=pcaudio&m=21002

...lol!

LOL rjamorim!!!!!!

Some people are true idiots.

I am sure he will make an ABX test between FLAC & APE very soon.

Yes, and I hope flac will win