Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multiformat 64 kbps Listening Test (Read 131656 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multiformat 64 kbps Listening Test

Based on the results of my last poll, the majority would like a 64 kbps multiformat test. This is also a great chance to see if Microsoft's test results (Nero HE-AAC vs. WMA Professional 10) were / are still valid.

I was thinking about using the same codecs as in the 48 kbps test with the exception of WMA Standard which can either be left out entirely or replaced by Winamp's HE-AAC for example. What do you guys think?
For those of you who don't know the last codecs tested:

Vorbis AoTuV 5 Beta
Nero HE-AAC
WMA Professional 10

iTunes LC-AAC @ 48 and @ 96 kbps for low and high anchor

I was now wondering about the settings to use for WMA Professional. If I recall correctly, it was possible to encode to 64 kbps VBR last time I checked, but looking at the test results Microsoft obtained, it seems that they used 1-pass CBR like in my 48 kbps test. So, if I have the option to encode to VBR, should I go with it or stick to what Microsoft recommended (CBR)? Personally, I would go with option #2. Maybe use both CBR and VBR in case nobody really wants Winamp HE-AAC.

The last test had 20 samples. In order to make things a bit easier for both testers and me, I would like to have a maximum of 18 samples this time. Therefore, I am also open for suggestions on what samples to remove or if you have any sample that is better than one used already, please tell.


Multiformat 64 kbps Listening Test

Reply #2


Welcome to the wonderful world of listening test feedback

Multiformat 64 kbps Listening Test

Reply #3
Your post isn't a day old yet and it's weekend, are you in a hurry, did you steal something? ;P

Seriously, I'd guess the knowledgeable people simply haven't seen the post yet.

I'm looking forward to the actual test, hope I get some stuff done until then, as it barely makes sense to take part with my current setup.


Multiformat 64 kbps Listening Test

Reply #5
I was thinking about using the same codecs as in the 48 kbps test with the exception of WMA Standard which can either be left out entirely or replaced by Winamp's HE-AAC for example. What do you guys think?

I'd prefer going for WMA Standard @Q25, because my personal experience tells me that this codec isn't uncommon being used on cheap low-memory flash players at this bitrate, at least those which don't support anything but MP3 and WMA. Besides, in a multiformat test I'd generally prefer testing entirely different codecs instead of just different implementations of the same format, like the mentioned HE-AAC ones. If we wanted to test the AAC codecs versus each other, then we could carry out a complete AAC test instead of a multiformat one, which was already done sometime in the last year.
Quote
So, if I have the option to encode to VBR, should I go with it or stick  to what Microsoft recommended (CBR)? Personally, I would go with option  #2. Maybe use both CBR and VBR in case nobody really wants Winamp  HE-AAC.

For practical reasons I'd go for VBR. To achieve the best possible quality people are usually recommended not to encode to CBR if there's no special utilisation like streaming via internet, though the VBR algorithm doesn't always implicitly deliver better results, as we already discussed in the 48 kbps test results topic.

Multiformat 64 kbps Listening Test

Reply #6
Well, I don't like to test too many codecs because is long and somewhat complicated.... said that, I would like to test only:

Vorbis AoTuV Beta5
Nero HE-AAC
WMA Professional 10

Something that I noted with WMA Pro 10 in Winamp encoder, is that in CBR the Sample Format is 16 bits using 44100 Hz and in VBR use 24 bits for 44100 Hz, because of that I think that VBR may have problems to convert 16 bits to 24 bits or something like that... and that the quality will be worse than CBR when used with 16 bits inputs (prety much all Audio CDs). Well this is with the Winamp encoder and with Windows Media Player I think is the same (it use CBR), not sure about any other WMA encoder...
JorSol
aoTuVb5 -q4

Multiformat 64 kbps Listening Test

Reply #7
Well, there also has to be a low and a high anchor, so we have at least 5 codecs.

As for the WMA CBR vs. VBR question, I think I will go with CBR since that is what Microsoft seems to recommend and it would be unfair to choose a different setting for the respective codec, while use recommended settings (by developers) for all other contenders. Also, like Junon stated, it might be wiser to either use a different format as fourth contender or leave it out entirely instead of testing another implementation of the same format. I don't know how interesting WMA Standard would be since it performed quite poorly in the 48 kbps.

Multiformat 64 kbps Listening Test

Reply #8
The last test had 20 samples. In order to make things a bit easier for both testers and me, I would like to have a maximum of 18 samples this time. Therefore, I am also open for suggestions on what samples to remove or if you have any sample that is better than one used already, please tell.


Maybe remove samples for which listeners have least interest, by checking number of results received for each sample in previous tests.
Maybe aditional criteria is to remove samples which are to hard to encode at low bitrates (like fatboy). It seems to me that there are some very hard to encode samples which were used in previous listening test at 48 kbps (those that got very low score).

Multiformat 64 kbps Listening Test

Reply #9
I'm pretty new to formal listening tests, so take my with a grain of salt, please.

- Fourth test subject: Agreed it should be a different format, but I don't have a preference of which one to add. 

- CBR/ABR/VBR: use developer recommendations.  That said, I'm curious if Microsoft recommends CBR on the assumption that people will choose this bitrate specifically for streaming.  I haven't read their literature on it.

- Samples to remove: The ones that most closely match the overall results from your 48kbps test. EDIT: In a sense, they have the least impact.

Multiformat 64 kbps Listening Test

Reply #10
- Samples to remove: The ones that most closely match the overall results from your 48kbps test. In a sense, they have the least impact.


I don't agree with this one since you can not predict if those samples will have the same effect at 64 kbps. But if there is a sample which in many tests don't reveal differences among encoders, then it should be removed.


Multiformat 64 kbps Listening Test

Reply #12
Well, I was expecting some feedback because it is weekend. Anyways, let's see how this turns out in a few days.
Which shows that most people accessed HA from their work... don't you guys have jobs to do?


Some of us are doing our job when we read HA

Quote
Something that I noted with WMA Pro 10 in Winamp encoder, is that in CBR the Sample Format is 16 bits using 44100 Hz and in VBR use 24 bits for 44100 Hz, because of that I think that VBR may have problems to convert 16 bits to 24 bits or something like that... and that the quality will be worse than CBR when used with 16 bits inputs (prety much all Audio CDs). Well this is with the Winamp encoder and with Windows Media Player I think is the same (it use CBR), not sure about any other WMA encoder...


I believe this has to do with the internal precision of the decoder.  It should not effect coding quality at all.

Multiformat 64 kbps Listening Test

Reply #13
Maybe it will be more fair to use a totaly new samples cause new encoders maye be highestly optimizied for the  samples from previous tests. Especially that ones from 48/80 kbps test and hard lame samples 

Multiformat 64 kbps Listening Test

Reply #14
I also feared this several times, but I don't think you can optimize several samples only. I mean, if you optimize for those samples, chances are good that those optimizations will affect other samples as well.

Multiformat 64 kbps Listening Test

Reply #15
I also feared this several times, but I don't think you can optimize several samples only. I mean, if you optimize for those samples, chances are good that those optimizations will affect other samples as well.


This is true if samples chosen for listening test are representative subset of all existing samples. And for listening test to be good, chosen samples should be representative subset.


Multiformat 64 kbps Listening Test

Reply #17
Let A represent set of all possible samples and B and C represent samples used in a listening test.
If B and C are representative subset of A (meaning that each of them cover all possible distortions that might appear in A due to encoding and that these distortions are evenly distributed) then it does not matter whether you choose B or C for a listening test.
And if some encoder is optimized for B, then it will also give good results for C.
But if you choose subset D which is not representative subset of A (it does not contain all distortions or some distortions are represented more than others) then encoder optimized for B might produce worse results for D.

I hope this did not introduce more confusion 

Basicaly diverse samples should be used for a listening test and some distortion should not be present much more than other distortions (i.e. not to many preecho problematic samples).

Multiformat 64 kbps Listening Test

Reply #18
Vorbis AoTuV 5 Beta
Nero HE-AAC
WMA Professional 10

My $0.02...
Also mp3 and lc-aac at 64kbps would be nice.
Personally, I couldn't care less about high/low anchors.

[offtopic]
work? 
[/offtopic]
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune


Multiformat 64 kbps Listening Test

Reply #20
Would it be possible to include LC AAC (nero encoder) as well? Since LC seems to be supported by more hardware devices, it might be more beneficial?


 

Multiformat 64 kbps Listening Test

Reply #22
I think the three mentioned in your first post sound like a great selection.
davidnaylor.org

Multiformat 64 kbps Listening Test

Reply #23
I think the three mentioned in your first post sound like a great selection.

I second that. 

Maybe a quick preliminary test to determine whether vbr or cbr is better for wmaPro is in order (much like the Nero AAC pre-test conducted before the 48kbps AAC test).
Copy Restriction, Annulment, & Protection = C.R.A.P. -Supacon

Multiformat 64 kbps Listening Test

Reply #24
I will go with CBR because it seems that CBR is recommended by Microsoft. They instructed NTSL to use CBR and not VBR, so I guess they have their reasons. Choosing VBR would go against the decision of using the recommended settings by the developers.

So you think a fourth contender would be too much?