IPB

Welcome Guest ( Log In | Register )

6 Pages V   1 2 3 > »   
Reply to this topicStart new topic
Multiformat 64 kbps Listening Test, Pre-Test Discussion
Sebastian Mares
post Mar 2 2007, 17:45
Post #1





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Based on the results of my last poll, the majority would like a 64 kbps multiformat test. This is also a great chance to see if Microsoft's test results (Nero HE-AAC vs. WMA Professional 10) were / are still valid.

I was thinking about using the same codecs as in the 48 kbps test with the exception of WMA Standard which can either be left out entirely or replaced by Winamp's HE-AAC for example. What do you guys think?
For those of you who don't know the last codecs tested:

Vorbis AoTuV 5 Beta
Nero HE-AAC
WMA Professional 10

iTunes LC-AAC @ 48 and @ 96 kbps for low and high anchor

I was now wondering about the settings to use for WMA Professional. If I recall correctly, it was possible to encode to 64 kbps VBR last time I checked, but looking at the test results Microsoft obtained, it seems that they used 1-pass CBR like in my 48 kbps test. So, if I have the option to encode to VBR, should I go with it or stick to what Microsoft recommended (CBR)? Personally, I would go with option #2. Maybe use both CBR and VBR in case nobody really wants Winamp HE-AAC.

The last test had 20 samples. In order to make things a bit easier for both testers and me, I would like to have a maximum of 18 samples this time. Therefore, I am also open for suggestions on what samples to remove or if you have any sample that is better than one used already, please tell.

This post has been edited by Sebastian Mares: Mar 2 2007, 17:50


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 3 2007, 16:00
Post #2





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Wow, look at the countless replies I've received!


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
rjamorim
post Mar 3 2007, 16:09
Post #3


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



biggrin.gif

Welcome to the wonderful world of listening test feedback


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Hollunder
post Mar 3 2007, 16:10
Post #4





Group: Members
Posts: 152
Joined: 1-February 06
From: Austria
Member No.: 27471



Your post isn't a day old yet and it's weekend, are you in a hurry, did you steal something? ;P

Seriously, I'd guess the knowledgeable people simply haven't seen the post yet.

I'm looking forward to the actual test, hope I get some stuff done until then, as it barely makes sense to take part with my current setup.
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 3 2007, 16:39
Post #5





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Well, I was expecting some feedback because it is weekend. Anyways, let's see how this turns out in a few days.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Junon
post Mar 3 2007, 17:21
Post #6





Group: Members
Posts: 520
Joined: 27-August 06
From: Germany
Member No.: 34518



QUOTE (Sebastian Mares @ Mar 2 2007, 17:45) *
I was thinking about using the same codecs as in the 48 kbps test with the exception of WMA Standard which can either be left out entirely or replaced by Winamp's HE-AAC for example. What do you guys think?

I'd prefer going for WMA Standard @Q25, because my personal experience tells me that this codec isn't uncommon being used on cheap low-memory flash players at this bitrate, at least those which don't support anything but MP3 and WMA. Besides, in a multiformat test I'd generally prefer testing entirely different codecs instead of just different implementations of the same format, like the mentioned HE-AAC ones. If we wanted to test the AAC codecs versus each other, then we could carry out a complete AAC test instead of a multiformat one, which was already done sometime in the last year.
QUOTE
So, if I have the option to encode to VBR, should I go with it or stick to what Microsoft recommended (CBR)? Personally, I would go with option #2. Maybe use both CBR and VBR in case nobody really wants Winamp HE-AAC.

For practical reasons I'd go for VBR. To achieve the best possible quality people are usually recommended not to encode to CBR if there's no special utilisation like streaming via internet, though the VBR algorithm doesn't always implicitly deliver better results, as we already discussed in the 48 kbps test results topic.
Go to the top of the page
+Quote Post
jorsol
post Mar 3 2007, 18:03
Post #7





Group: Members
Posts: 89
Joined: 7-March 05
From: Managua
Member No.: 20434



Well, I don't like to test too many codecs because is long and somewhat complicated.... said that, I would like to test only:

Vorbis AoTuV Beta5
Nero HE-AAC
WMA Professional 10

Something that I noted with WMA Pro 10 in Winamp encoder, is that in CBR the Sample Format is 16 bits using 44100 Hz and in VBR use 24 bits for 44100 Hz, because of that I think that VBR may have problems to convert 16 bits to 24 bits or something like that... and that the quality will be worse than CBR when used with 16 bits inputs (prety much all Audio CDs). Well this is with the Winamp encoder and with Windows Media Player I think is the same (it use CBR), not sure about any other WMA encoder...

This post has been edited by jorsol: Mar 3 2007, 18:08


--------------------
JorSol
aoTuVb5 -q4
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 4 2007, 23:12
Post #8





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Well, there also has to be a low and a high anchor, so we have at least 5 codecs.

As for the WMA CBR vs. VBR question, I think I will go with CBR since that is what Microsoft seems to recommend and it would be unfair to choose a different setting for the respective codec, while use recommended settings (by developers) for all other contenders. Also, like Junon stated, it might be wiser to either use a different format as fourth contender or leave it out entirely instead of testing another implementation of the same format. I don't know how interesting WMA Standard would be since it performed quite poorly in the 48 kbps.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
muaddib
post Mar 5 2007, 14:33
Post #9





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



QUOTE (Sebastian Mares @ Mar 2 2007, 17:45) *
The last test had 20 samples. In order to make things a bit easier for both testers and me, I would like to have a maximum of 18 samples this time. Therefore, I am also open for suggestions on what samples to remove or if you have any sample that is better than one used already, please tell.


Maybe remove samples for which listeners have least interest, by checking number of results received for each sample in previous tests.
Maybe aditional criteria is to remove samples which are to hard to encode at low bitrates (like fatboy). It seems to me that there are some very hard to encode samples which were used in previous listening test at 48 kbps (those that got very low score).
Go to the top of the page
+Quote Post
sketchy_c
post Mar 5 2007, 14:58
Post #10





Group: Members
Posts: 105
Joined: 18-December 06
Member No.: 38800



I'm pretty new to formal listening tests, so take my with a grain of salt, please.

- Fourth test subject: Agreed it should be a different format, but I don't have a preference of which one to add.

- CBR/ABR/VBR: use developer recommendations. That said, I'm curious if Microsoft recommends CBR on the assumption that people will choose this bitrate specifically for streaming. I haven't read their literature on it.

- Samples to remove: The ones that most closely match the overall results from your 48kbps test. EDIT: In a sense, they have the least impact.

This post has been edited by sketchy_c: Mar 5 2007, 15:00
Go to the top of the page
+Quote Post
muaddib
post Mar 5 2007, 15:06
Post #11





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



QUOTE (sketchy_c @ Mar 5 2007, 14:58) *
- Samples to remove: The ones that most closely match the overall results from your 48kbps test. In a sense, they have the least impact.


I don't agree with this one since you can not predict if those samples will have the same effect at 64 kbps. But if there is a sample which in many tests don't reveal differences among encoders, then it should be removed.
Go to the top of the page
+Quote Post
pepoluan
post Mar 5 2007, 17:07
Post #12





Group: Members
Posts: 1455
Joined: 22-November 05
From: Jakarta
Member No.: 25929



QUOTE (Sebastian Mares @ Mar 3 2007, 22:39) *
Well, I was expecting some feedback because it is weekend. Anyways, let's see how this turns out in a few days.
Which shows that most people accessed HA from their work... don't you guys have jobs to do? tongue.gif


--------------------
Nobody is Perfect.
I am Nobody.

http://pandu.poluan.info
Go to the top of the page
+Quote Post
benski
post Mar 5 2007, 18:12
Post #13


Winamp Developer


Group: Developer
Posts: 669
Joined: 17-July 05
From: Ashburn, VA
Member No.: 23375



QUOTE (pepoluan @ Mar 5 2007, 11:07) *
QUOTE (Sebastian Mares @ Mar 3 2007, 22:39) *
Well, I was expecting some feedback because it is weekend. Anyways, let's see how this turns out in a few days.
Which shows that most people accessed HA from their work... don't you guys have jobs to do? tongue.gif


Some of us are doing our job when we read HA

QUOTE
Something that I noted with WMA Pro 10 in Winamp encoder, is that in CBR the Sample Format is 16 bits using 44100 Hz and in VBR use 24 bits for 44100 Hz, because of that I think that VBR may have problems to convert 16 bits to 24 bits or something like that... and that the quality will be worse than CBR when used with 16 bits inputs (prety much all Audio CDs). Well this is with the Winamp encoder and with Windows Media Player I think is the same (it use CBR), not sure about any other WMA encoder...


I believe this has to do with the internal precision of the decoder. It should not effect coding quality at all.

This post has been edited by benski: Mar 5 2007, 18:14
Go to the top of the page
+Quote Post
IgorC
post Mar 5 2007, 22:40
Post #14





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



Maybe it will be more fair to use a totaly new samples cause new encoders maye be highestly optimizied for the samples from previous tests. Especially that ones from 48/80 kbps test and hard lame samples wink.gif
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 6 2007, 07:05
Post #15





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



I also feared this several times, but I don't think you can optimize several samples only. I mean, if you optimize for those samples, chances are good that those optimizations will affect other samples as well.

This post has been edited by Sebastian Mares: Mar 6 2007, 07:05


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
muaddib
post Mar 6 2007, 09:42
Post #16





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



QUOTE (Sebastian Mares @ Mar 6 2007, 07:05) *
I also feared this several times, but I don't think you can optimize several samples only. I mean, if you optimize for those samples, chances are good that those optimizations will affect other samples as well.


This is true if samples chosen for listening test are representative subset of all existing samples. And for listening test to be good, chosen samples should be representative subset.
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 6 2007, 14:10
Post #17





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



I don't get your point. The last samples used are a mixture of all important music genres and also some difficult samples.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
muaddib
post Mar 6 2007, 15:30
Post #18





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



Let A represent set of all possible samples and B and C represent samples used in a listening test.
If B and C are representative subset of A (meaning that each of them cover all possible distortions that might appear in A due to encoding and that these distortions are evenly distributed) then it does not matter whether you choose B or C for a listening test.
And if some encoder is optimized for B, then it will also give good results for C.
But if you choose subset D which is not representative subset of A (it does not contain all distortions or some distortions are represented more than others) then encoder optimized for B might produce worse results for D.

I hope this did not introduce more confusion tongue.gif

Basicaly diverse samples should be used for a listening test and some distortion should not be present much more than other distortions (i.e. not to many preecho problematic samples).
Go to the top of the page
+Quote Post
gameplaya15143
post Mar 8 2007, 19:01
Post #19





Group: Members
Posts: 484
Joined: 8-January 06
From: Earth
Member No.: 26978



QUOTE (Sebastian Mares @ Mar 2 2007, 11:45) *
Vorbis AoTuV 5 Beta
Nero HE-AAC
WMA Professional 10

My $0.02...
Also mp3 and lc-aac at 64kbps would be nice.
Personally, I couldn't care less about high/low anchors.

[offtopic]
work? huh.gif
[/offtopic]

This post has been edited by gameplaya15143: Mar 8 2007, 19:04


--------------------
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 9 2007, 07:11
Post #20





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Anchors have to be there regardless if you care about that or not.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
kanak
post Mar 9 2007, 07:13
Post #21





Group: Members
Posts: 1190
Joined: 12-January 06
From: Cambridge, MA
Member No.: 27052



Would it be possible to include LC AAC (nero encoder) as well? Since LC seems to be supported by more hardware devices, it might be more beneficial?
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 9 2007, 07:40
Post #22





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



At 64? I would rather use LC for the next 80 kbps test.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
naylor83
post Mar 9 2007, 11:40
Post #23





Group: Members
Posts: 204
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



I think the three mentioned in your first post sound like a great selection.


--------------------
davidnaylor.org

Vorbis Q4, please. AoTuv b5, preferably.
Go to the top of the page
+Quote Post
The Sheep of DEA...
post Mar 10 2007, 04:37
Post #24





Group: Developer
Posts: 174
Joined: 16-April 06
Member No.: 29596



QUOTE (naylor83 @ Mar 9 2007, 04:40) *
I think the three mentioned in your first post sound like a great selection.

I second that.

Maybe a quick preliminary test to determine whether vbr or cbr is better for wmaPro is in order (much like the Nero AAC pre-test conducted before the 48kbps AAC test).


--------------------
Copy Restriction, Annulment, & Protection = C.R.A.P. -Supacon
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 10 2007, 07:33
Post #25





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



I will go with CBR because it seems that CBR is recommended by Microsoft. They instructed NTSL to use CBR and not VBR, so I guess they have their reasons. Choosing VBR would go against the decision of using the recommended settings by the developers.

So you think a fourth contender would be too much?


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post

6 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 18th April 2014 - 19:11