Multiformat 128 kbps Listening Test, Pre-Test Discussion Options

[naylor83]

Two more things... Seing that the bitrate is around 140 kbps, I think I can tell Vorbis to encode to 4.2 or 4.25 (what do you guys think - which one to go with?).
Also, I will cut the voice from Elizabeth since it's a bit "confusing".

...But, shouldn't the settings be tuned for a target bitrate on a large music corpus, and *not* for the sample corpus?

[Sebastian Mares]

Two more things... Seing that the bitrate is around 140 kbps, I think I can tell Vorbis to encode to 4.2 or 4.25 (what do you guys think - which one to go with?).
Also, I will cut the voice from Elizabeth since it's a bit "confusing".

...But, shouldn't the settings be tuned for a target bitrate on a large music corpus, and *not* for the sample corpus?

Yes, but the difference between the encoders shouldn't be more than 10% so that's why I think 4.25 can still be used. I mixed that up yesterday evening when I thought that the bitrate must not deviate more than 10% from the target bitrate of the listening test. My bad.

This post has been edited by Sebastian Mares: Dec 1 2005, 17:52

[Halcyon]

You can safely skip this question - unless you are interested in statistics (and conducting your own test in the future), so that I won't anger Triza again

...But, shouldn't the settings be tuned for a target bitrate on a large music corpus, and *not* for the sample corpus?

Yes, unless the sample is known to represent the population under study, in which case both should result in the same settings. However, being that the population is not clearly defined...

This of course, combined with what Guruboolez has already posted about bit-rate estimation methods and their related accuracies.

However, as it is a relatively big undertaking (esp. proving it), I think people should be relatively happy with a non-random sampling. Esp. when manually picked to include various tracks of different bit-rate averages. Good enough.

[ChiGung]

2-pass encoding: why it is necessary to extract samples from an entire encoding and the original track.....

For those still interested, a reply discussing these results and suggestions of how 2pass method can be more fairly employed here (in case its missed).

[Sebastian Mares]

Updated bitrate table with Vorbis -q 4.25: http://maresweb.de/bitrates2.htm

[naylor83]

Two more things... Seing that the bitrate is around 140 kbps, I think I can tell Vorbis to encode to 4.2 or 4.25 (what do you guys think - which one to go with?).
Also, I will cut the voice from Elizabeth since it's a bit "confusing".

...But, shouldn't the settings be tuned for a target bitrate on a large music corpus, and *not* for the sample corpus?

Yes, but the difference between the encoders shouldn't be more than 10% so that's why I think 4.25 can still be used. I mixed that up yesterday evening when I thought that the bitrate must not deviate more than 10% from the target bitrate of the listening test. My bad.

Ok. Well, as long as you're tweaking it and looking at the effects on a more "general" avg bitrate. (And you are, if I understand you correctly.)

[Sebastian Mares]

I didn't understand what you mean. What I wanted to say is that if -q 4.25 produces a bitrate that is more than 10% away from the other competitors, I have to replace a sample. So far, everything looks good, I think.

[ChiGung]

If its fair to say that encodes which result in an unusualy high or low bitrate compared to the others are examples of samples which the encoder concernsed finds more unusual ~in a way than the others do..... then there is a possibility that the encoder would be selectively advantaged by discarding them

[Sebastian Mares]

Yes, but if the difference is too high, you cannot compare the two encoders.

[ChiGung]

Yes, but if the difference is too high, you cannot compare the two encoders.

Ah I see what you mean -for the mean bitrate over the whole corpus rather than individual samples.
Whats odd is if the mean corpus bitrate is far off for one encoder, then there is something about the corpus that that encoder finds unusual - assuming the corpus is large enough to average out the random differences. hmmm...

[Sebastian Mares]

Yes, but if the difference is too high, you cannot compare the two encoders.

Ah I see what you mean -for the mean bitrate over the whole corpus rather than individual samples.

Exactly.

[ChiGung]

Yes, but if the difference is too high, you cannot compare the two encoders.

Ah I see what you mean -for the mean bitrate over the whole corpus rather than individual samples.

Exactly.

How about this, if all the encoders are close to 128 kbs for a large target corpus, and for the listening corpus are all somewhat above it, (resulting from a correlation between perceptual listening interest and codec-wide demand factor) Some codecs average behaviour might be to stick tighter to the target bitrate than others, so those codecs would produce encodes closer to the target bitrate than others. One codec that 'sticks out' of line with the listening corpus, might be just displaying its normal behaviour in that while it achieves the target over normal average material, its choosen discreet allocation varys more widely than the others, it might look like its unusualy high compared to the others, but if the others are all higher than the target over average, it might just be displaying more 'reactivity' to demand factor, and manipulating the samples to deliberately lower bit allocation for it alone, could be distortive to its performance.

Im not sure yet, but the way to avoid such uncertainty of fairness, could be to try to make *all codecs meet a target bit allocation mean, accross the targeting material AND the sample material.

An axiom that Id hold onto, is the more samples used the less danger of random variations surviving in the result,
and the less reactive manipulation to the samples, the less danger of affecting a codec unfairly.

(???)

This post has been edited by ChiGung: Dec 1 2005, 20:04

[Alex B]

A couple of comments about the electronic genre samples:

- If I happen to have the same 29 s version of the "Kraftwerk" sample (The Robots) it has an obvious background hiss. (I downloaded my sample a long time ago from a HA link.) Is it a vinyl recording?

- The "Electronic" sample I provided is a bit short. I would give the testers a bit longer time to adopt. The sample is quite extraordinary.

I have a representative collection of electronic music and I would like to propose a couple of new samples, but I don't have time to do it immediately. (Would tomorrow be OK?)

For example: A bit longer "Yello" sample and a sample from the new live version of "Die Roboter" from the latest Kraftwerk album. You may also want to consider selecting only one electronic sample. I may be able to find the strong and shrill female voice I requested earlier, which could be used instead one of the electronic samples.

This post has been edited by Alex B: Dec 1 2005, 20:17

[[JAZ]]

An axiom that Id hold onto, [...]

I thought I would never see this word out of an algebraic book...

[Sebastian Mares]

- If I happen to have the same 29 s version of the "Kraftwerk" sample (The Robots) it has an obvious background hiss. (I downloaded my sample a long time ago from a HA link.) Is it a vinyl recording?

I have no idea where the sample comes from - I have it from Roberto's collection.

I have a representative collection of electronic music and I would like to propose a couple of new samples, but I don't have time to do it immediately. (Would tomorrow be OK?)

Well, I would like to complete the sample set tomorrow morning, clear up some final stuff regarding settings and then prepare the listening test so that everything is ready on 3rd.

For example: A bit longer "Yello" sample and a sample from the new live version of "Die Roboter" from the latest Kraftwerk album. You may also want to consider selecting only one electronic sample. I may be able to find the strong and shrill female voice I requested earlier, which could be used instead one of the electronic samples.

Well, I agree that 3 electronic samples are a bit much, but they are different. The Sash sample is a mixture of Pop and Trance and it's also quite difficult (1012 kbps WavPack). Kraftwerk is pure techno and it's not very "wild". The sample you posted is pretty aggressive and IMO, the 10 seconds I have are enough.

Edit: You could post the female voice sample and I'll have a look at it. Eventually, I might replace kraftwerk and Yello and use ravel and your female sample.

This post has been edited by Sebastian Mares: Dec 1 2005, 21:06

[yahknow1]

I'm new to this (still) and excuse this question if it silly or I'm jumping ahead of things, but, when I try saving the clips, they all come up as "index.php"? What am I doing wrong?

Which browser are you using?

IE 6.0...never ran into troubles like this with any other sample?

Did you try just left-clicking, instead of right-click -> save as?

I've tried both: 1) Left-click gets me the "save file" window. The file that results is labeled "index.php"(or whatever I decide to name it) and after saving it, nothing will open the file

2) "Right-click" gets me the "Smart-Window" which has a few options:

"Open"
"Open in new window"
"Save Target as..."
"Print Target"
"Copy Shortcut"
"Add To Favorites" &
"Properties"

I've tried all these that could logically be tried, in every way I can imagine? I've even downloaded and installed "Wavpack" to see if the file is a type my machine doesn't recognize, but still nothing?

I tried looking at the "Properties" and it gives me a little information about the link itself, they are:

"Address URL:"
http://www.hydrogenaudio.org/forums/index....pe=post&id=1833

AND

"Type:"
PHP?ACT=ATTACH&TYPE=POST&ID=1833 File

I'm hoping that one of you sharper peeps can tell me something else to try?

Any info you could give would be greatly appreciated...Thanks!

[Alex B]

Well, I agree that 3 electronic samples are a bit much, but they are different. The Sash sample is a mixture of Pop and Trance and it's also quite difficult (1012 kbps WavPack). Kraftwerk is pure techno and it's not very "wild". The sample you posted is pretty aggressive and IMO, the 10 seconds I have are enough.

OK, I am just proposing things.

The hiss I mentioned is present in the both old Kraftwerk samples (4 s and 29 s at http://ff123.net/samples.html). I suppose the samples are OK if the hiss has not bothered anyone before. Perhaps it is just a part of the old recording.

Edit: You could post the female voice sample and I'll have a look at it. Eventually, I might replace kraftwerk and Yello and use ravel and your female sample.

I'll browse through my archives and hopefully I'll find her.
I have a couple of artists in my mind.

This post has been edited by Alex B: Dec 1 2005, 21:49

[naylor83]

Any info you could give would be greatly appreciated...Thanks!

Well, for a start - have you tried Firefox? (In general I mean, not specifically for this task...) If you haven't I seriously would recommend getting it. They've just launched version 1.5 at http://mozilla.com yesterday. (The links to the samples worked in my Firefox anyway. No reason for them not to work in IE either, really...)

Edit: mistakes

This post has been edited by naylor83: Dec 1 2005, 21:48

[sehested]

Edit: You could post the female voice sample and I'll have a look at it. Eventually, I might replace kraftwerk and Yello and use ravel and your female sample.

I'll browse through my archives and hopefully I'll find her.
I have a couple of artists in my mind.

@ Alex B: What did you think about this one:
Cæcilie Norby - Life on Mars

[sehested]

Updated bitrate table with Vorbis -q 4.25: http://maresweb.de/bitrates2.htm

I think that it would be more fair to use a lower quality setting for Vorbis.

Since the bit rate control of Vorbis is very flexible it seems wrong to select a setting that will give it the highest average bitrate of all encoders. People could challenge such a decision as favoritism towards Vorbis.

At -q 4 Vorbis has an average bitrate similar to that of iTunes and Nero and that would in my opinion be the right setting.

[yahknow1]

Any info you could give would be greatly appreciated...Thanks!

Well, for a start - have you tried Firefox? (In general I mean, not specifically for this task...) If you haven't I seriously would recommend getting it. They've just launched version 1.5 at http://mozilla.com yesterday. (The links to the samples worked in my Firefox anyway. No reason for them not to work in IE either, really...)

Edit: mistakes

Yes Naylor, I've used Firefox before. The only thing I don't like about it is no sound, so I stick with IE, no big deal though.

I think somethings just "not in the cards", when it comes to me participating in these public listening tests....I seem to get these strange technical dificulties just when I'd like to try?

[Sebastian Mares]

Updated bitrate table with Vorbis -q 4.25: http://maresweb.de/bitrates2.htm

I think that it would be more fair to use a lower quality setting for Vorbis.

Since the bit rate control of Vorbis is very flexible it seems wrong to select a setting that will give it the highest average bitrate of all encoders. People could challenge such a decision as favoritism towards Vorbis.

At -q 4 Vorbis has an average bitrate similar to that of iTunes and Nero and that would in my opinion be the right setting.

According to this post:

I updated my bitrate table with Vorbis -q 4.20 and gathered all previous results in the same table:

bitrates_public2.xls

Vorbis -q 4 reached an average of exactly 128 kbps on Alex B's side. On the other hand, -q 4.2 and -q 4.25 didn't produce too high bitrates (around 134 kbps which is OK). So, what do the experts thing - should -q 4 be used or -q 4.2(5)?

[Sebastian Mares]

I am going to replace Kraftwerk with Ravel since 3 electronica are really too much.

[naylor83]

Vorbis -q 4 reached an average of exactly 128 kbps on Alex B's side. On the other hand, -q 4.2 and -q 4.25 didn't produce too high bitrates (around 134 kbps which is OK). So, what do the experts thing - should -q 4 be used or -q 4.2(5)?

Well, my spontaneous reaction is that you should keep it at q4, since 128 kbps was the target of this test - at least to begin with

[naylor83]

Yes Naylor, I've used Firefox before. The only thing I don't like about it is no sound, so I stick with IE, no big deal though.

(OT) You could try this extension:
https://addons.mozilla.org/extensions/morei...ication=firefox

It'll import the default windows sounds into Firefox. (Most noticably, the 'click'.)

I think somethings just "not in the cards", when it comes to me participating in these public listening tests....I seem to get these strange technical dificulties just when I'd like to try?

Of course you should participate. It's my first test too

This post has been edited by naylor83: Dec 1 2005, 23:34