64 kbps listening test 2005, Pre-test thread Options

[sehested]

Yet another sample suggestion:

Dire Straits - Iron Hand

[S_O]

The HE-AAC encoder from Helix Producer hasn't been improve since last 64kbps listening tests. There is only new thing : introduction of new target bitrates (32 & 48 kbps).

Huh? I cannot remember a listening test with Helix HE-AAC being tested. There were a 64kbps test with RealAudio Cook once, but with Helix HE-AAC? I cannot remember.

[rjamorim]

But if you need to run a separate test anyway then you might as well do it beforehand. Maybe there are serious flaws which could ruin Vorbis' reputation!

Yes, I believe it would be very bad if Vorbis reputation got ruined thanks to a badly tuned encoder.

Also, this test is only supposed to start after Apple releases iTunes 5 / QuickTime 7. That should give people plenty of time to conduce paralel tests, if there is really any interest...

The HE-AAC encoder from Helix Producer hasn't been improve since last 64kbps listening tests.

It wasn't even tested back then :B

[Acid Orange Juic...]

IMO this sample could be very useful for this test. In particular he cause a lot of phasing problems with lame --preset cbr 128 and Vorbis low bitrates.

Download HERE

[znode]

But if you need to run a separate test anyway then you might as well do it beforehand. Maybe there are serious flaws which could ruin Vorbis' reputation!

Yeah, misinterpretation of results is quite a problem. I've seen countless people claim vorbis to be always better than any other codec, in all cases, because of those http://www.rjamorim.com/test/multiformat128/results.html results at 128kbps.

[HotshotGG]

If wma would be left outside, half of wannabee slashdotters would be asking where is wma, the cd quality at 64 kbps codec. Were the HA Lame and mpc lovers afraid of it?

I don't know what to make of that site so I don't ask. Informative yes. Always on par with issues at hand no. Rating system nah For a site full of edcuated nerds they sure don't act like it sometimes .

IMO this sample could be very useful for this test. In particular he cause a lot of phasing problems with lame --preset cbr 128 and Vorbis low bitrates.

channel coupling related maybe? hmm I will have to test that out myself after.

Yeah, misinterpretation of results is quite a problem. I've seen countless people claim vorbis to be always better than any other codec, in all cases, because of those tests

Well it would be great two see both Nero HE-AAC and Vorbis tied for first ;-D. A streaming listening test was definitely going to be needed though eventually.

[kurtnoise]

The HE-AAC encoder from Helix Producer hasn't been improve since last 64kbps listening tests.

It wasn't even tested back then :B

ouups, sorry for the confusion... I was tired last night.



So, He-AAC encoder from Producer could be interesting though...

[Sebastian Mares]

Sample proposition: the beginning of "Money" by Pink Floyd.

I do not have it available, but I am sure some Pink Floyd fan could upload it.
Basically it is background music with coins and cash machine sounds. I think that the coins coud be interesting.

http://www.hydrogenaudio.org/forums/index....showtopic=32628

Anyone interested in Time?

This post has been edited by Sebastian Mares: Mar 23 2005, 10:45

[Latexxx]

Why not taking also 3gpp's CT HE + PS AAC?

[Sebastian Mares]

Why not taking also 3gpp's CT HE + PS AAC?

Wouldn't that be a HE-AAC 64 kbps listening test then?

[Gabriel]

All those propositions are transforming this into a 64kbps MPEG test...

[Sebastian Mares]

Anyways...

Regarding Vorbis, I would love if some Vorbis users could start a small listening test and compare AoTuV3 and Xiph 1.1 so that the better version will be used in this test.

As for the discussion about mp3PRO or ATRAC3+, I think that I will use ATRAC3+ since it is more wide-spread than mp3PRO and since mp3PRO didn't change since the last test.

Still not sure what to do with WMA - either Standard or Professional. I, personally, would choose Standard since it's the format you find in music stores and it's also what most people use so it's compatible with their players.

[moi]

I think you really need to include WMA Standard, as it is probably the most common format encoded at 64kbps, people will want to see how it compares with others in the test. Of course the newest, WMA 9.1, which is installed with WMP10.

You might also wish to include WMA Pro, to see how it compares at that bit rate to standard.

I really don't see why LAME at 128kbps should be included in a 64kbps listening test, as it was the other time. Probably has something to do with the claim that WMA at 64kbps sounds "as good as" MP3 at 128kbps. I don't think many here believe that claim. In any case, IMO, a 64kbps listening test should only include music encoded at 64kbps. It is misleading to encode 128kbps in one format, and 64kbps in all the others. Everything in a 64kbps listening test should be encoded at 64kbps.

I think MP3Pro should be included, as it did very well on some 64kbps tests in the past. Not supported by many players, but by some, I think it should be included, whether or not it changed.

[Redmond]

You might also wish to include WMA Pro, to see how it compares at that bit rate to standard.

"Publicly available" WMA Pro encoders do not go down to 64Kb/s stereo.

[Latexxx]

I really don't see why LAME at 128kbps should be  included in a 64kbps listening test, as it was the other time. Probably has something to do with the claim that WMA at 64kbps sounds "as good as" MP3 at 128kbps. I don't think many here believe that claim. In any case, IMO, a 64kbps listening test should only include music encoded at 64kbps. It is misleading to encode 128kbps in one format, and 64kbps in all the others. Everything in a 64kbps listening test should be encoded at 64kbps.

A credible listening test should have a low and high anchor.

[Aoyumi]

Regarding Vorbis, I would love if some Vorbis users could start a small listening test and compare AoTuV3 and Xiph 1.1 so that the better version will be used in this test.

When the test is performed, I need to submit the newest experiment version.
It is more clearly than aoTuV beta3 good with some samples (setting to the low bit rate).

[PoisonDan]

Regarding Vorbis, I would love if some Vorbis users could start a small listening test and compare AoTuV3 and Xiph 1.1 so that the better version will be used in this test.

When the test is performed, I need to submit the newest experiment version.
It is more clearly than aoTuV beta3 good with some samples (setting to the low bit rate).

Were you planning on releasing a new version soon anyway? I wouldn't want you to feel rushed to get a version out the door just to be in time for this listening test...

At this moment, I'm extremely busy with real-life and work-related stuff, but next week I'll probably have some time to do a few Vorbis listening tests...

[Sebastian Mares]

Well, take your time, since the test will start after Apple releases their HE-AAC encoder.

[rjamorim]

"Publicly available" WMA Pro encoders do not go down to 64Kb/s stereo.

If I remember correctly, the publicly available encoder stays around 64kbps if you choose the lowest VBR setting (10).

[Sebastian Mares]

Regarding the low anchor, do you think LAME or FhG should be used at 64 kbps?

[sehested]

Regarding the low anchor, do you think LAME or FhG should be used at 64 kbps?

I would like to see LAME as low anchor.

That would also demonstrate the improvements of the other codecs compared to the best MP3 encoder available.

[ff123]

Regarding the low anchor, do you think LAME or FhG should be used at 64 kbps?

I would like to see LAME as low anchor.

That would also demonstrate the improvements of the other codecs compared to the best MP3 encoder available.

This presumes that lame is the best mp3 encoder at 64 kbps, which isn't a given. The question of which mp3 encoder to use as a low anchor probably deserves a pre-test if people are interested in using the best-sounding one.

ff123

[guruboolez]

Two suggestions:

• I think that the current scale isn't really really suited to a 64 kbps and the expected distortions.
Artifacts “perceptible but not annoying” (4.0) are maybe not very common at this bitrate. And few encoders are able to reproduce (in my opinion) a sound with only "slightly annoying" (3.0) difference at 64 kbps. It's possible to change the corresponding scale with schnofler's abc/hr, and I wonder if it's not worth to think about it. If I remember correctly, the average notation I gave to most encoders during the 32 kbps was inferior to 1.5/5


• A also suggest to reduce the length of all samples. I'm the first one to provide 30 seconds samples, but I perfectly know the drawbacks. Some people will rate one encoder on a short range located at the beginning, some other will evaluate another part (totally different from the first one), etc... Finally it's exactly if people have evaluate different samples. I suggest to limit the duration to 6 or 7 seconds.

[Sebastian Mares]

• A also suggest to reduce the length of all samples. I'm the first one to provide 30 seconds samples, but I perfectly know the drawbacks. Some people will rate one encoder on a short range located at the beginning, some other will evaluate another part (totally different from the first one), etc... Finally it's exactly if people have evaluate different samples. I suggest to limit the duration to 6 or 7 seconds.

I understand what you mean, but why not let testers decide which portion they want to ABX?

This post has been edited by Sebastian Mares: Mar 24 2005, 13:09

[Gabriel]

I also think that 30s might be too long.
Perhaps 6s is too short, but I think that 15s should be enough.

Letting testers deciding which portion to use is perhaps reducing "usefullness" of results. It is like they are testing different samples, but it makes correlation between results for the same sample harder.

If a sample has some quite different parts in a 30s set, then it could be intersting to split it into 2 samples, making interpretation of results easier.