128kbps Extension Test - OPEN Options

[rjamorim]

Hello.

I'd like to announce the start of the 128kbps extension test, comparing the AAC winner (QuickTime) to Musepack, Vorbis, WMA Pro and Lame MP3.

Instructions are available at the announcement page:
http://audio.ciara.us/test/128extension/pr...esentation.html

If questions or issues arise, please post at this thread.

Thank-you.

Regards;

Roberto Amorim

This post has been edited by rjamorim: Jul 29 2003, 18:02

[Canar]

Any .torrents going to be made available this time?

And should this test be advertised to the broader community? (ie. Slashdot, Kuro5hin, etc)

I think it should be mentioned that you're comparing to Blade MP3 as well, even though it's meant to show just how bad a psychoacoustic codec can be. That way it gives people a sense of why the 128kbps MP3s they downloaded really suck.

This post has been edited by Canar: Jul 24 2003, 02:55

[rjamorim]

Any .torrents going to be made available this time?

Probably, yes.

But Dibrom has been too busy lately, and he plans to create the torrents after HA moves to another server.

And should this test be advertised to the broader community? (ie. Slashdot, Kuro5hin, etc)

Yes, it would be very welcome.

Unfortunately, I'm not a member of Slashdot or Kuro5hin, so I don't know about article submissions. I would be very grateful if some insider can post about this test there, pointing to the presentation page.

I think it should be mentioned that you're comparing to Blade MP3 as well, even though it's meant to show just how bad a psychoacoustic codec can be. That way it gives people a sense of why the 128kbps MP3s they downloaded really suck.

It will only be mentioned at the results page

That way, people won't be confused, since it's there for perspective purposes, not really to be compared.

Regards;

Roberto.

This post has been edited by rjamorim: Jul 24 2003, 03:10

[dev0]

Just submitted it to slashdot.

[ff123]

Posted to newsgroups:

rec.audio.opinion
rec.audio.high-end
alt.music.mp3, rec.audio.misc, uk.rec.audio

ff123

[Artemis3]

Made a little comment at Ars Technica.

[Cobra]

How it`s done? All files are VBR but calculated averange 128kbps? So you encoed each file in each format in VBR mode unless acheving e.g. 127-129 kbps averange? IMO it`s best choice to test >>>128kbps VBR<<< files.

EDIT:
"Ogg Vorbis 1.0 post-CVS -q 4.25" - it`s not `128kbps every time, you can`t say that it is TRUE 128kbps test!

This post has been edited by Cobra: Jul 24 2003, 09:04

[ilikedirtthe2nd]

see this threat:

http://www.hydrogenaudio.org/forums/index....showtopic=11134

regards; ilikedirt

[Cobra]

ogg q 4.25

122.70901287553649, 131.1936462778568, 123.71958369470946, 126.4998828125, 124.25014547160957, 135.24413014700758, 125.75588978120831, 130.33543256174835, 128.77903642773208, 121.00131765659843
127.599113763

- these samples (exept one) are not in 128kbps...

[ilikedirtthe2nd]

ogg q 4.25

122.70901287553649, 131.1936462778568, 123.71958369470946, 126.4998828125, 124.25014547160957, 135.24413014700758, 125.75588978120831, 130.33543256174835, 128.77903642773208, 121.00131765659843
127.599113763

- these samples (exept one) are not in 128kbps...

nobody said that, but their average bitrates are closesed to 128 from the tested ogg settings (~127).

i think this method was chosen, because vbr encoders get used this way (through quality settings). usually nobody would encode a files several times with vbr to get the bitrate closest to 128kb

regards; ilikedirt

[Cobra]

Of course nobody will encode in such way! But we test _128kbps_ files. After test we can say that in for 128kbps for samples X codec Y was better. What if WMA will have 130 kbps and Vorbis 110 kbps?

I just want to say that normal users behaviour is completely not important in 128kbps test.

[ilikedirtthe2nd]

of course i see your point here. maybe rjamorim can tell you why he decided to test this way...

regards; ilikedirt

[Cobra]

I downloaded 12 files with test samples. Two questions: 1. why it`s not a blind test? 2. Why only original, mp4 and wma files in packages?

[Gabriel]

Of course nobody will encode in such way! But we test _128kbps_ files. After test we can say that in for 128kbps for samples X codec Y was better. What if WMA will have 130 kbps and Vorbis 110 kbps?

What is important is that the overall bitrate is about the same for every codec. If for a specific track codec A is using an average bitrate of 130 and codec B an average bitrate of 110, it is just the decision made by the codec. It is programmed this way, an no one manually forced codec A to use an higher bitrate.

Manually forcing each codec to reach exactly 128kbps for each sample would be unfair, as the codecs are not designed to exhibit such a behaviour. What is important is the overall bitrate.

This post has been edited by Gabriel: Jul 24 2003, 09:53

[Mac]

As Gabriel points out - this way you are also testing a codecs ability to judge how hard a sample is. If Vorbis only gives the sample 110kbs and ends up sounding bad because of it - that is the fault of Vorbis's VBR handling, and so it should be punished

As far as I can tell, you can see which files are which because you load them into ABC-HR - which does all the blinding for you Otherwise stick a biro in your eyes and you will be blind! (joke)

[Volcano]

I downloaded 12 files with test samples. Two questions: 1. why it`s not a blind test? 2. Why only original, mp4 and wma files in packages?

Have you looked at the contents of the zip archives, or even read the readme?

The abc-hr_bin.zip package includes various CLI en-/decoders - and each sample package comes with a batch file which uses these CLI en-/decoders to convert the supplied FLAC file to a lossy format (and decode back to WAV again for ABC/HR to accept it). For those formats where this isn't possible because of the lack of CLI encoders (WMA, MP4), readily encoded files are supplied. This method saves loads of bandwidth.

[Daybreak]

Okay I'm new to this thing....


Does using the ABX tool and comparing the original and sample x repeatedly before giving an actual rating skew the results?


Or are we supposed to simply listen from the rating table itself?

[bond]

just one thing to say:

thanks a lot for this great test, rjamorim!!!

[AstralStorm]

Two tests done... death2 and waiting.

/EDIT\
No, comparing two samples before deciding on the result doesn't skew the results.

ABX testing proves your results, especially if the difference is subtle.
\EDIT/

This post has been edited by AstralStorm: Jul 24 2003, 12:03

[Daybreak]

Another question on this testing thing ...

Rating 1-5 is against the original file right? Not against each other right? So assuming two codecs sound the same to you in comparision with the original file, they would rate about the same score? ( that's what I get from reading the Pratice With ABC/HR page )

Or is scoring done differently?

Also, are comments absolutely necessary?

[AstralStorm]

The tests are rating against original.

Comments aren't necessary, but useful for others.

This post has been edited by AstralStorm: Jul 24 2003, 12:48

[rjamorim]

Just submitted it to slashdot.

Posted to newsgroups:

rec.audio.opinion
rec.audio.high-end
alt.music.mp3, rec.audio.misc, uk.rec.audio

Made a little comment at Ars Technica.

Thanks a lot!

And thanks to everybody that is taking care or answering the questions that arise.

Regards;

Roberto.

[AstralStorm]

Oh BTW, will the personal results of the tests be published on the net or just the summary?
I'm especially interested in comments.

This post has been edited by AstralStorm: Jul 24 2003, 13:09

[rjamorim]

Oh BTW, will the personal results of the tests be published on the net or just the summary?
I'm especially interested in comments.

This time they will. I changed the readme for that purpose.

Only when the test finishes though.

[Night Rain]

Submitted to Warp2Search.