Hello.

I'd like to announce the start of the 128kbps extension test, comparing the AAC winner (QuickTime) to Musepack, Vorbis, WMA Pro and Lame MP3.

Instructions are available at the announcement page:
http://audio.ciara.us/test/128extension/pr...esentation.html

If questions or issues arise, please post at this thread.

Thank-you.

Regards;

Roberto Amorim

Any .torrents going to be made available this time?

And should this test be advertised to the broader community? (ie. Slashdot, Kuro5hin, etc)

I think it should be mentioned that you're comparing to Blade MP3 as well, even though it's meant to show just how bad a psychoacoustic codec can be. That way it gives people a sense of why the 128kbps MP3s they downloaded really suck.

Probably, yes.

But Dibrom has been too busy lately, and he plans to create the torrents after HA moves to another server.



Yes, it would be very welcome.

Unfortunately, I'm not a member of Slashdot or Kuro5hin, so I don't know about article submissions. I would be very grateful if some insider can post about this test there, pointing to the presentation page.



It will only be mentioned at the results page

That way, people won't be confused, since it's there for perspective purposes, not really to be compared.

Regards;

Roberto.

Just submitted it to slashdot.

Posted to newsgroups:

rec.audio.opinion
rec.audio.high-end
alt.music.mp3, rec.audio.misc, uk.rec.audio

ff123

Made a little comment at Ars Technica.

How it`s done? All files are VBR but calculated averange 128kbps? So you encoed each file in each format in VBR mode unless acheving e.g. 127-129 kbps averange? IMO it`s best choice to test >>>128kbps VBR<<< files.

EDIT:
"Ogg Vorbis 1.0 post-CVS -q 4.25" - it`s not `128kbps every time, you can`t say that it is TRUE 128kbps test!

see this threat:

http://www.hydrogenaudio.org/forums/index....showtopic=11134

regards; ilikedirt

- these samples (exept one) are not in 128kbps...

nobody said that, but their average bitrates are closesed to 128 from the tested ogg settings (~127).

i think this method was chosen, because vbr encoders get used this way (through quality settings). usually nobody would encode a files several times with vbr to get the bitrate closest to 128kb

regards; ilikedirt

Of course nobody will encode in such way! But we test _128kbps_ files. After test we can say that in for 128kbps for samples X codec Y was better. What if WMA will have 130 kbps and Vorbis 110 kbps?

I just want to say that normal users behaviour is completely not important in 128kbps test.

of course i see your point here. maybe rjamorim can tell you why he decided to test this way...

regards; ilikedirt

I downloaded 12 files with test samples. Two questions: 1. why it`s not a blind test? 2. Why only original, mp4 and wma files in packages?

What is important is that the overall bitrate is about the same for every codec. If for a specific track codec A is using an average bitrate of 130 and codec B an average bitrate of 110, it is just the decision made by the codec. It is programmed this way, an no one manually forced codec A to use an higher bitrate.

Manually forcing each codec to reach exactly 128kbps for each sample would be unfair, as the codecs are not designed to exhibit such a behaviour. What is important is the overall bitrate.

As Gabriel points out - this way you are also testing a codecs ability to judge how hard a sample is. If Vorbis only gives the sample 110kbs and ends up sounding bad because of it - that is the fault of Vorbis's VBR handling, and so it should be punished

As far as I can tell, you can see which files are which because you load them into ABC-HR - which does all the blinding for you Otherwise stick a biro in your eyes and you will be blind! (joke)

Have you looked at the contents of the zip archives, or even read the readme?

The abc-hr_bin.zip package includes various CLI en-/decoders - and each sample package comes with a batch file which uses these CLI en-/decoders to convert the supplied FLAC file to a lossy format (and decode back to WAV again for ABC/HR to accept it). For those formats where this isn't possible because of the lack of CLI encoders (WMA, MP4), readily encoded files are supplied. This method saves loads of bandwidth.

Okay I'm new to this thing....


Does using the ABX tool and comparing the original and sample x repeatedly before giving an actual rating skew the results?


Or are we supposed to simply listen from the rating table itself?

just one thing to say:

thanks a lot for this great test, rjamorim!!!

Two tests done... death2 and waiting.

/EDIT\
No, comparing two samples before deciding on the result doesn't skew the results.

ABX testing proves your results, especially if the difference is subtle.
\EDIT/

Another question on this testing thing ...

Rating 1-5 is against the original file right? Not against each other right? So assuming two codecs sound the same to you in comparision with the original file, they would rate about the same score? ( that's what I get from reading the Pratice With ABC/HR page )

Or is scoring done differently?

Also, are comments absolutely necessary?

The tests are rating against original.

Comments aren't necessary, but useful for others.

Thanks a lot!

And thanks to everybody that is taking care or answering the questions that arise.

Regards;

Roberto.

Oh BTW, will the personal results of the tests be published on the net or just the summary?
I'm especially interested in comments.

This time they will. I changed the readme for that purpose.

Only when the test finishes though.

Submitted to Warp2Search.

Why mppenc 1.14 and not 1.15r? As far as I know, it has been very well tested.

Someone else should try it...

dev0

Submitted to Ars Technica news desk.

If you're submitting to slashdot you should make clear that this is not another duplicate submission of the original 128kbps test which they accidentally reposted a few days ago.

http://slashdot.org/article.pl?sid=03/07/2...tid=181&tid=188

No-one seemed to notice, though the comments seem of even lower quality.

had a quick go... good god its hard... only really able to pick out 1-2 ones that sound different.. my hearing must be very bad i DID spot preecho on fatboy at 320kbs --alt-preset insane once but had to turn it up so high as it got so quiet at that bitrate... I just dont know what i am looking for in here... death2 is the only one so far where i've heard something on 2 files...

Are most people able to pick out easily on a lot of them? these codecs sure are good at 128k (or there abouts)

Ummm, I think we have a problem!

I just ran the sample3.bat, and a couple of weird numbers caught my eye, so I changed the bat file so that it didn't delete the vorbis and mpc files...

Apples and oranges!

According to foobar, Bachpsichord.mpc - bitrate=198
Bachpsichord_ogg.ogg - bitrate=172

I would have thought it was a bit pointless directly comparing these against the .mp3 and .mp4 which both come in at 128 kbit according to foobar.



Den.

there is always one blade mp3 encode in the range. i try to track it first, abx it 10/10 ( ) and then take a look at the harder ones...

i personaly find sample 12 and sample 07 (here only the initial applaud) to be quite easy.

regards; ilikedirt

Also sample1's .mpc is 166 kbits.

I would have thought that this is not close enough to 128...

Den.

Thankfully I'm not the only deaf one... Only sample 1 seems easy for me.. the rest are pretty difficult..

i can confirm that

winamp tells me:
Bachpsichord_ogg.ogg - Average bitrate : 164 kbps - Nominal bitrate : 128 kbps
Bachpsichord.mpc - Bitrate: VBR 197.9 kbps

the others seem fine


i wouldnt call it extension test!
call it something like hydrogenaudio 128kbps audio codec comparison test

but perhaps it would be wise to wait




and rjamorim can change the ogg and mpc settings if necessary (?) so that they are around 128kbps
if yes, i think it should help do rename the output files in sampleXX.bat to something like
"Sample_ogg2.wav" so that rjamorim can see in the results if the old ogg/mpc was tested or the newer ones...

due to the ","/"." failure in oggenc all .ogg files were encoded with "-q 4" (and not with "-q 4,25"):

sample1:
41_30sec_ogg.ogg - Average bitrate : 136 kbps
41_30sec.mpc - Bitrate: VBR 166.3 kbps
41_30sec.wav.mp3 - 126kbit (vbr) lame

sample3:
Bachpsichord_ogg.ogg - Average bitrate : 164 kbps
Bachpsichord.mpc - Bitrate: VBR 197.9 kbps
Bachpsichord.wav.mp3 - 125kbit (vbr) lame

sample7:
Layla_ogg.ogg - Average bitrate : 148 kbps
Layla.mpc - Bitrate: VBR 151.1 kbps
Layla.wav.mp3 - 130kbit (vbr) lame

sample11:
TheSource_ogg.ogg - Average bitrate : 121 kbps
TheSource.mpc - Bitrate: VBR 128.0 kbps
TheSource.wav.mp3 - 134kbit (VBR) lame

sample12:
Waiting_ogg.ogg - Average bitrate : 131 kbps
Waiting.mpc - Bitrate: VBR 147.6 kbps
Waiting.wav.mp3 - 122kbit (VBR) lame


perhaps in the end rjamorim should divide the points given to a codec through the bits used and than calculate an average for 128kbps

No, but it's better if you do not know which is the codec you are ABXing, so that you can't possibly learn the sonic signature of it for that sample. If you do, this could lead to identifying each codec in the ABC/HR test, and would weaken the "blindness" of the procedure. So, just use the built-in ABX comparator in ABC/HR, but not an external one.

In my opinion, it doesn't really matter. 1.15r quality is really close to 1.14 performance. In rare cases, benefit is audible (amnesia for exemple). But in others, 1.14 may be better. And this applies to --standard profile only...

But a most important thing to note is : 1.15r encodings are bigger than 1.14. Especially on some samples, as harpsicord (+10-15 kbps). And I doesn't ear any difference... For a mid-bitrate listening test, this small difference is maybe more annoying than in a archive perspective.


______

For people that are surprised with some bitrate deviation, don't forget two things :

- first, this is perfectly normal when you're testing the same VBR setting with various samples. You can't expect a constant value for different complexity samples : that's against VBR logical.

- second, you can't take a 30 seconds samples as a basis. For exemple, Bachpsichord 20 seconds are probably the highest ones of the whole double-disc of English Suites. Others samples where selected, and cutted, for their encoding difficulties. We can't expect anything else than a serious bitrate inflation with a well-tuned VBR setting (most famous exemple : first seconds of Kalifornia from Fatboy Slim).

Just take a whole Metallica album : bitrate will be near ~128 kbps with mppenc --radio. Isolate samples will probably reach 170-180 samples. Will people be annoyed by it ? Will they even notice it, without cutting a small part of the original PCM file and encode it ? No...

As a consequence, we haven't to be bothered by bitrate values of different encodings. It's a non-sense to criticize the bitrate amplitude of each VBR format on isolated samples. If you want 128 kbps for each format, choose CBR. If you choose VBR, enjoy the amplitude !

The point of the thread listed below was to find the appropriate settings for ogg and mpc which would average about 128 kbit/s across many album's worth of music:

http://www.hydrogenaudio.org/forums/index....showtopic=11134

But being VBR, they will sometimes have much higher bitrates on samples with lots of transients. It's not fair to reduce the bitrates on these samples down to 128 kbit/s because that's not the way they'd actually be used.

ff123

hm, i see

ok, let's continue listening

It's frustrating to leave some of the codecs at a rating of 5, isn't it? At least it is to me.

Here's how I usually listen (it takes some time):

First, I make sure everything's as quiet as I can get it around the house (last night and tonight are good, because my wife and kids are away).

For a particular sample, I listen to each codec in its entirety (unfortunately, that means listening to each of the hidden references too). I can usually pick out the 1 or 2 worst entries this way. Then I'll go through the sample a short section at a time to listen for subtle differences I might not have picked up in the whole-sample listening. Most of the time I'll pick up a problem in one codec in one section, but another problem in another codec in a different section.

For very subtle differences, ABX'ing usually helps to hone my ability to hear a defect.

Still, this comparison a lot more difficult than I think some people may have thought, given that this is "only" 128 kbit/s.

ff123

Thanks ff123 for patiently explaining this to people who obviously missed Gabriel's explanation on the first page.. I prolly wouldn't have patience for as cool answer for a question which was answered once in this thread already and in a separate thread..
I hope nobody asks this for the 4th time in this thread..............

Maybe Roberto should put the explanation to his listening test page as well.

I for one am having a hard time even finding the Blade encoded file. This test will take a lot longer than the few hours I had planned to devote to it.

Regarding LAME alt-preset 128, which is ABR not VBR:

To have a fair competition for the ABR samples, we have to encode a complete file containing a problem-sample. The whole file must have an average bitrate of 128 kbps. Then we have to cut off the part with the problem sample for the listening-test. Only in this case, we give ABR-settings like alt-preset 128 the chance to increase the bitrate if needed like the VBR-settings do.

[Hope someone understand my bad english]

I've found this test quite dis-heartening. Out of 5 samples I have tried, I could only pick out Blade one 1 of them. I can't even ABX another codec in any of them (why isn't there a crying emoticon?)

Hmm.. I don't know if this is very big issue with Lame ABR since it's 1 pass encode and ABR has unlimited bit reservour anyway.

If someone has time, check how the bit allocation goes in these 2 situations for example with encspot.

Edit. Hmm.. now that I actually think of it, yeah, this might be an issue exactly because of the unlimited bit reservour.

Old issue with oggenc rises the head again:



Anyone with regional settings specifying other character than '.' for decimal separator will get too low quality Vorbis files.

Ouch.. this is definitely a problem, although not catastrophic. -q4 is officially 128kbps nominal anyway.

One could almost say the same thing about any codec which uses a bit-reservoir.

In practice, the only codec where it might matter whether or not the sample was "sliced from" a whole song, post-encoding, would be WMA9Pro 2-pass VBR. Ideally, one would perform the 2-passes on the entire song (if not the entire album) and then slice out the sample afterwards.

But that's not very practical for this test, among other reasons being that Roberto doesn't have copies of the entire songs for the test suite. So this test ends up using the 2-pass on just the 20 second samples. Note that plain 1-pass VBR for WMA9Pro was found to be too variable in the bitrate thread mentioned above.

ff123

I uploaded modified oggenc.exe that will use proper quality level, it uses recent CVS libraries.