48 kbps AAC Encoders Test - Q1 2006 Edition Options

[Ivan Dimkovic]

Hello,

This is the invitation for a discussion about 48 Kbps AAC test, that should be done before the next 48 kbps multiformat listening test.

Gabriel from LAME development team and myself decided to organize this test - there are few reasons why this test should be organized, and why it should be organized before the next 48 kbps multiformat test:

Reasons:

* Test the state-of-the-art HE-AAC v1 and HE-AAC v2 (Parametric Stereo) commercial encoders, as well as non-commercial 3GPP reference encoder

* Evaluate the usefullnes of Parametric Stereo tool at bit rate of 48 Kb/s

* Make the pre-selection of the best HE-AAC/HE-AAC v2 solutions for the multiformat listening test

* Compare all these solutions with the low-anchored Low Complexity (LC) AAC, in order to verify how the technology has improved over past few years

* Compare against higher bit rate MP3 to verify how much could state-of-the-art 48 kbps codec approach the good old MP3 technology at significantly higher bit rate.

Which high bit rate of MP3 should be used is debatable - should it be 80, 96 or 128 kbps - as stated above, this is open for debate.

Codecs

At this point (alphabetically):

- 3GPP Reference Encoder
- Coding Technologies aacPlus (HE-AAC)
- Coding Technologies Enhanced aacPlus (HE-AAC v2)
- Nero Digital HE-AAC v1
- Nero Digital HE-AAC v2

Anchors:
- LAME MP3 (high) - bit rate yet TBD / High Anchor
- QuickTime/iTunes LC-AAC @48 kbps / Low Anchor

Samples

My idea was to use combined sample set from multiformat 128 kbps listening test and previous Roberto's 64 kbps multiformat test - I thought of chosing several (5-6) samples from both of these two sets by using the worst scored samples.

Worst scored = Lowest average subjective difference grade (SDG) for all codecs in the particular set, except the low and (where applicable) high anchors.

Gabriel's addition was:

* 1 spoken (german male speech?)
* 1 with strong L/R differences, in order to know how PS fares in bad conditions. Something like some old Beatles songs, with singer on 1 channel and instruments on the other one.

Sample selection is definitely open for further discussion here

When

Test should happen at the end of January - to make enough break after the current 128 kbps multiformat test.

Encoding Settings, Encoders, Bit Rates

Like with all other tests, we won't be testing closed or produced solutions - encoders used in the test will be widely available - as well as encoding settings and encoding material. Regarding the Coding Technologies' aacPlus encoder, my proposal is to use their imlpementation in the Winamp, which is widely available product.

Prior to the test, we will conduct public bit rate verification - in order to make sure codecs behave within the bitrate constraints.

[elmar3rd]

Prior to the test, we will conduct public bit rate verification - in order to make sure codecs behave within the bitrate constraints.

So this test will be a mix of CBR (aacPlus) and VBR?
48 kb/s ist very interesting for streaming internet radio. Is VBR generally suitable for this? Otherwise, the result won't be very interesting for streaming.

[Shade[ST]]

I have a few samples which you might like, and are fairly easy to make out. Also, I suggest using vorbis as a high anchor (since it will probably win the 128kbps test anyways..)

PM or email me, to get the samples or for further ideas (please do)

[[JAZ]]

About High Anchor:

If MP3, then, a CBR (or ABR) 128kbps will be interesting, as to compare to internet-radio quality, as well as DVD Rips.

Else, i would suggest AAC, with Apple's implementation at 96kbps (CBR? VBR?) (or 128 if 96 is not discernable enough).

[dimzon]

And please add avoTuVo Vorbis @ 48kbsp to - just to compare it to AAC+
PS. There are no other pretendents for 48kbps tests - just multiple AAC+(2) and avoTuVo Vorbis - why not to make this test multiformat?

This post has been edited by dimzon: Dec 14 2005, 16:29

[Shade[ST]]

And please add avoTuVo Vorbis @ 48kbsp to - just to compare it to AAC+

........

READ!!!

it's an AAC test.

[dimzon]

READ!!!

it's an AAC test.

Oh sorry, it's my mistake...
But... There are no other pretendents for 48kbps tests - just multiple AAC+(2) implementations and avoTuVo Vorbis - why not to make this test multiformat?

[Shade[ST]]

There are over 4 implementations of AAC-HE, and a 48kbps multiformat test _is_ coming up. It'll just be after we find out which AAC implementation is best at this bitrate.

[dimzon]

Oh!
And don't forget - Coding Technologies aacPlus encoder (at least WinAmp variant) have ParametricStereo/Stereo/IndependentStereo modes !

[Gabriel]

So this test will be a mix of CBR (aacPlus) and VBR?
48 kb/s ist very interesting for streaming internet radio. Is VBR generally suitable for this? Otherwise, the result won't be very interesting for streaming.

Considering the bitrate, it might makes sense to use the contenders in cbr mode.

[Ivan Dimkovic]

48 kb/s ist very interesting for streaming internet radio. Is VBR generally suitable for this? Otherwise, the result won't be very interesting for streaming.

I don't think true VBR has too much sense here - we might use manageable bit-rate ABR mode - but this is still not decided.

dimzon
But... There are no other pretendents for 48kbps tests - just multiple AAC+(2) implementations and avoTuVo Vorbis - why not to make this test multiformat?

One of the purposes of this test would be actually to pick up the best HE-AAC/v2 implementations and use them in the upcoming 48 kbps multiformat test.

Multiformat test should include many other formats, like WMA, Real, mp3Pro, etc... - so there definitely won't be space for 5 (five) AAC encoders - that's why we want to make sure the best ones get into the test.

[guruboolez]

So this test will be a mix of CBR (aacPlus) and VBR?
48 kb/s ist very interesting for streaming internet radio. Is VBR generally suitable for this? Otherwise, the result won't be very interesting for streaming.

Are really 48 kbps encoding useful for streaming?
56K can't stream such encodings, CBR or not. Theoretically, it should be possible, but in fact, it doesn't work.
For people having a better download bandwith (the next step is 128 kbps if I'm not wrong), I'm not sure that 48 kbps really makes sense: 64, 80 or even 96 kbps are probably more interesting.

In my opinion, if internet streaming is proposed as practical purpose of the test, then:
- CBR should be used over VBR
- 32 kbps should be prefered to 48 kbps

But if streaming isn't invoked, then:
- 48 kbps is fine
- VBR should be prefered over CBR.

[dimzon]

Multiformat test should include many other formats, like WMA, Real, mp3Pro, etc... - so there definitely won't be space for 5 (five) AAC encoders - that's why we want to make sure the best ones get into the test.

IMHO nobody can't beat AAC+/Vorbis at such bitrates!
AAC+ and Vorbis are only real concurrents...
there are no any progress in mp3pro, Real, WMA since last lowbitrate test and this encoders defeats to early HE-AAC implementation!
and, in theory, aac+ is better than mp3pro by definition (of couse if AAC+ use good LC encoder and then both use same SBR encoder)

[IgorC]

For many people here 48 kbit/s has sence.
It was mentioned it is highest bitrate for PS. Particullart for me 48 kbit/s will be very interesting . I will use for different aplications. Nowdays 56 kbit/s dial-up internet has the same price as cablemodem/dsl 128 kbit/s. there a lot of people who are interesting in 48 kbit/s test.

This post has been edited by IgorC: Dec 14 2005, 17:10

[Ivan Dimkovic]

Hmm - nobody really tested WMA @48 kbps vs. anything else here - and, even if someone did it would be very good to know how codecs compare with each other today, for the further reference.

Otherwise, somebody could always claim that codec X might be better at 48 kbps, as it was not tested, etc...

guruboolez
Are really 48 kbps encoding useful for streaming?
56K can't stream such encodings, CBR or not. Theoretically, it should be possible, but in fact, it doesn't work.

Indeed - maybe good GPRS/EDGE mobile connections could stream that easily.

But there is also another point in 48 (or around) kbps - I believe (yet to be proven) that state-of-the-art could bring significant advantage and lower down the streaming bit rate for DSL connections to 48 kbps for audio - comparable to the MP3 quality of much higher bit rate.

This is important IMHO because of two reasons:

- For the broadcasters, especially online radio stations - it is very important, as they do pay the bandwidth originating from them - and cutting from 100 kbps to 48 kbps means roughly 50% of savings in money spent for bandwidth costs

- Also, for comined audio-video streams, it is very important as more bits will be available for video - if we can prove that 48 kbps state-of-the-art provides comparable quality to higher bit rate solutions, it would be a huge benefit for the video quality, too.

[Shade[ST]]

I'm not sure that 48 kbps really makes sense: 64, 80 or even 96 kbps are probably more interesting.

Cell phones or choppy streaming is fine...

And I've seen 56k modems download at 6.7kBps, so streaming 48kbps audio shouldn't be a HUGE problem.. In any case, it's always interesting if you have a DSL line but greatly limited bandwidth, or an ISDN connection (RNIS)

edit : Argh!! Ivan Dimkovic shot the sheriff...

This post has been edited by Shade[ST]: Dec 14 2005, 17:13

[dimzon]

Hmm - nobody really tested WMA @48 kbps vs. anything else here - and, even if someone did it would be very good to know how codecs compare with each other today,  for the further reference.

http://www.rjamorim.com/test/64test/results.html
http://www.rjamorim.com/test/32kbps/results.html
just interpolate results

[Ivan Dimkovic]

I wouldn't go interpolating results that easily

By the way, Woodinville also mentioned new WMA coming in January - so it will be perfect timing when the AAC test is done - to put the brand new WMA in the multiformat test

[dimzon]

I beleive it wood be much more usable to add old HE-AAC implementation to this test. It will be very intresting to know which progress does Nero do

[guruboolez]

For many people here 48 kbit/s has sence.
It was mentioned it is highest bitrate for PS.  Particullart for me 48 kbit/s will be very interesting . I will use for different aplications. Nowdays  56 kbit/s dial-up internet has the same price as cablemodem/dsl 128 kbit/s. there a lot of people who are interesting in 48 kbit/s test.

Right. But if people have a 128 kbps bandwith connection or more, why shouldn't "true VBR [make] much sense here" (Ivan Dimkovic)? Why testing 48 kbps and not 64 or 80 kbps? I'm sorry, but if I had a modern DSL/cable connection, the last thing I'll looking for would be 48 kbps encodings.

This post has been edited by guruboolez: Dec 14 2005, 17:20

[stephanV]

At the 32 kbps test WMA and Real were tied with Vorbis.

So this comment:

IMHO nobody can't beat AAC+/Vorbis at such bitrates

just doesn't make sense.

[IgorC]

Somebody want test at 32 kbps , another at 64 and higher. Also many people want at 48 kbit/s. It´s difficult to establish an average bitrate to satisfy all users.
48 kbit/s is an average bitrate ... maybe most wanted. I know teh results can be different for 32-48-64-.... bitrates. Or maybe let´s vote (making poll)?

[Ivan Dimkovic]

guruboolez
Right. But if people have a 128 kbps bandwith connection or more, why shouldn't "true VBR [make] much sense here" (Ivan Dimkovic)? Why testing 48 kbps and not 64 or 80 kbps? I'm sorry, but if I had a modern DSL/cable connection, the last thing I'll looking for would be 48 kbps encodings.

Well if the VBR is predictable and manageable - and requires no extra long buffering for 48 kbps streaming, I guess it is OK - but have in mind that not all competitors have VBR capabilities.

As for the DSL/Cable connection - I believe that the point of the test is not to show "what is best possible solution for a consumer, concerning audio-only streaming" - as with todays DSL networks, answer is quite simple - modern codec ~128 kbps would do - and we will find out later this year could 96 kbps do, too.

But this is not really about that - I am quite sure that broadcasters and individuals hosting their own radio stations and content libraries would like to know how low they can go without damaging quality too much for ordinary users, and still save some significant bandwidth costs, and also - could they penentrate the upcoming mobile GPRS/EDGE/UMTS market - remember, there it still counts how much you transfer

Also, please have in mind that there are more mobile phones than fixed lines now worldwide.

Also for the user would be interesting if he could get higher quality video - cutting from 128 to 48 kbps is 80 kbps - and that means a lot for streaming video.

[guruboolez]

Somebody want test at 32 kbps , another at 64 and higher. Also many people want at 48 kbit/s.  It´s difficult to establish an average bitrate  to satisfy all users.
48 kbit/s is an average  bitrate ... maybe most wanted. I know teh results  can be different for 32-48-64-.... bitrates. Or maybe let´s vote (making poll)?

I have nothing against 48 kbps; I'm not in favor of 32 or 64 kbps instead. But I'm just looking for coherence. Why discarding VBR and using 48 kbps at the same time? If someone can stream 48 kbps (i.e. if he doesn't have a dial-up), then he should be able to handle VBR without problem. No?

[IgorC]

[Also for the user would be interesting if he could get higher quality video - cutting from 128 to 48 kbps is 80 kbps - and that means a lot for streaming video.

Finally somebody understand me and other people like me, Dimzon froom D9...
AAC+2 epscually good for soundtracks. There is some background music, noises, speech etc. It´s much harder to listen SBR distortion than on music samples.

This post has been edited by IgorC: Dec 14 2005, 17:38