Hello,

This is the invitation for a discussion about 48 Kbps AAC test, that should be done before the next 48 kbps multiformat listening test.

Gabriel from LAME development team and myself decided to organize this test - there are few reasons why this test should be organized, and why it should be organized before the next 48 kbps multiformat test:

Reasons:

* Test the state-of-the-art HE-AAC v1 and HE-AAC v2 (Parametric Stereo) commercial encoders, as well as non-commercial 3GPP reference encoder

* Evaluate the usefullnes of Parametric Stereo tool at bit rate of 48 Kb/s

* Make the pre-selection of the best HE-AAC/HE-AAC v2 solutions for the multiformat listening test

* Compare all these solutions with the low-anchored Low Complexity (LC) AAC, in order to verify how the technology has improved over past few years

* Compare against higher bit rate MP3 to verify how much could state-of-the-art 48 kbps codec approach the good old MP3 technology at significantly higher bit rate.

Which high bit rate of MP3 should be used is debatable - should it be 80, 96 or 128 kbps - as stated above, this is open for debate.

Codecs

At this point (alphabetically):

- 3GPP Reference Encoder
- Coding Technologies aacPlus (HE-AAC)
- Coding Technologies Enhanced aacPlus (HE-AAC v2)
- Nero Digital HE-AAC v1
- Nero Digital HE-AAC v2

Anchors:
- LAME MP3 (high) - bit rate yet TBD / High Anchor
- QuickTime/iTunes LC-AAC @48 kbps / Low Anchor

Samples

My idea was to use combined sample set from multiformat 128 kbps listening test and previous Roberto's 64 kbps multiformat test - I thought of chosing several (5-6) samples from both of these two sets by using the worst scored samples.

Worst scored = Lowest average subjective difference grade (SDG) for all codecs in the particular set, except the low and (where applicable) high anchors.

Gabriel's addition was:

* 1 spoken (german male speech?)
* 1 with strong L/R differences, in order to know how PS fares in bad conditions. Something like some old Beatles songs, with singer on 1 channel and instruments on the other one.

Sample selection is definitely open for further discussion here

When

Test should happen at the end of January - to make enough break after the current 128 kbps multiformat test.

Encoding Settings, Encoders, Bit Rates

Like with all other tests, we won't be testing closed or produced solutions - encoders used in the test will be widely available - as well as encoding settings and encoding material. Regarding the Coding Technologies' aacPlus encoder, my proposal is to use their imlpementation in the Winamp, which is widely available product.

Prior to the test, we will conduct public bit rate verification - in order to make sure codecs behave within the bitrate constraints.

So this test will be a mix of CBR (aacPlus) and VBR?
48 kb/s ist very interesting for streaming internet radio. Is VBR generally suitable for this? Otherwise, the result won't be very interesting for streaming.

I have a few samples which you might like, and are fairly easy to make out. Also, I suggest using vorbis as a high anchor (since it will probably win the 128kbps test anyways..)

PM or email me, to get the samples or for further ideas (please do)

About High Anchor:

If MP3, then, a CBR (or ABR) 128kbps will be interesting, as to compare to internet-radio quality, as well as DVD Rips.

Else, i would suggest AAC, with Apple's implementation at 96kbps (CBR? VBR?) (or 128 if 96 is not discernable enough).

And please add avoTuVo Vorbis @ 48kbsp to - just to compare it to AAC+
PS. There are no other pretendents for 48kbps tests - just multiple AAC+(2) and avoTuVo Vorbis - why not to make this test multiformat?

........

READ!!!

it's an AAC test.

Oh sorry, it's my mistake...
But... There are no other pretendents for 48kbps tests - just multiple AAC+(2) implementations and avoTuVo Vorbis - why not to make this test multiformat?

There are over 4 implementations of AAC-HE, and a 48kbps multiformat test _is_ coming up. It'll just be after we find out which AAC implementation is best at this bitrate.

Oh!
And don't forget - Coding Technologies aacPlus encoder (at least WinAmp variant) have ParametricStereo/Stereo/IndependentStereo modes !

Considering the bitrate, it might makes sense to use the contenders in cbr mode.

I don't think true VBR has too much sense here - we might use manageable bit-rate ABR mode - but this is still not decided.



One of the purposes of this test would be actually to pick up the best HE-AAC/v2 implementations and use them in the upcoming 48 kbps multiformat test.

Multiformat test should include many other formats, like WMA, Real, mp3Pro, etc... - so there definitely won't be space for 5 (five) AAC encoders - that's why we want to make sure the best ones get into the test.

Are really 48 kbps encoding useful for streaming?
56K can't stream such encodings, CBR or not. Theoretically, it should be possible, but in fact, it doesn't work.
For people having a better download bandwith (the next step is 128 kbps if I'm not wrong), I'm not sure that 48 kbps really makes sense: 64, 80 or even 96 kbps are probably more interesting.

In my opinion, if internet streaming is proposed as practical purpose of the test, then:
- CBR should be used over VBR
- 32 kbps should be prefered to 48 kbps

But if streaming isn't invoked, then:
- 48 kbps is fine
- VBR should be prefered over CBR.

IMHO nobody can't beat AAC+/Vorbis at such bitrates!
AAC+ and Vorbis are only real concurrents...
there are no any progress in mp3pro, Real, WMA since last lowbitrate test and this encoders defeats to early HE-AAC implementation!
and, in theory, aac+ is better than mp3pro by definition (of couse if AAC+ use good LC encoder and then both use same SBR encoder)

For many people here 48 kbit/s has sence.
It was mentioned it is highest bitrate for PS. Particullart for me 48 kbit/s will be very interesting . I will use for different aplications. Nowdays 56 kbit/s dial-up internet has the same price as cablemodem/dsl 128 kbit/s. there a lot of people who are interesting in 48 kbit/s test.

Hmm - nobody really tested WMA @48 kbps vs. anything else here - and, even if someone did it would be very good to know how codecs compare with each other today, for the further reference.

Otherwise, somebody could always claim that codec X might be better at 48 kbps, as it was not tested, etc...



Indeed - maybe good GPRS/EDGE mobile connections could stream that easily.

But there is also another point in 48 (or around) kbps - I believe (yet to be proven) that state-of-the-art could bring significant advantage and lower down the streaming bit rate for DSL connections to 48 kbps for audio - comparable to the MP3 quality of much higher bit rate.

This is important IMHO because of two reasons:

- For the broadcasters, especially online radio stations - it is very important, as they do pay the bandwidth originating from them - and cutting from 100 kbps to 48 kbps means roughly 50% of savings in money spent for bandwidth costs

- Also, for comined audio-video streams, it is very important as more bits will be available for video - if we can prove that 48 kbps state-of-the-art provides comparable quality to higher bit rate solutions, it would be a huge benefit for the video quality, too.

Cell phones or choppy streaming is fine...

And I've seen 56k modems download at 6.7kBps, so streaming 48kbps audio shouldn't be a HUGE problem.. In any case, it's always interesting if you have a DSL line but greatly limited bandwidth, or an ISDN connection (RNIS)

edit : Argh!! Ivan Dimkovic shot the sheriff...

http://www.rjamorim.com/test/64test/results.html
http://www.rjamorim.com/test/32kbps/results.html
just interpolate results

I wouldn't go interpolating results that easily

By the way, Woodinville also mentioned new WMA coming in January - so it will be perfect timing when the AAC test is done - to put the brand new WMA in the multiformat test

I beleive it wood be much more usable to add old HE-AAC implementation to this test. It will be very intresting to know which progress does Nero do

Right. But if people have a 128 kbps bandwith connection or more, why shouldn't "true VBR [make] much sense here" (Ivan Dimkovic)? Why testing 48 kbps and not 64 or 80 kbps? I'm sorry, but if I had a modern DSL/cable connection, the last thing I'll looking for would be 48 kbps encodings.

At the 32 kbps test WMA and Real were tied with Vorbis.

So this comment:

just doesn't make sense.

Somebody want test at 32 kbps , another at 64 and higher. Also many people want at 48 kbit/s. It´s difficult to establish an average bitrate to satisfy all users.
48 kbit/s is an average bitrate ... maybe most wanted. I know teh results can be different for 32-48-64-.... bitrates. Or maybe let´s vote (making poll)?

Well if the VBR is predictable and manageable - and requires no extra long buffering for 48 kbps streaming, I guess it is OK - but have in mind that not all competitors have VBR capabilities.

As for the DSL/Cable connection - I believe that the point of the test is not to show "what is best possible solution for a consumer, concerning audio-only streaming" - as with todays DSL networks, answer is quite simple - modern codec ~128 kbps would do - and we will find out later this year could 96 kbps do, too.

But this is not really about that - I am quite sure that broadcasters and individuals hosting their own radio stations and content libraries would like to know how low they can go without damaging quality too much for ordinary users, and still save some significant bandwidth costs, and also - could they penentrate the upcoming mobile GPRS/EDGE/UMTS market - remember, there it still counts how much you transfer

Also, please have in mind that there are more mobile phones than fixed lines now worldwide.

Also for the user would be interesting if he could get higher quality video - cutting from 128 to 48 kbps is 80 kbps - and that means a lot for streaming video.

I have nothing against 48 kbps; I'm not in favor of 32 or 64 kbps instead. But I'm just looking for coherence. Why discarding VBR and using 48 kbps at the same time? If someone can stream 48 kbps (i.e. if he doesn't have a dial-up), then he should be able to handle VBR without problem. No?

Finally somebody understand me and other people like me, Dimzon froom D9...
AAC+2 epscually good for soundtracks. There is some background music, noises, speech etc. It´s much harder to listen SBR distortion than on music samples.

Please, use both modes CBR/VBR if possible!

Streaming 48kbps on dial-up IS possible.

I dont mind about VBR. But it less predictable for final size. In consequence less fair. Maybe ABR will be good.

Yeah! AAC+ is perfect choice for soundtrack:
H264 + HE-AAC =MP4 container

Well - theoretically we (Nero) could do CBR, quality (threshold) based VBR, bit rate manageable VBR, ABR with large bit buffer, 2-pass CBR, etc..

I am afraid that we cannot test all that - number of codecs would be too big, I am for using one codec-one mode approach here, otherwise we might be lost in tons of codecs and settings.

WOW!!!!!!!!
It means BEST bitrate distribution @ suggested bitrate!
I really like it!

Did you try?

Do you mean that forcing CBR good VBR implementations is really something fair? If VBR perform better than CBR, it should be tested. BTW, VBR should be used in the second test (the multiformat one): Vorbis...
My question is: why discarding VBR (VBR has always be considered as better than CBR/ABR on this board)? Predictible size? Bullshit: VBR has always be priviledged in listening tests. Streaming? Nonsense, because VBR could be stream at ~48 kbps with every kind of internet connection excepted dial-up.

I did. And it worked. It wasn't HE-AAC, but standard aac. "dans un coin perdu du québec, en plus"

Not here.

Problem with 2-pass is that it is not available in the software being shipped, only in the experimental debug encoder - and making it in would take considerable amount of time and modification to make it there, without internal justification and proof that it is actually better (especially compared to ABR which does not require multipass) - so I wouldn't use it for now.



Ok, we will use something that is average 48 kbps and possible to stream without large buffers - as well as highest quality.

It´s not b***it. Yes. Predictible size. The idea of HE-AAC2 is save some bits (from 128 kbps to 48 kbps). And If intead of 128 there will be 130-133 kbps then I would say it´s b***it.

Don't make a drama. VBR is enought predictable - You will get approx +/- 8 kbps final bitrate deviation...

+/- 8 kbps? Are you serious? No. I want to see 128 kbit/s mp3 and 48 kbps/s HE-aac2. I cant give more bitrate. Not even +/- 2 kbit/s.

VBR is better than ABR. Why is better? VBR has bigger size. Thats all.
ABR two pass should be on par with VBR (or even better). And the size isn´t issue for 135-180kbps. but it´s issue for 48 bkit/s comparing to 128 kbit/s.
But the purpose of this test at low bitrate is to safe some bits.

Guru is pruposing here to use VBR as for 135-180 tests when +/- 100/200 kbyte isn´t issue. But not here.

Not for me - I can sacrify bitrate a little to have const quality... I'm calculating target video bitrate after audio encoding 8kbps is nothing for video!

VBR/CBR issue:
As you know, modern codecs are using a bit reservoir. It can be directly handled in frames (like mp3), or handled by a regulation part (like some mpeg4 modes).
If your encoder respects the normative constraints regarding CBR, then it is cbr, altough its instantaneous bitrate can vary.

CBR is a way to be sure that you can transmit the stream using a given bandwidth, provided that the decoder has a specific buffer size.

The mp3 cbr is very restricted, as the bit reservoir is only 4088 bits. The AAC reservoir is way higher than that, allowing more large instantaneous fluctuations while still beeing cbr.
If you are using VBR but are enforcing a given average bitrate over a given sliding time slice, then the same bit allocation could be achieved in cbr with a bit reservoir (or whatever its name) that would be the size of your given time slice.

IE: if allowed instantaneous variation is big enough, VBR does not provide any advantage while trying to keep a resonable decoding buffer and a given average bitrate.

question: what is the typical aac buffer size for cbr @48kbps?

edit: bytes -> bits

This is true. The question is: do we need to ensure streaming conditions for 48 kbps encodings? If not, I don't see any reason to arbitrary discard VBR. Or is there something special with 48 kbps compared to 128 or 32 kbps?

Saving bits is the idea of lossy encoding in general (from PCM to encoded file). And VBR is a tool to gain additional space (by maintaining quality at lower bitrate or increasing quality at the same bitrate than CBR).
Consequently, if you're interesting by efficiency (and you are apparently) you should be interested by VBR.

ISO AAC specifies CBR bit buffer constrains as:

(MAXIMUM_FRAME_SIZE * number_of_channels) - average_frame_length;

Maximum frame size in AAC is 6144 bits

So,

For 48 kHz, Stereo - and 128 kbps AAC bit rate maximum bit reservoir size is 10240 bits

For 22.05 kHz, Stereo (HE-AAC, AAC operates at fs/2) and 48 kbps it should be 7830 bits.

Note, basically you could speficy much bigger bit reservoirs - but then it would take much longer for the decoders to pre-buffer.

100% agreed

Something is worrying me.
The purpose of this test is to select the best AAC implementation/profile at 48 kbps, in order to get the best contender for the upcoming multiformat listening test at the same bitrate, right?

Shouldn't be fair to do the same for other competitors? I especially have Vorbis in mind. CVS or Aoyumi? resampled -q-1 or orginal one? Several people have reported here than resampling help a lot Vorbis. I didn't really test it myself, but the few encodings I've done confirmed it (resampling -> less distortions).
In other words, there are different settings for Vorbis. Shouldn't we also perform a pre-listening test to get the best vorbis competitor? Could we consider as fair a dedicated pool for AAC but not for others? Personnaly, I wouldn't. As long as different encoders/settings could pretend to be the best, pools (preliminary listening tests) are needed.
Such pool is necessary for AAC, but I'd say that there elements enough to legitimate such pool for Vorbis (CVS-aoTuV / resampling or not / which sampling rate / ?VBR-ABR?).

I don't mind preparing a Vorbis + MP3 test at this same bitrate; Testing FHG, aTouV, LAME and regular vorbis at different settings (default, and resampled, I think, should be enough..)

48kbps Streaming IS reasonable :

http://www.polskastacja.pl/

I can confirm that it works (at least here in Germany, but don't ask me what provider since it was on a mate's PC).



I think single channel ISDN has 64.

http://www.tuner2.com/ has a lot of 48 kbps streams.