Whatever the encoder decides to use.

Bitrate table (classical music only, 150 tracks, 16hours of music):


I recall that it corresponds to classical music and that several encoders tend to use less bits with such material (with VBR of course).


Sebastian, in my opinion you should stay with your original idea and use a 4+2 basis for your test: 4 competitors and 2 anchors.
I recall to everyone that the purpose of multiple encoders listening test is not to see if people are able to distinguish or not the competitors, but the goal is mainly to rate them all accurately. It supposes several cross-comparisons between all files. It's long, it's boring, it's usually much harder than any ABX comparison. The more competitors (+ anchors) you have the longer it takes.

The 2 anchors are needed, especially for a 48 kbps comparison; they are as important if not more than any other competiting encoder. 4 competitors should therefore be the maximum allowed.

The proposed formats are:
• AAC-HE (two implementations are suggested)
• Vorbis aoTuV VBR
• WMA VBR
• WMAPro CBR
• Real Audio
• Atrac3+

My opinion is:
-> one HE-AAC format and certainly not two. If people missed a part of the story, I recall that our last listening test was a test dedicated to HE-AAC at 48 kbps. We know that all implementations (including the ISO code) are close each others. There's strictly no need to compare them another time. I would use Nero, because:
- it allows VBR (2 advantages: it's supposed to be better than CBR and it will makes bitrate tuning easier)
- it ended with the bast mark on the last test (first ex-æquo though)

-> Real Audio and atrac3+: I see several reasons to not priviliged these formats. I would simply remind that both Helix and Sony are now supporting MPEG-4 Audio and I suspect both company to not give importance anymore to their closed formats.

Is there any confirmation that the Zune does not play WMAPro? Anyway, Windows Mobile 2005 does, and since millions of PDA's are used as DAP this remains an interesting codec. Also, I really question the need for testing WMA standard. I think we all know that it will finish last, it's a previous-generation codec and IMO just as pointless to test as MP3 or LC-AAC. WMAPro is its successor, every current Microsoft-supported platform (WMP/Flip4Mac/Windows Mobile) plays it. If you insist on adding it, then as a low anchor.

I agree with guruboolez, I question the need for ATRAC3+, as even Sony doesn't really promote it anymore. Like vqf and mp3pro, the ship to the future has sailed and this format is not on it.

RealAudio is indeed HE-AAC in a Real container, but it does not use either the CT or Nero implementation. I'd reckon that in terms of audience, RealAudio (through all the webradio stations using it) is used a lot more than any other HE-AAC encoder - so testing it would be relevant. If it cannot encode at 48k, then discard it, but if it can (come close) it's a valid contender.

My proposal would be 3 state-of-the-art codecs and 5 implementations:
- WMAPro (as MS will not give a preference, VBR if it comes close to 48k, CBR if it does not)
- Vorbis aoTuV VBR
- HE-AAC by RealAudio (if 48k is at all possible?)
- HE-AAC by Nero (ABR if Ivan indeed says so)
- HE-AAC by CT (VBR or CBR?)

High anchor: LC-AAC @ 128k
Low anchor: LC-AAC @ 48k or WMA-std @ 48k

http://www.realnetworks.com/company/press/...codingtech.html
http://wiki.hydrogenaudio.org/index.php?ti...implementations


I think that Real still uses its own format (Real Audio, not MPEG-4 Audio) for low bitrate or for some applications.

I have no confirmation or announce or information about this. But according to the Official Zune Site, They say only WMA Std (and they don't say aacPlus or Advance Profile of wmv Support in this site too).
Don't trust me, maybe I'm violate some crappy TOS. Who's care?


Edit: Out of topic again, I can't wait for new windows mobile. And don't be sad, there's still a hope for firmware that made by M$ fanboy.

RealAudio 10 (their latest version) has the following codecs (the 4 letters are the codec ID in the RealMedia container):
raac: MPEG-4 LC-AAC
racp: MPEG-4 HE-AAC
ralf: RealAudio Lossless Format

RealAudio 8 had ATRAC3, and the last proprietary codec they had was the "Cook" codec in RA 6. See also http://en.wikipedia.org/wiki/RealAudio

Thanks. But did you specifically try with low bitrate?
According to the official site, AAC is used for 128 kbps and more:
http://www.realnetworks.com/products/codecs/realaudio.html


EDIT: I mean RealAudio. A standalone encoder (HELIX) exists and allows HE-AAC at 48 kbps.

My vote for contenders :

• Nero HE-AAC
• aoTuV b5
• WMA PRO 10

my vote for anchors:

• low: Apple LC-AAC
• high: LAME @128kbps

Well, problem with Real is that it uses 44 kbps CBR. Not only is CBR a disatvantage, but the lower bitrate, too. Therefore, I see no point testing it.

If the changes in Nero HE-AAC and CT HE-AAC aren't very big since the last listening test, I also don't see need for testing both again. Therefore, I would go with Nero HE-AAC ABR.

ATRAC3+ might be interesting for people using discmans supporting this format. As far as I know, ATRAC3+ is also gapless.

Guru, thanks for your bitrate table. Unfortunately, according to my tests based on Progressive Rock and New Wave music, the bitrate for WMA Professional 10, Q10 1-pass VBR tends to be higher than 60 kbps which is pretty much. Therefore, I think only CBR can be used in this case. BTW, does anyone know the point of 2-pass CBR?

The CT encoder has gone through a major revision since the last listening test (version 7.2 versus 8.0). CT has said that there should be improvements, and would be interested in seeing the results of a new test.

Hmm... I am really wondering if ATRAC3+ should be tested. I know that MiniDiscs had a quite large market share in Asia, but does anyone know the situation with ATRAC3+ CD and HDD/Flash players?

LAME@128 is too high anchor for 48 kbits. LAME@96 kbps would be more comparable.

I don't think so...or try to explain why.


FYI, LAME@128kbps was used in the last AAC @48kbps listening tests.

An anchor has to be clearly discernible from the rest of the test cases, not comparable. In this regard, HE-AACv2 at 48 could sound in par or better than LAME@96 (think about the lowpass also).

This is not an AAC but multiformat listening tests...Only one AAC encoder is enough imo.

Ok. A few month ago I did internal (not enable in public) 64 kbps ABX test just for me. With Lame -V 5.
The score for lame was 4.5 and for better (in my test) codec wma10pro 64 cbr was 3.5 .
So LAME@128 comparable to 64-96 kbps for modern codecs. And LAME@96 should be comparable to modern codecs at 48 kbps.

Multi AAC test was completed enough time ago. It's a reason why new CT HE-AAC encoder can be included.

I'm agree.



Yes, it could on some samples. Average quality?That's why it needs to be tested.
LAME@96 has something like 12-14 khz lowpass and HE-AAC has only SBR part at freq. higher than 11-12 khz.

Igor, you know it's illegal to post WMA 10 Beta results, don't you?

Personally, I would go with LC-AAC 96 kbps as high anchor.

Good one . That's why it was internal test.


Sounds good.

I don't get what 2-pass CBR should be good for. If it's CBR, it should always encode at a given bitrate regardless of input complexity. If nobody gives good arguments for 2-pass CBR, I will use 1-pass CBR for WMA Professional 10 (default encoding mode therefore).

I decided to go against Guru's recommendation ( ) and to include CT HE-AAC. benski said that CT would be interested and it seems Ivan would like to see it, too.
So, we have 5 contenders now: Nero Digital HE-AAC, CT HE-AAC, Ogg Vorbis AoTuV, WMA Standard and WMA Professional. Beside this, we also have two anchors: iTunes LC-AAC 48 kbps as low anchor and iTunes LC-AAC 96 kbps as high anchor.

Real will not be included for sure.
ATRAC3+ is something I have to think about. Six contenders is pretty much and ATRAC3+ is a dead format. The only thing that makes me hesitate is that it supports gapless playback and some people might use it in portable players (while MP3 players are not very expensive nowadays, they don't support gapless playback which might be useful for some people like me - which is why I still have a MD player which I share with my dad).

I can't imagine many people still using ATRAC just because of gapless. Not now so much has happened regarding gapless lately. Ogg is gapless, AAC made by the two most common encoders is gapless and MP3 can be considered gapless when made by the most popular encoder. Even the worlds most popular DAP (the iPod, whether you like it or not) now supports gapless playback of AAC and MP3.

Two years ago, I would have agreed, nowadays I can't imagine many people still choose ATRAC because of this.

With gapless, I also mean the portable player supporting gapless playback, not only the format itself. So far, the only gapless players I know are iPod, Rio Karma and ATRAC players (I did not count Rockbox players).

I hope I won't appear as a bad loser, but could you explain what was the point of a listening test dedicated to a single format if we ignore the results few months later?
What do you expect from the updated encoder? Our previous listening test revealed that the ISO encoder is close (and statistically the same quality...) than all highly tuned HE-AAC implementations. Unless Coding Technologies brought revolutionnary tunings on their encoder, my hand on fire that the final result will end on an identical quality between Nero Digital and Coding Technology's one.
Yes, I'm also confident that Ivan and CT's guys are interested in such test. But it's a multifomat listening test, not HE-AAC contest, part 2. It's not like we were in the darkness about HE-AAC different implementation quality.

So, please, keep one HE-AAC implementation (use a poll if needed) and stay far away from the inflation of competitors. I recall that the more competitors have the higher is the statistical noise. It means: higher risks of "tied" results. May I remind that the two previous listening tests ended on "everything is tied". The risk to get for the third consecutive time the same boring conclusion is even higher if few people are participating to the test, discouradged by the difficulty of rating 7 or 8 encoders per sample. With 6 files (including two anchors) it's already hard enough. We can't test everything, so please...

I strongly agree with guruboolez. In the poll, I voted for "multiformat 48kbps, will participate", but seeing so many contenders mentioned, I'm starting to quickly have doubts about participating. My 2 cents are that we could have:

Aotuv
Nero HE-AAC v.1
WMA Pro
WMA std
and anchors

guruboolez, we can always do a shorter pre-test for CT and Nero... and pick the winner of said pre-test for this multiformat test.

IIRC, WMA 2-pass CBR is actually ABR.

I agree with guruboolez and some listener of previous 2 test also will, I think. In previous test, we got 2 conclusion: a) all HE-AAC encoders@48kbps are tied and b) 48kbps AAC is definitely not as good as 128kbps.



We don't afford 2nd HE-AAC comparison.


As for the ATRAC thing, SONY gave up the hope of spreading own proprietary codec any longer. They began to implement another codecs like AAC and WMA. Minidisc has been replaced by flash player or something so that codec doesn't matter outside minidisc. The demand for ATRAC suite is going to decrease.

No time for this. A pre-test would take some time and you have to consider Christmas and New Year where people won't spend time testing codecs.

As for ATRAC3+, I agree it makes no point testing it since it would be another contender and even Sony pretty much gave it up.

Now to the HE-AAC question... benski mentioned a quite big version jump which might be caused by big improvement. I am not sure if Nero improved much since the last listening test - I doubt it. Thing is that we had an AAC listening test already and a new one won't follow soon probably. Guru is right about the two HE-AAC contenders being more or less tied. How about doing some private tests after New Year and publishing the results? In that way, we can see if any progress was made and the multiformat test now would only feature 4+2 encoders.



I always thought ABR was the 2-pass bitrate based VBR.

I'm glad to see that I'm not the only one worried by the possible huge amount of competitors.
____
My point is simple: all listening tests involving living formats are quickly outdated. Every new release of any encoder limits the significance of the latest results. As a consequence if we bother to start again a new comparison each time an encoder is updated, we will pass our time to... update our own tests.

The existence of a new multiformat listening test shouldn't be used to update the result of a fresh one dedicated to a single format. I understand that people are worried by the fact that both encoders (Coding Tech but also more marginally Nero Digital) were updated since last winter and that we can't now rely to 100% to the past conclusion. But even if we try a second time to check the quality of the main HE-AAC contenders, our up-to-date results will appear as pointless in a few months (new updates from Nero, CT, maybe Apple, etc...). That's life. We should immediately accept the ephemeral side of all results and use the last ones as starting point rather than spending additional energy to update our knowledge for a specific purpose.
In any case, HE-AAC will stay one main actor in the audiocoding landscape for many years. We will still have the opportunity to test it in the future.

I would suggest to make it the other way: use either Nero or CT for the multiformat test and when the results of the multiformat test show some anomalies of AAC compared to the other formats, then check to see if it's a specific problem of the former or latter encoder.

My preference for this test is:
- Nero's HE-AAC encoder
- Ogg Vorbis AoTuV
- WMA Pro
+ some arbitrary low anchor
+ some arbitrary high anchor

If WMA Pro reaches average bitrates around 55 +- 5 kbps on a large sample library of different musical styles, then tune the other contenders settings to reach a similar bitrate on the same sample library.

My fear is, more encoders under test will result in less precise conclusions to draw from this test.

Where did you get this information?

I have always wondered how WMA 2-pass CBR works. Obviously it analyses the complete file on the first pass and adjusts encoding parameters during the second pass. I have not found a clear explanation what exactly it adjusts. Does it work like a bitrate restricted 2-pass VBR mode or does it keep the CBR bitrate practically constant (to avoid streaming bandwidth problems) and adjusts other parameters (that are unkown to me) like stereo mode, lowpass, perhaps things that are smilar to MP3's bitreservoir and block switching and so on.

I also find it very weird since there is a special mode called bitrate managed VBR which is also 2-pass. Beside this, we have 1-pass CBR, peak bitrate VBR (which will not use bitrates higher than the target bitrate, but will allow encoding at lower bitrate for certain frames AFAIK) and free VBR which is similar to Vorbis (quality based).



Well, I noticed that Q10 reaches bitrates close to 64 kbps so adjusting the other encoders to also reach this bitrate will turn this test into a multiformat test at around 64 kbps.

Microsoft selected WMA Pro CBR 64 kbps to be used in the listening test, which was discussed here: http://www.hydrogenaudio.org/forums/index....showtopic=40589 (The test report at MS site: http://download.microsoft.com/download/4/b...-v2%20final.pdf)

Now MS has selected the CBR mode to be the only available WMA Pro option in WMP11.

In my opinion this makes the CBR mode recommended by the MS developers. Why wouldn't they select an optimal encoding mode for their new WMP version especially because they have made claims about the enhanced WMA Pro quality? It should be obvious that the users start testing the available WMP11 encoding settings after the release.

Edit

On a second thought, I have no idea what mode WMP11 uses for WMP Pro @ 48 kbps

Actually, you could use WMP11 for encoding the samples instead of using other methods that are possibly not exactly recommended by MS. You could mount an ISO image and rip the samples with WMP11.

Edit

Since we cannot be sure what exact mode WMP11 uses it would be best to let it decide. It has an option for WMP Pro @ 48 kbps.

This cannot be done because it might use bitrate managed 2-pass VBR.

Mp3infp says files encoded by WMP11 are 1-pass CBR.

(It correctly identifies 1 and 2-pass CBR as well as 2-pass VBR (encoded with dBPowerAMP))

What was the problem with 2-pass VBR? (Sorry, I missed the corresponding discussion )

Results would not correspond to real-world usage when using short clips as samples. One would have to use full tracks. Problem is this: let's say you have a song that has these parts: ABAB. Consider A being very difficult to encode and B being pretty easy to encode. If you use 2-pass VBR with an average bitrate of 160 kbps, A would get 256 kbps and B would get 64 kbps. Bitrate will then be (256+64+256+64)/4=160. However, if you extract part A and encode it as sample with the same target bitrate of 160 kbps, part A would have to be encoded with 160 kbps; this 96 kbps less than what it would get if it was part of a full song. 1-pass VBR doesn't have this flaw because "free" VBR is not bound to any target bitrate; it will always allocate the same number of bits.

Even CBR is better in such a case because similar to VBR which maintains a constant quality, CBR will always use the same number of bits regardless of the complexity of the song. With 2-pass VBR, you get different results depending on the rest of the material.

My question now: isn't Nero ABR affected by the same problem?

It depends on how big the ABR "window" is. I guess a Nero dev should answer.

Some fun with WMAPro 10 CBR 48 kbps.
I create an ISO file and mounted it on my system. Then I 'ripped' from the HDD one track with three different applications:
• VUplayer (2.44)
• Winamp (5.31)
• WMP 11
The same setting was used: CBR 48 kbps (16 bits when it was mentionned)

I expected bit-to-bit identical results but this is what I get (with foobar2000 0.9.4.1):

=> VUPlayer and Winamp are obviously creating different outputs.


=> different length between VUplayer & WMP11


=> different length between Winamp & WMP11

In other words, three different applications = three different files...


Technical information for all files (from foobar2000 & mp3inf):


As you can see there's even different length [6130 samples] between VUPlayer & Winamp (difference which wasn't reported on fb2k's log).

Any explanation?


EDIT No.1

Audition gives me AFTER decoding the same exact length for both Winamp and VUplayer 'WMA' files: 11416020 samples. But the previewed 'file information' box of Audition says (BEFORE decoding) "4:18.901" for VUplayer and "4:19.040" for Winamp.

EDIT No2

Audition gives me AFTER decoding 11415432 samples for WMP11 file. It's slightly shorter. Are samples missing with WMP11 or are samples added with VUPlayer / WInamp?
I'll investigate later.

We need now to test this new CT encoder form Winamp 5.3 an newer.

I think I know better now. I ripped a few test tracks with WMP11. Obviously it uses CBR 1-pass or at least it writes that info to the file header.

Hex editor:


JRMC tag dump:

Perhaps those programs detect track starting points differently. You could rip in wave format for testing that.

Edit

I ripped a test track from a scratchless CD with WMP11 (digital, error correction enabled) and EAC (secure mode, offset correction 0) in wave format. Foobar's Bit Compare didn't find differences.

WMP encoded file was 588 samples shorter (IIRC it corresponds to one CD sector). I add 588 samples at the end of the decoded WMP files, and the bit-comparison gives me this:




=> WMP, Winamp & VUPlayer are all giving the same output... on the beginning only. Then difference are visible. I did a substraction, and the difference I saw was similar to the one you can get with different compilation of the same encoder: some packets are identical, some aren't. I didn't notice any quality issue with one of these encoder.
I'll probably try with a real CD to confirm this phenomenon.



Alex B:
ISO ripping in WAV: there are length difference between WMP & Winamp... foobar2000 reports this with the bit-to-bit comparator and it can be confirmed by the properties of the file. But... if I open both files with Audition, the total number of samples for both files is absolutely identical. Do you understand something? To increase the difficulty, the "mix-paste" function reveals that both streams doesn't match: it introduces a kind of reverberation. After investigation, the WMP stream is shifted by +11 samples.
It's crazy...

Thanks. By the way, there is "Windows Media File Editor" utility from Windows Media Encoder suite that can trim WMA files.

I suggest testing 1-pass only because 48k content is supposed to be for streaming purposes and hardly for playback on mobile devices. Streaming obviously implies 1-pass only.

Guru - it might help to take CD ripping engine out of the equation for Winamp, and use the transcoder - http://www.srcf.ucam.org/~wdhf2/transcoder/ - to convert to WMA. Certainly one less thing to worry about effecting the results.

Oh no, hopefully not a can of worms.

I've just tried with a real CD and the same 'issues' occur:
- length mismatch even while ripping in PCM format
- different WMA output.

>benski - I'll try.
Result: I can't get identical WMA output. Winamp still gives me a different file from WMP whether using the transcoder or the ripping engine.
The WMA encoding produced by Winamp's ripper is identical to the one produced by the transcoder (with fb2k .wav ripping as source). In other words, winamp ripping engine can't apparently be blame. May the problem comes from WMP ripping engine? Or from my drive (LG 4163)? Maybe.

Could someone try to get two identical wma files with two different applications?
Could someone

Will do Guru...

Ripped by Windows Media Player 11:



Ripped by Winamp 5.31:



Will also do a bit compare now...

Edit:



Damn, I just encoded a WAV to WMA using the VB script and that file doesn't match any of the two others either.