Pre-echo: Castanets, --api vs. iTunes AAC @ 320, Interesting Results! Options

[Cygnus X1]

I've played around with iTunes AAC but ended up settling on ALAC, because I had this nagging feeling that I was hearing pre-echo on very sharp attacks that I did not hear with LAME. Tonight, I fired up Virtual PC (Java ABC/HR is goofy on my machine) and did two quick ABX tests using a castanets sample, encoded by LAME 3.96.1 and iTunes 4.6, both @ 320kbps CBR.

(Edit: re-ran test for 16 trials each. Not done sequentially as to avoid alpha inflation (increased Type I error rate).

The results for AAC @320:

1R = C:\Documents and Settings\Angelo\Desktop\castanets aac.wav

---------------------------------------
General Comments:

---------------------------------------
ABX Results:
Original vs C:\Documents and Settings\Angelo\Desktop\castanets aac.wav
16 out of 16, pval < 0.001


Now for LAME:


1R = C:\Documents and Settings\Angelo\Desktop\castanets insane 320.wav

---------------------------------------
General Comments:

---------------------------------------
ABX Results:
Original vs C:\Documents and Settings\Angelo\Desktop\castanets insane 320.wav
11 out of 16, pval = 0.105

This is VERY surprising. Actually, I almost hit significance at the .05 level and somehow lost my concentration or couldn't tell the difference anymore (just like the last time). In any case, this was much harder than the AAC test. So, what's wrong with my ears...aren't I supposed to be able to perceive pre-echo better with MP3 as opposed to AAC?

And finally, LAME vs. AAC:

1L = C:\Documents and Settings\Angelo\Desktop\castanets aac.wav

---------------------------------------
General Comments:

---------------------------------------
ABX Results:
Original vs C:\Documents and Settings\Angelo\Desktop\castanets aac.wav
16 out of 16, pval < 0.001

Results: I could tell the AAC encoding apart from the original, p<.001. I could NOT tell LAME --insane from the original, p>.10. And finally, I could tell the LAME encoding apart from the AAC one, p<.001.

This post has been edited by Cygnus X1: Jul 29 2004, 04:07

[westgroveg]

It would be more interesting if you did a 16/16 or more test to be sure Lame was really better.

aren't I supposed to be able to perceive pre-echo better with MP3 as opposed to AAC?

Not all MP3 encoders are created equally.

[Cygnus X1]

It would be more interesting if you did a 16/16 or more test to be sure Lame was really better.

aren't I supposed to be able to perceive pre-echo better with MP3 as opposed to AAC?

Not all MP3 encoders are created equally.

I think that can be arranged. I'm assuming you mean 16 trials of each codec compared to the reference? I also might try comparing LAME to the AAC file as well...I've already established that I can hear a difference with the AAC file, but not LAME, so that might be interesting.

[westgroveg]

It would be more interesting if you did a 16/16 or more test to be sure Lame was really better.

aren't I supposed to be able to perceive pre-echo better with MP3 as opposed to AAC?

Not all MP3 encoders are created equally.

I think that can be arranged. I'm assuming you mean 16 trials of each codec compared to the reference? I also might try comparing LAME to the AAC file as well...I've already established that I can hear a difference with the AAC file, but not LAME, so that might be interesting.

Yes 16 trials, I think it would also be interesting & I would apprecate it, while your at it to chuck Nero's AAC encoder & FhG fastenc -hq.

[Cygnus X1]

Ok, have a look at my edited results...significance level increased to p<.001.

By the way, I'm on OS X and don't have a copy of Nero to run on VPC. Does the trial version of Nero come with the AAC encoder?

[Garf]

Yes.

[Garf]

So, what's wrong with my ears...aren't I supposed to be able to perceive pre-echo better with MP3 as opposed to AAC?

MP3 encoders, notably LAME with presets, have had much more tuning than the relatively newer AAC encoders. So stuff like this can happen, but it should not be the general case.

This post has been edited by Garf: Jul 29 2004, 18:41

[guruboolez]

Your results surprised me. I did some pre-echo tests in the past with QT AAC encoder, and results were really good. MP3 have inherent flaws, and honestly, I would be surprised if one day one MP3 encoder could achieve this quality, even at 640 kbps. Maybe is latest iTunes encoder broken somewhere? To be sure, I've quickly compared iTunes 4.6 AAC at 320 kbps to lame 3.97a3 at the same bitrate.

CODE

foo_abx v1.2 report
foobar2000 v0.8.3
2004/07/30 02:31:34

File A: file://C:\ABX\castanets-lame3.97a3-320.mp3
File B: file://C:\ABX\castanets-iTunes320.m4a

02:31:35 : Test started.
02:32:00 : 01/01  50.0%
02:32:08 : 02/02  25.0%
02:32:16 : 03/03  12.5%
02:32:24 : 04/04  6.3%
02:32:31 : 05/05  3.1%
02:32:43 : 06/06  1.6%
02:32:50 : 07/07  0.8%
02:32:54 : 08/08  0.4%
02:32:58 : 09/09  0.2%
02:33:02 : 10/10  0.1%
02:33:07 : 11/11  0.0%
02:33:11 : 12/12  0.0%
02:33:15 : 13/13  0.0%
02:33:20 : 14/14  0.0%
02:33:24 : 15/15  0.0%
02:33:29 : 16/16  0.0%
02:33:33 : 17/17  0.0%
02:33:38 : 18/18  0.0%
02:33:42 : 19/19  0.0%
02:33:47 : 20/20  0.0%
02:33:49 : Test finished.

----------
Total: 20/20 (0.0%)

This test was very easy. The first file (MP3) has serious smearing (first three attacks - the rest is less disturbing). The second one sounds natural to my ears. I'm not sure that I can ABX it against the reference (I'm too tired now). This AAC encoding was very sharp, and I don't ear any other problems.

[westgroveg]

guruboolez, the best way to compare your results with Cygnus would have been to use lame 3.96 not an alpha.

This post has been edited by westgroveg: Jul 30 2004, 02:55

[Cygnus X1]

Maybe is latest iTunes encoder broken somewhere? To be sure, I've quickly compared iTunes 4.6 AAC at 320 kbps to lame 3.97a3 at the same bitrate.

I think that's quite possible. You're using QT 6.5 to encode, right? Also, I used LAME 3.96 as opposed to 3.97a3. I will upload my samples for you...see if my 320kbps AAC file sounds different.

What I was hearing in the AAC file was a smearing/combining of a few "clicks" after the 2.5 second mark, along with a sort of flanging that went from L to R, whereas the original and MP3 file seemed to emanate from the "front" of the channel and the smearing was harder to recognize.

[Cygnus X1]

I think I just solved the mystery, and the news is not good.

I was just playing around with samples, using QT instead of iTunes. When played through QT itself, the AAC file exhibits NO artifacts whatsoever. However, for the ABX test, I decoded the AAC back to wav in iTunes. The wav file decoded by iTunes has artifacts that the AAC file does not!!

Worse still, I have AAC files encoded with PsyTel --extreme that play back as though half the frames are missing! They play back fine in Real. Let me run a quick ABX test to make sure that the decoded file and the AAC file really do sound different. If so, this means that iTunes has some problems with AAC decoding!

[Cygnus X1]

Sure enough, when decoded using iTunes 4.6, the 320kbps AAC file has artifacts that give it away every time, and the one decoded through QT sounds perfect.

Sample A: castanets aac.wav
Sample B: castanets aac iTunes.wav

1 of 1, p = 0.5
2 of 2, p = 0.25
3 of 3, p = 0.125
4 of 4, p = 0.062
5 of 5, p = 0.031
6 of 6, p = 0.015
7 of 7, p = 0.0070
8 of 8, p = 0.0030
9 of 9, p = 0.0010
10 of 10, p = <.001
11 of 11, p = <.001
12 of 12, p = <.001
13 of 13, p = <.001
14 of 14, p = <.001
15 of 15, p = <.001
16 of 16, p = <.001

Looks like I just found a bug in iTunes

[guruboolez]

Can't sleep
I could confirm the artifact (something electric/metallic in the middle of the castenet's bunch: very obvious). More strangely, the first four attacks (and not three as I said previously) are also unsharp compared to faad2 decoding (foobar2000). Something like pre-echo/hollow sound. It's not very pronounced, but not very hard to hear. I'll ABX it tomorrow to be really sure (I'm between reality and dream on the moment ).

[Cygnus X1]

I'll ABX it tomorrow to be really sure (I'm between reality and dream on the moment ).

Hopefully, you'll not hear castanets in your dreams I myself have to be especially careful about what I listen to before bedtime!

[guruboolez]

It wasn't a hallucination. iTunes decoding has less sharpness than faad2 on the first four attacks of castanets.wav sample. Difference is in my opinon realy obvious (90 seconds only for a 21/21 score - listening range was 0.0 - 1.5). It's the first time I can hear pre-echo introduced by a DEcoder.
Did someone (Menno ) have an explanation? TNS decoding?

CODE

foo_abx v1.2 report
foobar2000 v0.8.3
2004/07/30 13:32:49

File A: file://C:\ABX\castanets-iTunes320.m4a
File B: file://C:\ABX\castanets-iTunes_decoding.wav

13:32:51 : Test started.
13:32:56 : 01/01  50.0%
13:33:00 : 02/02  25.0%
13:33:04 : 03/03  12.5%
13:33:08 : 04/04  6.3%
13:33:11 : 05/05  3.1%
13:33:16 : 06/06  1.6%
13:33:20 : 07/07  0.8%
13:33:24 : 08/08  0.4%
13:33:27 : 09/09  0.2%
13:33:32 : 10/10  0.1%
13:33:36 : 11/11  0.0%
13:33:39 : 12/12  0.0%
13:33:43 : 13/13  0.0%
13:33:46 : 14/14  0.0%
13:33:50 : 15/15  0.0%
13:33:54 : 16/16  0.0%
13:33:57 : 17/17  0.0%
13:34:01 : 18/18  0.0%
13:34:05 : 19/19  0.0%
13:34:09 : 20/20  0.0%
13:34:14 : 21/21  0.0%
13:34:15 : Test finished.

----------
Total: 21/21 (0.0%)

[guruboolez]

People might be interested to note that iTunes is not alone to suffer from this problem. I've just tried on a fresh installed Winamp 5.04 (no additionnal component like faad2), and the castanet.mp4 file exhibits the same problems (lack of sharpness, artifact ["clicks"] in the middle).
More strangely, if the same file is demuxed in a raw .aac form, Winamp 5.04 could play it flawlessly.

Can just say that: many thanks to menno for his great job on faad2.1

[Garf]

Apple's encoder is, IIRC, originally based on Dolby's. Likely, their decoder is also. The Dolby decoder has known bugs with TNS, and is used in Winamp too, when decoding .MP4.

When winamp decodes .AAC's, it uses the CodingTech decoder which does not have this bug. FAAD does not have this bug either.

This post has been edited by Garf: Jul 30 2004, 13:23

[menno]

Confirmed that iTunes does not produce proper output. It doesn't look like the same bug as the Dolby decoder in Winamp has. The problem here seems to be in the frequencies below 1000 Hz.

Menno

[Cygnus X1]

So, it sounds like the pre-echo problems I thought were attributed to an encoder flaw turned out to be a decoder problem, NOT a problem with QT AAC at 320kbps. I wonder if this occurs at other bitrates as well?

Later today, I can conduct more ABX tests at different bitrates, comparing QT decoded AAC to that of iTunes. Let me know if you think this would be helpful.

This post has been edited by Cygnus X1: Jul 30 2004, 15:15

[loophole]

It seems strange that QT would use a different decoder to iTunes...

[wkwai]

Confirmed that iTunes does not produce proper output. It doesn't look like the same bug as the Dolby decoder in Winamp has. The problem here seems to be in the frequencies below 1000 Hz.

Menno

Have you try conducting conformance tests on the iTunes decoder?

wkwai

[Cygnus X1]

I recently did a clean install of OS X 10.3 in order to go back to QT 6.4 and iTunes 4.2....the decoder problems are gone. Both iTunes encoded files and PsyTel VBR play back without any of the very obvious noise and glitches that I saw in 6.5.1/ iTunes 4.6. I am afraid of upgrading either iTunes or QT until Apple comes out with a version that corrects the decoding AND encoding glitches seen in the latest version.