Nero 48 kbps HE-AAC (+v2) test - FINISHED! Options

[Ivan Dimkovic]

Okay, so results of the Nero Listening test are here:

Test1: HE-AAC v1 @48 kbps

VBR1 - 3.56
ABR 48 - 3.53
CBR 48 - 3.48
VBR2 - 3.29
MP3 128 - 4.41
MP3 48 - 1.19




Test2: HE-AAC v2 @48 kbps

VBR1 - 3.12
ABR 48 - 3.11
CBR 48 - 3.11
VBR2 - 3.08
MP3 128 - 4.08
MP3 48 - 1.14



This post has been edited by Ivan Dimkovic: Feb 6 2006, 01:16

Attached thumbnail(s)

 

[Ivan Dimkovic]

And... the test keys (attached with this post)

Codec IDs:

TEST 1 - HE TEST

Codec A - ABR 48 kbps
CODEC B - VBR1 - quality factor 0.15
CODEC C - VBR2 - quality factor 0.37
CODEC D - CBR 48 kbps
CODEC E - MP3 128 kbps
CODEC F - MP3 48 kbps

TEST 2 - HE v2 TEST

Codec A - ABR 48 kbps
CODEC B - VBR1 - quality factor 0.32
CODEC C - VBR2 - quality factor 0.54
CODEC D - CBR 48 kbps
CODEC E - MP3 128 kbps
CODEC F - MP3 48 kbps

This post has been edited by Ivan Dimkovic: Feb 6 2006, 01:40

Attached File(s)

 nero_test_keys.zip ( 1.86K ) Number of downloads: 415

 

[Alex B]

I tried only 2x3 samples (sorry, I didn't have more time) and got these results:

CODE

Test1    Sandman      Sympathy     Harp40_1     Average
A         2.00         1.70         2.70         2.13
B         2.40         1.90         2.70         2.33
C         2.20         1.90         2.00         2.03
D         2.00         1.50         2.30         1.93
E         4.30         4.50         1.40         3.40
F         1.00         1.00         1.00         1.00

Test2    Sandman      Sympathy     Harp40_1     Average
A         2.60         2.10         3.50         2.73
B         2.80         1.80         3.50         2.70
C         2.20         2.00         3.50         2.57
D         2.20         2.10         2.90         2.40
E         4.30         4.50         1.40         3.40
F         1.00         1.00         1.00         1.00

I would say that all contenders were pretty much tied with the Sandman and Sympathy sample.

The Harp40 _1 sample was quite interesting. Because the high anchor has a clear problem with this sample the AAC encoders were much better. In the Test 1, A and B settings didn't produce as ugly artifacts as C and D with this sample. This was actually the only obvious difference I heard in any of these 2x3 AAC samples.

Also, In general I would say that the encoder in Test 2 was slightly, but audibly better than the encoder in the Test 1 with all samples.

BTW, the LAME -b 128 / Harp40_1 problem is the worst I have ever heard!


Edit: fixed the Test2 heading in the table.

This post has been edited by Alex B: Feb 6 2006, 10:17

[Ivan Dimkovic]

Also, In general I would say that the encoder in Test 2 was slightly, but audibly better than the encoder in the Test 1 with all samples.

Interesting, as it seems - although, outside of the scope of the tests, pn average PS was ranked significantly lower than HE - tests were indeed separated, but the anchors were the same.

It is also worth noting that LAME -b128 was ranked 4.41 on the first test, and only 4.08 at the second one.

Dunno what exactly caused this difference - but I think the tests cannot be directly compared (as it was planned), and the upcoming 48 kbps multivendor AAC test will finally give the most accurate answer of HE-AAC v1 vs. V2 ranking.

[Garf]

Dunno what exactly caused this difference

IMHO It's almost certainly listener training which improves after performing test1.

[pepoluan]

Reflected in the Listening Tests wiki page. Please check if any error.

Oh and also changed the Current Events page.

[Alex B]

Interesting, as it seems - although, outside of the scope of the tests, pn average PS was ranked significantly lower than HE - tests were indeed separated, but the anchors were the same.

It is also worth noting that LAME -b128 was ranked 4.41 on the first test, and only 4.08 at the second one.

Dunno what exactly caused this difference - but I think the tests cannot be directly compared (as it was planned), and the upcoming 48 kbps multivendor AAC test will finally give the most accurate answer of HE-AAC v1  vs. V2 ranking.

I tested only 6 of the samples. How did the different samples score? Is there any pattern? Also, at one stage you had more results from the Test1. Did that change? If not, would it be possible that those who send also Test2 results were more experienced listeners, thus more critical in general?

Listener training is probably a big factor, if most of the testers started with Test1 and then continued with Test2.

Perhaps my test order eliminated the listener training effect in my case. It was this:

1. Sandman Test2
2. Sandman Test1
3. Sympathy Test2
4. Sympathy Test1
5. Harp40_1 Test1
6. Harp40_1 Test2

I did all testing in the same evening. It took about 5 hours, so I spent about 8 minutes with each codec/setting (including a couple of short breaks). Most of time I tried to put the AAC encoders in order and that was pretty tough. The AAC samples had more or less subtle differences, but it was not easy to decide which artifacts are less annoying. I listened the samples over and over again until I was pretty sure about the results. All 28 samples would have taken about 24 hours to test at this testing speed. I guess I am hopelessly slow and need more training...

I didn't know it was LAME -b 128 that failed badly with the harp40_1 sample and got 1.4. I spent a lot of time trying to find the high anchor from the samples that were audibly better. Though, I was a bit surprised that the 1.4 samples were quite good after the first two totally broken notes. As I said earlier, the only other obvious difference was with the harp40_1 test1 AAC samples. Two of the AAC settings didn't have one particularly bad artifact with one of the higher notes. I thought one of those better contenders was LAME@128.

[Firon]

On most samples, I honestly couldn't tell the difference between the different AAC samples, they tended to sound about the same (not that great, but then again, it's 48kbps...), but there were a few exceptions.
It was kind of weird to hear LAME -b 128 doing worse on harp40_1, I thought I'd made a mistake and had to recheck my results

There were more than a few PS samples that sounded slightly better to me than the non-PS counterparts, interestingly enough.

[IgorC]

Nero 7 package was updated ( 6th of feb). is submited Nero codec for this test in this package?

[Ivan Dimkovic]

Nero 7 package was updated ( 6th of feb). is submited Nero codec for this test in this package?

No - the codec will be published by Gabriel.

New AAC codec will be available as a completely new release - ETA is still not public, but we committed to Gabriel's condition that codecs will be integrated "within 3 months", so the release date is somewhere in that time frame