MP3 Listening Test at 128 kbps, Call for encoders and settings Options

[Sebastian Mares]

But how come I had different results while testing on the same machine - an FSC Amilo Xi 2550 notebook with a dual core CPU with XP and Vista? I am really confused.

Edit: Found the problem. XP was really set to /numproc=1 /noguiboot.

This post has been edited by Sebastian Mares: Oct 7 2008, 18:51

[Sebastian Mares]

Tested the new 8.0.1 version and I get the same results on my Vista machine.

[Alex B]

Oh, is there a new version of iTunes?

Have you already tried the effect of the quality settings on XP and Vista? It would be useful to know if those settings are ineffective when the encoder appears to be creating only ABR like MP3 files.

This post has been edited by Alex B: Oct 3 2008, 10:55

[Sebastian Mares]

I only have a Vista machine at home and today is Unification Day in Germany so I am not at work. I can try on a VM with Vista and XP, though (which I am going to do now).

Edit: And yeah, I turned on my PC today and the Apple Updater popped up.

This post has been edited by Sebastian Mares: Oct 3 2008, 11:10

[Alex B]

I only have a Vista machine at home and today is Unification Day in Germany so I am not at work. I can try on a VM with Vista and XP, though (which I am going to do now).

I'd guess the settings have a clear effect whenever the files appear to be real VBR, like in my case.
I can't test the other case because none of the PCs I have access at the moment produce those ABR like results.

[Sebastian Mares]

BTW, do you have a multicore machine or a singlecore one?

[Alex B]

Single.

My "HTPC" would have HyperThreading, but it is having strange problems and I have disassembled it for testing the individual components. I may need to replace the motherboard. It is the machine that produced the lower iTunes VBR bitrates a year ago (on XP).

[robert]

It seems iTunes encoder has some problem on multi core machines. How does it fit with nao's result on a MAC, is the G5 a multi core machine too?

I don't have iTunes, does it feature some switch/check-box to en-/dis-able multi core usage?

[Sebastian Mares]

So, I just did some tests with fatboy on my dualcore machine running Vista. The lowest VBR quality setting produces 118 kbps, the mid one 120 kbps and the highest one 122 kbps. The two options Smart Encoding and Filter 10 Hz also have an impact, but a very small one (bitrate difference was only 1 kbps).

I am afraid iTunes does not have an option to enable or disable multicore usage.

While at it - Alex B, could you please encode fatboy to 128 kbps VBR AAC using iTunes? Wondering if the AAC encoder has the same problem. Please tell me what bitrate fb2k reports.

This post has been edited by Sebastian Mares: Oct 7 2008, 18:52

[nao]

How does it fit with nao's result on a MAC, is the G5 a multi core machine too?

My G5 machine has a dual-core CPU. Only G4 machine has a single-core CPU in my test. So, probably the insight that iTunes mp3 encoder has a problem on multi-CPU environment is correct. Unfortunately, there is no option in iTunes to turn-off multithreading.

[Sebastian Mares]

OK, that convinced me. I filed a bug report to Apple. Hopefully they will reply.

BTW, regarding the AAC. fb2k reports 131 kbps and iTunes 128 kbps.

This post has been edited by Sebastian Mares: Oct 3 2008, 13:09

[Alex B]

I will have to check why I got different results at work.

So, I just did some tests with fatboy on my dualcore machine running Vista. The lowest VBR quality setting produces 118 kbps, the mid one 120 kbps and the highest one 122 kbps. The two options Smart Encoding and Filter 10 Hz also have an impact, but a very small one (bitrate difference was only 1 kbps).

So the quality settings have some very slight effect. Apparently the selected setting is memorized and used, but it does not work properly (I suppose we can assume that the setting is designed to be able to adjust the amount of bitrate fluctuation.)

While at it - Alex B, could you please encode fatboy to 128 kbps VBR AAC using iTunes? Wondering if the AAC encoder has the same problem. Please tell me what bitrate fb2k reports.

From foobar's file properties window:

File Name : fatboy_30sec.m4a
File Size : 478KB (489 618 bytes)
Duration : 0:29.207 (1288011 samples)
Sample Rate : 44100 Hz
Channels : 2
Bitrate : 131 kbps
Codec : AAC
Codec Profile : AAC LC
Encoding : lossy
Tool : iTunes 8.0.0.35, QuickTime 7.5.5

[nao]

iTunes AAC encoder does not support multi-threaded encoding, so probably it is not affected by the problem.

[Alex B]

Sebastian, please upload the 122 kbps "VBR 112" sample. I'd like to compare the files in a listening test.

This post has been edited by Alex B: Oct 3 2008, 14:14

[Alex B]

Actually a "dual core" VBR 128 encoded sample would be better. It would have been the selected setting if the bitrates were tested only on machines that trigger this problem.

This post has been edited by Alex B: Oct 3 2008, 14:14

[Sebastian Mares]

Here it is: http://www.hydrogenaudio.org/forums/index....showtopic=66253

This post has been edited by Sebastian Mares: Oct 3 2008, 14:24

[lvqcl]

Did you notice the following text just below encoder settings:

Details

56 kbps (mono)/112 kbps (stereo), VBR (Highest quality), joint stereo, optimized for MMX/SSE, using MP.

for dualcore and the same text without "using MP" - for singlecore?

I encoded one album (~1 hour long) with MP3 CBR 160 kbps:
1 core: 3m 20s (= 200s)
2 cores: 1m 57s (= 117s)

The files have different comment tag, but identical audio content. With VBR, this isn't so:

VBR 112 kbps:
1 core: 136 kbps 3m 37s (= 217s)
2 cores: 118 kbps 2m 12s (= 132s)

So multiprocessing support is really good but gives incorrect result with VBR enabled.

[Alex B]

A listening test didn't bring up any further surprises.

ABC/HR for Java, Version 0.5b, 03 October 2008
Testname: fatboy - itunes

Tester: Alex B

1R = C:\bitratetest\dualcore_vbr128.wav
2L = C:\bitratetest\singlecore_vbr112.wav

---------------------------------------
General Comments:
---------------------------------------
1R File: C:\bitratetest\dualcore_vbr128.wav
1R Rating: 1.4
1R Comment:
---------------------------------------
2L File: C:\bitratetest\singlecore_vbr112.wav
2L Rating: 2.0
2L Comment:
---------------------------------------

ABX Results:
C:\bitratetest\dualcore_vbr128.wav vs C:\bitratetest\singlecore_vbr112.wav
8 out of 8, pval = 0.0030


---- Detailed ABX results ----
C:\bitratetest\dualcore_vbr128.wav vs C:\bitratetest\singlecore_vbr112.wav
Playback Range: 03.623 to 05.910
6:17:58 PM p 1/1 pval = 0.5
6:18:15 PM p 2/2 pval = 0.25
6:18:37 PM p 3/3 pval = 0.125
6:18:59 PM p 4/4 pval = 0.062
6:19:37 PM p 5/5 pval = 0.031
6:19:55 PM p 6/6 pval = 0.015
6:20:44 PM p 7/7 pval = 0.0070
6:21:01 PM p 8/8 pval = 0.0030

As expected, both samples were seriously distorted. I did the ABX test only for checking that I can reliably distinguish the MP3 samples from each other.

Both were bad, but the single core VBR112 version was clearly better.

This post has been edited by Alex B: Oct 3 2008, 16:46

[singaiya]

Does anybody else think we should consider dropping iTunes as a contender as a result of all this? In my opinion, we shouldn't have buggy encoders in the test.

[lvqcl]

Does anybody else think we should consider dropping iTunes as a contender as a result of all this? In my opinion, we shouldn't have buggy encoders in the test.

But Apple can fix this issue easily just by disabling multiprocessor support when VBR mode is on .

[Alex B]

Does anybody else think we should consider dropping iTunes as a contender as a result of all this? In my opinion, we shouldn't have buggy encoders in the test.

That's exactly the reason I wanted to compare the two files. I wanted to know if the "single core" version produces expected (i.e. better) quality when the encoded file has a higher bitrate and thus the single core version could be considered as normal.

I doubt Apple will fix anything quickly so we have only two practical options: either we can drop iTunes completely or we can test samples that are created with a single core machine. We can test them now and Sebastian can decide later what to do. In the past Nero's AAC encoder has been disqualified from the final results because it wasn't working correctly. However, its results were published separately because that was considered to be fair for the high quality Nero encoder and because the fixes Nero did after the test wouldn't have changed the test results considerably.

EDIT

It might be good to test a few other samples before making the decision. The fatboy sample is a real killer so maybe a couple of easier samples should be tested.

This post has been edited by Alex B: Oct 4 2008, 10:54

[kornchild2002]

OK, that convinced me. I filed a bug report to Apple. Hopefully they will reply.

BTW, regarding the AAC. fb2k reports 131 kbps and iTunes 128 kbps.

Yes, this is common practice for iTunes and has been for sometime now with both their mp3 and AAC encoders. It wasn't until recently where Apple changed what iTunes would display for iTunes encoded VBR AAC files. It would display the overall average bitrate and that only confused people. They would encode a 128kbps VBR AAC files and iTunes would show a bitrate of 130kbps+. Then Apple changed things again so that iTunes would display the setting used. For a 128kbps VBR AAC file, it would display 128kbps (VBR) for the bitrate thus confusing less people (I guess). It is a shame as I would rather have iTunes display the overall average bitrate of songs (it does this for Lame encoded mp3 files) and then put (VBR) in parenthesis (again, it does this for Lame and FhG encoded mp3 files).

It is true that Apple can fix display/encoding issues with iTunes rather quickly but they don't. Apple just released a new version of iTunes and it doesn't look like they are going to update it anytime soon (at least the mp3 encoding part). The iTunes mp3 encoder receives updates even less frequently than either iTunes or its AAC encoder. So don't hold your breath for an mp3 encoder update even if it is a reported bug. Everyone on hydrogenaudio could report this bug and it still wouldn't speed Apple up. Alex B is right though, some more tests should be conducted before ruling out iTunes.

[singaiya]

I also agree that other samples should be tested for differences. It could provide some sense of the bug's scope. But I'm still sceptical about including iTunes, even after just this one result. IIRC, the bug in the old NeroAAC contender was only discovered after the test had begun. So we had the results of that contender already. Since we know of this bug before the test, it doesn't make sense to me to ask people to listen and rate it.

[lvqcl]

I installed several iTunes versions and tested them with fatboy (5-sec long) sample. Encoding settings are the same: 112kbps (VBR, max quality).

CODE

                   1 core    2 cores
iTunes 4.9.0.17       135        137
iTunes 5.0.1.4        194        119
iTunes 6.0.4.2        194        119
iTunes 7.0.2.16       207        125
iTunes 7.7.0.43       207        126

iTunes 5.0.1.4 was released at september'2005. No-one noticed this bug before?

[kornchild2002]

I guess that is because a small community uses the iTunes mp3 encoder, all others look towards Lame and the various programs using the FhG encoder (Windows Media Player, the Zune PC software, etc.).