MP3 encoders (HARD ROCK-ACOUSTIC) ABX listening tests (11 Jan 2012), Moderation: inconclusive ABX results (p-value > 0.05), no ABC/HR Options

[zmejce]

Hello,
Ive tested mp3 encoders today and I like to share my results.
I havent done spectrum analisys, only ABX listening tests.

Programs used for encoding:
1. dbpoweramp 14.2 (with fraunhofer 4.0.3-latest)
2. iTunes 10.5.2.11 (uses fraunhofer mp3)
3. Easy CD-DA 15 (uses some version of lame 3.98)
4. foobar 1.1.8 (with lame3.99-20111018)
5. Nero 11 (uses fraunhofer mp3 encoder)

*note: even if there are few fraunhofer encoders, they are not exactly the same, also I've tested only 2 tracks of hard/glam rock genre (band name: Extreme). Ive chose that music because it has nice rhythm, drums and guitars, has nice effects on vocals and also contains samples of acoustic guitars and harmonics-and i didnt chose those tracks because of their names neither because of the lyrics that they are containing. Also i wanted to test WMP12's mp3 encoder, but i didnt found a way to encode a flac file to mp3 with it, also Ive found out that the flac decoder filter for wmp is with lower quality than other players (ex. winamp, foobar).

The settings used:
CBR 320kbps Joint Stereo, on itunes 10 ive checked "filter frequencies below 10Hz" & "smart encoding adjustments".


Filesize results:

_________Extreme1_______Extreme2
Flac_______26.2MB________27.4MB
Lame3.99__8.68MB________9.34MB
CD-DA_____8.69MB________9.35MB
Nero11_____8.68MB________9.34MB
iTunes10___8.68MB________9.34MB
dbPower___8.67MB________9.33MB

*note: iTunes tends to add 1-2 seconds of silence at the beggining and/or end of the track
Winner: dbPoweramp with fraunhofer

Encoding speed results:
Speed results were minimal in difference, the fastest was itunes, dbpoweramp was same or almost same as itunes, while the other encoders were few seconds late.
Winners: iTunes & dbPoweramp (both use fraunhofer)


ABX Listening Results:
1. Easy CD-DA
Extreme1

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:10:28

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\cd-da\- 01 Little Girls.mp3

17:10:28 : Test started.
17:11:29 : 01/01 50.0%
17:12:12 : 02/02 25.0%
17:12:53 : 02/03 50.0%
17:13:26 : 02/04 68.8%
17:14:00 : 02/05 81.3%
17:14:05 : Test finished.

----------
Total: 2/5 (81.3%)

Extreme2

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:15:00

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\cd-da\- 03 Kid Ego.mp3

17:15:00 : Test started.
17:16:37 : 01/01 50.0%
17:16:49 : 02/02 25.0%
17:17:00 : 02/03 50.0%
17:17:15 : 03/04 31.3%
17:17:34 : 03/05 50.0%
17:18:42 : 03/06 65.6%
17:18:46 : Test finished.

----------
Total: 3/6 (65.6%)

Description:
This is my first ever ABX test I think of it as a warm-up , all i remember that the difference was noticable and probably it was similar as lame3.99.


2. lame3.99-20111018
Extreme1

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:26:48

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\lame3.99-20111018\- 01 Little Girls.mp3

17:26:48 : Test started.
17:27:16 : 01/01 50.0%
17:27:36 : 02/02 25.0%
17:27:54 : 03/03 12.5%
17:28:12 : 04/04 6.3%
17:28:31 : 04/05 18.8%
17:28:33 : Test finished.

----------
Total: 4/5 (18.8%)

Extreme2

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:29:28

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\lame3.99-20111018\- 03 Kid Ego.mp3

17:29:28 : Test started.
17:29:51 : 01/01 50.0%
17:30:09 : 02/02 25.0%
17:30:23 : 03/03 12.5%
17:30:36 : 04/04 6.3%
17:30:49 : 05/05 3.1%
17:30:50 : Test finished.

----------
Total: 5/5 (3.1%)

Description:
The Difference was very notable, especially on the first acoustic part (its in the first 15-20 secs of the song) of Extreme1 track and also on the on the vocals that had effects on the Extreme2 track. Absolutely the worst choice for compressing music to mp3.

3. Nero 11 (fraunhofer):
Extreme1

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:19:26

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\nero\- 01 Little Girls.mp3

17:19:26 : Test started.
17:20:23 : 01/01 50.0%
17:20:57 : 01/02 75.0%
17:21:31 : 01/03 87.5%
17:22:57 : 02/04 68.8%
17:23:31 : 02/05 81.3%
17:23:33 : Test finished.

----------
Total: 2/5 (81.3%)

Extreme2

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:24:15

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\nero\- 03 Kid Ego.mp3

17:24:15 : Test started.
17:24:37 : 01/01 50.0%
17:24:57 : 01/02 75.0%
17:25:18 : 02/03 50.0%
17:25:36 : 03/04 31.3%
17:25:56 : 04/05 18.8%
17:26:14 : 04/06 34.4%
17:26:17 : Test finished.

----------
Total: 4/6 (34.4%)

Description:
Nero 11's fraunhofer encoder is a better choice than the lame versions used for testing. The result was very little noticable difference between the Flac's and the mp3's encoded with Nero11 .

4. iTunes 10.5.2 (fraunhofer)
Extreme1

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:31:31

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\itunes\- 01 Little Girls.mp3

17:31:31 : Test started.
17:31:54 : 00/01 100.0%
17:32:13 : 01/02 75.0%
17:32:29 : 02/03 50.0%
17:32:47 : 02/04 68.8%
17:33:01 : 03/05 50.0%
17:33:03 : Test finished.

----------
Total: 3/5 (50.0%)

Extreme2

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:33:34

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\itunes\- 03 Kid Ego.mp3

17:33:34 : Test started.
17:34:04 : 00/01 100.0%
17:34:16 : 01/02 75.0%
17:34:36 : 02/03 50.0%
17:34:49 : 03/04 31.3%
17:35:11 : 04/05 18.8%
17:35:14 : Test finished.

----------
Total: 4/5 (18.8%)

Description:
iTunes mp3 encoder is very nice choice for encoding music. Althou its not absolutely best its sertanly better than lame. In the testings I was also able to hear minimal difference between the flac's and the mp3's encoded with iTunes.

5. dbPowerAmp (with fraunhofer)
Extreme1

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:35:45

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\dbpoweramp\- 01 Little Girls.mp3

17:35:45 : Test started.
17:36:12 : 00/01 100.0%
17:36:34 : 00/02 100.0%
17:36:56 : 00/03 100.0%
17:37:23 : 00/04 100.0%
17:37:44 : 00/05 100.0%
17:37:46 : Test finished.

----------
Total: 0/5 (100.0%)

Extreme2

CODE

foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:38:19

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\dbpoweramp\- 03 Kid Ego.mp3

17:38:19 : Test started.
17:38:36 : 00/01 100.0%
17:38:50 : 01/02 75.0%
17:39:04 : 01/03 87.5%
17:39:18 : 02/04 68.8%
17:39:35 : 03/05 50.0%
17:39:36 : Test finished.

----------
Total: 3/5 (50.0%)

Description:
dbPowerAmp's fraunhofer encoder is the absolute winner. I wasnt able to hear any difference between the flac's and the mp3's encoded with dbPowerAmp's fraunhofer encoder. Its definitely "ears bit perfect" encoder.

*note about ABX testings: on the first track (little girls) i was able to hear the difference from 4 of the encoders from the acoustic guitar in the beginning of the song-somewhere in the first 15-20 seconds. On the 2nd track (kid ego) i was able to hear the difference from 4 of the encoders at the beginning of the song when appeared vocals with effects. During the main part of the song and electric guitars, all encoders sounded identical or almost identical.

For the test ive used:
Core 2 Quad 2.5ghz, audigy platinum ex, audio system: panasonic sa-ak27 2x100W with superwoofer (its 8yrs old but it has quite good quality for home usage)
Ive used Foobar 1.1.8 with ABX component.


Conclusion:

dbPowerAmp with fraunhofer makes the mp3s smallest in size and best in quality-I couldnt hear any difference between its mp3s and the flacs-also its fastest or next to itunes, itunes is also excelent althou not "ears bit perfect" as dbPowerAmp, Nero is fine too, while lame is something that even at Highest quality settings provides lower quality than the other mp3 encoders.

Thanks for reading.

[db1989]

Well if you hear any difference in the sound once and than you repeat that 3 times and hear the same difference is pretty fine. Ive did 5-6 times.

You still have not read about guessing vs. hearing, and how a lower number of trials means increases the likelihood that mere guesswork might suggest (to those unaware of standard ABXing procedures) the existence of differences that are not really there.

greynol’s metaphor is a common and very illustrative one: I might flip a coin five times now and avoid seeing the face of the Queen four times, but that wouldn’t do much to support my (hypothetical) claim to have the power to overthrow the monarchy; I’d have to do a lot better to be taken seriously.

Until you have taken the time to understand the purpose, basis (replicability vs. randomness/‘luck’), etc.), and (actual) procedure of an ABX test, don’t make pronouncements on how it can and cannot be used.