IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans, ABC/HR blind test, 1 Listener
Kamedo2
post Oct 2 2013, 18:13
Post #1





Group: Members
Posts: 168
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



Abstract:
This is a translation of an old test, finished in January 2012, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between Apple AAC-LC, Vo-aacenc(VisualOn), Microsoft WMA(standard), LAME CBR, at 128kbps and 192kbps.

Encoders:
qaac 1.18
FFmpeg r.36030
WMAEncode 0.2.9
LAME 3.98.4

Settings:
qaac --cvbr 128 -o out.mp4 in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 128k out.mp4
WMAEncode64 in.wav out.wma --bitrate 128
lame -q 0 -b 128 in.wav out.mp3

qaac --cvbr 192 -o out.mp in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 192k out.mp
WMAEncode64 in.wav out.wma --bitrate 192
lame -q 0 -b 192 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1(1st), SE-DIR800CII(2nd), took average between two.

Results



Conclusions & Observations:
The Apple AAC had the best quality among the 4 encoders tested. The Apple AAC was clearly superior than the Microsoft WMA. LAME MP3 was worse than the WMA, and VisualOn vo-aacenc was the worst encoder.

Anova analysis:
CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of Degrees Sum of Mean
variation of Freedom squares Square F p

Total 119 88.91
Testers (blocks) 14 8.12
Codecs eval'd 7 65.64 9.38 60.70 0.00E+000
Error 98 15.14 0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.285

Means:

qaac192 wmaen192 qaac128 lame192 wmaen128 lame128 voaac192 voaac128
4.78 4.47 4.05 4.00 3.73 2.85 2.83 2.74

---------------------------- p-value Matrix ---------------------------

wmaen192 qaac128 lame192 wmaen128 lame128 voaac192 voaac128
qaac192 0.035* 0.000* 0.000* 0.000* 0.000* 0.000* 0.000*
wmaen192 0.004* 0.001* 0.000* 0.000* 0.000* 0.000*
qaac128 0.694 0.025* 0.000* 0.000* 0.000*
lame192 0.063 0.000* 0.000* 0.000*
wmaen128 0.000* 0.000* 0.000*
lame128 0.889 0.445
voaac192 0.532
-----------------------------------------------------------------------

qaac192 is better than wmaen192, qaac128, lame192, wmaen128, lame128, voaac192, voaac128
wmaen192 is better than qaac128, lame192, wmaen128, lame128, voaac192, voaac128
qaac128 is better than wmaen128, lame128, voaac192, voaac128
lame192 is better than lame128, voaac192, voaac128
wmaen128 is better than lame128, voaac192, voaac128


Raw data:
CODE
% AAC/WMA/MP3 128/192kbps ABC/HR score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% http://zak.s206.xrea.com/bitratetest/graphmaker3.htm
qaac128 voaac128 wmaen128 lame128 qaac192 voaac192 wmaen192 lame192
%qaac vo-aacenc wmaencode lame qaac vo-aacenc wmaencode lame
%features 6 AAC AAC WMA MP3 AAC AAC WMA MP3
%features 7 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
3.850 2.600 3.300 2.100 5.000 2.750 3.650 3.300
4.350 3.400 4.000 2.900 4.600 3.850 5.000 3.700
4.550 2.700 4.600 2.350 5.000 2.900 5.000 4.500
3.800 2.800 3.100 2.300 4.650 3.150 4.200 3.950
3.400 2.450 3.350 2.450 5.000 2.400 4.050 3.350
4.550 2.750 5.000 3.350 5.000 2.800 4.650 4.650
5.000 2.350 3.900 2.750 5.000 2.350 5.000 3.900
4.100 2.300 3.600 3.100 5.000 2.300 4.250 4.450
3.800 2.450 2.650 2.950 4.450 2.550 3.500 3.200
3.950 2.300 4.350 2.850 5.000 2.300 4.700 5.000
3.650 2.850 3.600 3.300 4.250 3.200 4.600 4.100
3.750 2.600 3.700 2.800 4.150 2.650 4.600 3.950
3.800 3.800 3.700 3.450 5.000 2.850 5.000 3.950
3.700 3.050 3.500 2.800 4.600 3.450 4.450 3.850
4.550 2.700 3.550 3.300 5.000 2.950 4.450 4.100
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Vocal

It's not strange that some scores get 0.05 scale, as I tested twice per each sample.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 24th April 2014 - 09:45