Personal Listening Test of Opus, Celt, AAC at 75-100kbps, ABC/HR blind test, 1 Listener Options

[Kamedo2]

Abstract:
Blind Comparison between 2012/09 new Opusenc(tfsel5), old Celtenc 0.11.2, Apple AAC-LC tvbr, cvbr.
This is an English version of my original post in Japanese. http://d.hatena.ne.jp/kamedo2/20121116/1353099244#seemore

Encoders:
libopus 0.9.11-146-gdc4f83b-exp_analysis
https://people.xiph.org/~greg/opus-tools_exp_dc4f83be.zip
celt-0.11.2-win32
https://people.xiph.org/~greg/celt-0.11.2-win32.zip
qaac 1.40
qaac 1.40

Settings:
opusenc --bitrate 66 input.wav output.wav
celtenc input.48k.raw --bitrate 75 --comp 10 output.wav
qaac --cvbr 72 -o output.m4a input.wav
qaac --tvbr 27 -o output.m4a input.wav
opusenc --bitrate 90 input.wav output.wav
celtenc input.48k.raw --bitrate 100 --comp 10 output.wav
qaac --cvbr 96 -o output.m4a input.wav
qaac --tvbr 45 -o output.m4a input.wav

Samples:
20 Sounds of various genres, from easy to modestly critical.
http://zak.s206.xrea.com/bitratetest/main.htm
To download, access to the link above, 2nd paragraph, 3rd-6th links. (40_30sec - Run up)

Hardwares:
Sony PSP-3000 + RP-HT560(1st) , RP-HJE150(2nd), took the average of the two results.

Results:




Conclusions & Observations:
I could not detect a significant improvement in the new September 1st version of Opus, from the old Celtenc in 2011.
It's possibly because the new Opus inflates bitrates more than it improves qualities, although the set of sounds contain easy samples.
On 75kbps, Opus/Celt are markedly better. On 100kbps, there is no big difference between those codecs.

Raw data:
40 Logs and encoders, decoders log
http://zak.s206.xrea.com/bitratetest/log_o...kbps100kbps.zip

CODE

% Opus, AAC 75kbps, 100kbps ABC/HR Score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
opus_75k    celt_75k    cvbr_75k    tvbr_75k    opus100k    celt100k    cvbr100k    tvbr100k
%features 6 75kbps 75kbps 75kbps 75kbps 100kbps 100kbps 100kbps 100kbps
%features 7 OPUS OPUS AAC-LC AAC-LC OPUS OPUS AAC-LC AAC-LC
3.050    3.100    2.500    2.750    3.500    3.750    3.700    3.800    
3.750    2.950    2.700    2.750    4.050    3.800    4.000    3.950    
2.800    2.550    3.000    3.000    3.600    3.250    4.050    3.900    
2.700    3.150    2.350    2.300    3.350    3.800    3.600    3.700    
4.000    3.400    2.850    2.850    4.350    3.900    3.550    3.550    
2.600    2.550    2.800    2.800    3.350    3.150    3.950    3.900    
3.400    3.950    3.000    3.200    3.850    4.500    3.700    3.800    
3.450    3.500    2.900    2.800    3.850    4.050    4.050    4.150    
2.950    2.700    3.550    3.450    3.250    3.450    4.000    3.850    
3.100    3.400    2.750    2.600    3.800    3.850    4.150    4.000    
3.350    3.100    2.600    2.600    3.750    3.400    3.450    3.500    
3.750    3.350    2.800    2.950    4.050    3.750    3.800    3.850    
3.550    3.300    2.600    2.650    4.250    3.950    3.750    3.600    
3.100    3.350    2.750    2.550    3.650    3.700    3.850    3.800    
3.400    3.450    2.900    2.900    3.650    3.950    3.750    3.900    
3.250    3.300    2.750    2.800    3.650    3.850    3.950    3.750    
3.600    3.800    3.300    3.300    3.550    4.000    3.650    3.700    
3.700    3.350    3.300    3.300    3.900    3.650    4.100    4.000    
3.100    3.600    3.150    3.000    3.700    3.800    4.100    3.850    
3.650    4.050    3.000    2.900    4.050    4.250    3.750    3.550

It's not strange that some scores get 0.050 scale because I did tests twice per each music.

[Dynamic]

x-axis=actual bitrate

That was one query I did have about your method, Kamedo2 - mainly because I don't speak more than about 5 words of Japanese to find out for myself.

As you have been so thorough, I expect we can assume that you tested the encoders on a larger collection (maybe 10 or many more CDs) of normal music to determine the target settings to achieve the bitrate target, and that you plotted THAT collection-wide bitrate (for the large sample size) on the horizontal axis of your graphs, ignoring the actual bitrate used for the test samples?

The bitrate on a number of short samples (with more problem samples than usual) may properly be much higher than the average of a wide collection and is not relevant in the context of fair comparison. (That's the beauty of well tuned VBR, and to some extent modern constrained VBR as used by the 'CBR' modes - using much higher bitrate when the sound requires it and less when it doesn't without inflating the average over a whole collection).

There's also a chance that Opus produces lower bitrates for short samples as it doesn't use large codebooks compared to AAC, Vorbis etc., so using the sample-wide bitrate would give an over-emphasised disadvantage to AAC at lower bitrates on shorter-than-normal samples, again requiring the bitrate of a wider collection of representative CD releases, presumably encoded in the popular file-per-track mode without any pre-applied Replay Gain, as that's most representative of real use of lossy encoders for playback.

[Kamedo2]

x-axis=actual bitrate

That was one query I did have about your method, Kamedo2 - mainly because I don't speak more than about 5 words of Japanese to find out for myself.

As you have been so thorough, I expect we can assume that you tested the encoders on a larger collection (maybe 10 or many more CDs) of normal music to determine the target settings to achieve the bitrate target, and that you plotted THAT collection-wide bitrate (for the large sample size) on the horizontal axis of your graphs, ignoring the actual bitrate used for the test samples?

Horizontal axis is always the average actual bitrate including headers and footers of each sample.
I calculated filesize*8/(sample_num/44100) of each sample, typically 20-30sec.
Then, I took the average of the 20 bitrates in bps. Larger collections are NEVER used.
Encoded files are typically around 200KB. Opus should benefit form smaller codebooks in this case.

larger collection (maybe 10 or many more CDs) of normal music

My set of samples are fairly normal. At least they are not an array of super-critical fatboys, although short.
The last 5 samples from the 20 samples I used.(Reunion Blues ~ Run up)
http://zak.s206.xrea.com/bitratetest/bitra...st_wav25-29.zip

CODE

list of audio file size in KB(1000B)
286 377 187 220 231 151 153 278 104 232 252 204 182 246 187 251 267 255 155 170
283 278 182 231 175 165 191 264 100 237 260 192 188 265 190 282 289 256 186 189
303 280 202 233 190 163 191 266 108 233 273 199 188 263 184 313 278 256 187 189
319 216 208 224 175 163 191 253 108 209 268 205 163 240 154 341 233 242 169 158
389 509 252 298 313 204 202 375 139 311 342 276 244 334 238 341 344 344 210 229
379 373 242 309 234 220 255 353 133 316 346 257 250 354 253 376 386 343 249 252
415 370 265 310 252 222 252 355 143 312 372 264 248 354 249 411 366 341 250 245
425 287 275 310 239 222 262 347 143 282 372 282 231 340 212 455 305 330 241 222

list of audio file bitrate including headers and footers in kbps(1000bps)
  76 101  78  72  92  69  60  79  80  74  69  81  73  69  75  67  72  74  63  68
  75  74  76  76  70  76  75  75  77  76  72  76  75  75  76  75  78  75  75  76
  81  75  84  76  76  75  75  75  84  74  75  79  75  74  74  84  75  75  75  75
  85  58  87  73  70  75  75  72  84  67  74  81  65  68  62  91  63  71  68  63
104 136 105  97 125  93  79 106 107  99  94 110  98  94  96  91  93 100  84  92
101 100 101 101  94 101 100 100 103 101  95 102 100 100 102 100 104 100 100 101
111  99 111 101 101 102  99 100 111  99 102 105  99 100 100 110  99  99 101  98
113  77 115 101  96 102 103  98 111  90 102 112  92  96  86 121  83  96  97  89