Some background information

A couple of weeks ago we had a minor debate about the quality of the modern lossy encoders at a Finnish AV forum. My plan was to provide the forum users an opportunity to test their beliefs and hearing by providing a few test samples and instructions how to use the foo_abx tool.

I prepared five samples of different genres from my collection. I thought the selected samples would be a bit above average in complexity - nothing like killer samples, but not too easy for the encoders. I encoded the samples using three different quality levels and two encoders: Vorbis b4.5 (-q 1.5, -q 4.25 and -q 6.25) and LAME 3.97b1 (-V8, -V5 and -V2; all --vbr -new). The idea was to explain something like this:
1. the lowest quality is useful e.g. with portables, but not actually "hifi" quality
2. the middle quality is very good and finding differences is not easy
3. the highest quality is transparent or almost transparent.

My plan didn't work out. After ABX testing the lowest quality samples I realized that my samples are going to be way too easy for the encoders at the higher quality levels. For example, one of the local audio gurus who accepts only lossless files tried to ABX one of the samples. He could ABX MP3 -V8 and didn't like it, but the Vorbis -q 1.5 sample made him almost angry because he couldn't ABX it. He used his high-end speakers instead of headphones, but this tells something about the Vorbis quality anyway. There was no sense to continue the test with the higher quality samples.

I had no plans to publish any results here, but because of the ongoing debate about the 128 kbps test here is a nice example:


The test sample

hot_tequilla_brown.flac (genre: ~electronic/pop/funk (?), 21 s, 2.63 MB)

- This sample produces 91-96 kbps (my overall test target was ~88 kbps for this quality level).

The tested lossy files are available in this package: lossy_samples.zip (767 kB)


My ABX results


LAME 3.97 beta1 -V8 --vbr-new ~94 kbps

LAME was easy to ABX because of the obvious lowpass. I think -V8 is too low setting for anything that contains high frequencies. (Though, in general my high frequency hearing is not excellent. I can't ABX a lowpass over 16 kHz)

LAME at -V5 was much better with this sample. It sounded fine in casual listening, but I didn't ABX it.


Vorbis aoTuV beta 4.5 -q 1.5 ~96 kbps

Very difficult to ABX. It took me 5 minutes to find a passage where I could possibly hear a difference. Vorbis was almost transparent with this sample. In casual listening I couldn't hear any problems.


Window Media Audio

Today I tested WMA standard with the same sample. I had never properly tried WMA at VBR quality 25.

WMA 9.1 Standard VBR25 ~91 kbps

This was not easy, but I could hear the difference at a certain passage, mostly because of the slight lowpass and perhaps a bit narrower stereo width. But after the first seven tries my ears got tired and I had difficulties to ABX. I wanted to be sure and continued through 20 trials. It took over 20 minutes. In general I couldn't hear any obvious problems. (I didn't expect VBR25 to be this good. Am I becoming deaf?) Since I tested Vorbis two weeks ago I cannot directly compare these two codecs.


The test gear used: Terratec DMX 6fire 24/96 soundcard, Harman/Kardon AVI 200 MKII amp, KOSS HV/1A headphones.

Edit: typo
Edit 2: changed the lossless sample to FLAC format
Edit 3: added the lossy samples

EDIT 4:
I removed the samples to make room. PM me if you like to try them.

nice hint,
I second it, and repeat so one of my suggestions to the multiformat test:

Lower averaged target bitrate of music of various genres from 128k down to 10x or even 9x kbit/s. People testing will have hard work still, consider even the number of samples and formats.

This idea is somehow logical, as encoders made progress over the years, so the magic 128k area for "CD-quality" is lower these days..., especially for Joe Average.

And, thinking more, testing in these bitrates suitable for portables, we should collect comparable test results not only for our HA formats ogg aotuv, aac (in those 2 variants), mp3 lame, mpc (in past, though it should be tested against the modern encoders also), but also wma, and maybe wma-pro, but with lower priority, because wma-pro hasn't much hardware support yet and it is windows only anyway, or is there mac, linux support ?
Polls can be taken as support for decisions, but I don#t think, polling makes sense regarding selecting contenders for listening tests. As we don't have pressures on time, we should setup tests, so that there is logic in them. And we should more think in long time terms meanwhile, as we see the a little bit already the end of lossy format developments (regarding squeezing qualitywise even more out of given stereo bitrates). So, tests should contain something, that tests of modern encoders can be compared to older tests (of previous encoders of same format), otherwise it is even more difficult to say something to development of a format.

OK, finally! It's pretty hard to find any ABX comparator for mac... But once I got a perl-script running, I'm very confident that I can hear a difference also with the Vorbis file (8/8). It's pretty good though, and I'd say "perceptible but not annoying"...

I came here from the 128 kbit pre-test discussion, so it was of some interest to test it at -q4 also. This level is very close the original, and I'd rate it very close to "imperceptible".. Abx: 14/16

Just a note, ABX trials are only statistically valid if you either say "I will do x trials" before the ABX test and stick to it or hide the results until the end of the test.

I'm still amazed that there is such a huge difference between the average joe and people with some artifact training. My normal reaction would be: 1.5? That's horrible quality! No need to ABX something like that.

So I went ahead and tested a random sample of my own with Lancer 20051121, which includes the current aotuv tunings at -q 1.5. It sounded much better than expected (with speakers at least).

ABX with headphones was easy due to the overall "mushyness" and lack of highs: 8/8.
The sample in question was an excerpt from "Jake Walton - Seven Gurdies".

ABXing the hot_tequilla_brown sample was a little more difficult: 8/8. I focused on the lowpass mostly. But amazing quality nevertheless. I don't think I would have been able to do it with speakers. I also expected pre-echo to be much worse.

I guess this is a testament to aoyumi's fine work and a reminder why we need to backup our claims, which we otherwise would take for granted, with blind tests.

You are right. Better would have been to stop the test at 10 and continue with a new test after a rest break for seeing if the possibly changed physical condition and practice can make difference.

For my defense I'd like to add that I wasn't in the best shape for test. I wanted to include WMA in my report, but I was tired after a long workday and I found it difficult to concentrate. Too bad for science that hearing is not an absolute thing that doesn't change from time to time. In this case it changed during the test. At first I was sure about the difference and the results were similar. Then I lost my concentration and started guessing. After a break and a coffee cup (illegal doping) I could hear the difference again.

Perhaps the test was meaningful just for myself. I think I proved (to myself) that I can hear the difference and the difference is small enough to be difficult to hear if I am not in the best possible shape.

Edit: typo

Vorbis does sound really good at lower bitrates, but the HF boost / coarse sound is still there - Q4 is fixed, but at Q2 its definately there most of the time.

Thanks.

Would you mind to try the WMA sample?

I guess I should have not mentioned about the lowpass. When you start listening specifically to it you kind of lose the complete picture. A slight lowpass is often masked with other elements and difficult to realize.

Now it's my turn to be picky. Have you ABXed Vorbis aoTuV b4.5 or 4.51 at -q 2?

Exactly my thoughts too before and after.

I didn't read all your comments about the individual codecs. But you are right, if the original were allready lowpassed, things would get a lot more difficult.

Wma sounds just like I would expect such a low bitrate encoding to sound (contrary to the good performance of vorbis). Strong lowpass and what remained of the highs was metallic and squishy. Fastest ABX ever; 11 seconds to get 8/8.

Mp3 was even worse: lower lowpass than wma with some pre-echo. 8/8

On the abc/hr scale I would rank ogg around 4 (perceptible, but not annoying). Wma and mp3 would get something < 2 (Annoying), with mp3 being a little lower.

Gecko,

What equipment you used?

I am afraid that my HW chain may produce distortion in headphone listening.

I did with Aotuv 4. Haven't tried 4.5

Foobar -> Terratec Aureon Sky (with Prodigy drivers) -> Beyerdynamic DT 880

I guess I am in better shape now.

This time I tried to find out if Vorbis is different from WMA. Before starting the test I decided to do 20 tries.

Vorbis -q 1.5 vs WMA VBR25


No problems to ABX. Vorbis was clearly better. I can confirm Gecko's findings.

Also, I didn't hear any "headphone distortion" that would have masked artifacts. I think my gear is fine after all. The problem I had with ABXing WMA was caused by other factors.


Edit: codebox

I uploaded the classical sample I used in my test here :
http://www.hydrogenaudio.org/forums/index....ndpost&p=346739

Any comments about it?

Incredible!

I never thought I'd be unable to hear the difference at such a low bitrate; I'm thinking about reencoding my whole collection to ogg now -- maybe at -q 4 or so. Are there any problem samples I could check with, to see if I hear a difference at that type of bitrate?

It was easy to tell for mp3 and wma, though. 19/20 and 12/12

It's amazy sample. Even itunes v6.0.1.3 at 128 kbit/s VBR 136 kbit/s (aprox. 155 kbit/s real bitrate for this sample) wasn't transparent. Well maybe it's trasnparent. But I could hear artefacts and noise on human voices in both channels during ABX test.