Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Vorbis quality – wrong direction? (Read 53277 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Vorbis quality – wrong direction?

I promised to Quantum Knot to test his new tuning. I finally found the time, and tested QK2, 1.01 - compared to RC3 library. Yes, the two-years old RC3.


I. CHALLENGERS

The challengers are of course questionable. I had to explain my choices.
First, I didn't include Nyaochi and Aoyumi tunings. I really apologize to both persons. It might appear as disrespectful, but in my opinion, comparing different tunings in alpha stage each others is not something fair. I guess that alpha tunings are released for report, not for competition. Comparison will probably follow later, when all tunings will be advanced enough.

Second thing: why old vorbis encoders?
I've tested vorbis 1.00 and 1.01 many times with natural instruments, and it always sounded approximative and disapointing, generally worse than lame mp3. To my ears, there are serious problems: not only hiss or high frequencies boost (reported many times by other users), but imprecise or coarse sound, aggressive rendering. With classical music, or natural instruments well recorded, all these problems are clearly audible – probably more than with loud and (over-)compressed music.

These problems are like a vorbis “signature” (in blind test, it’s easy to find vorbis among different challengers). I never heard something comparable with other formats. But I’m sure that these flaws are not consubstantial to vorbis format, but that they are a consequence not only of a lack of tuning, but from contestable choices made in the past. I'm still convinced that something wrong happened between RC3 and final "1.00" version from July 2002, though I never really tested this difference. My suspicions are based on the following things.


Important changes were made between RC3 and Final library. This was particularly noticeable on encoding speed (final is much faster), and on sound quality at low bitrate, even for untrained people. Before “final” encoder, Vorbis wasn’t very competitive at low bitrate (especially the popular and symbolic 64 kbps). The reference two years ago was mp3pro, and vorbis was clearly behind. At low bitrate, Vorbis sounded like other non-SBR format (wma, real, aac): metallic, heavily distorted, etc… But final encoder sounded totally different: not metallic anymore, less distorted, but simply noisier (and some stereo issues). For people used to find traditional flaws at this bitrate, vorbis was simply amazing. With some habits, noise and stereo problems would be more noticeable, but even here, vorbis is a very good solution at 64 kbps (see Roberto’s recent test) compared to all present non-SBR format. Noise and stereo reduction are generally more acceptable and less ugly than metallic coloration a la wma or DivX audio...
But this incontestable victory (for an open-source and non-patented project) had some bad reverse. In my opinion, pre-final encoders were more transparent at mid-bitrate[/b] (especially 128 kbps or –q4 setting), at least on non-killer samples (90% or 99.9% of [my?] music, I can’t say) and at least with natural instruments recorded on hi-fi principles. At this bitrate, encodings are now sharing the same flaw as low bitrate: hiss, imprecise rendering… On cool samples, where even mp3 sounds flawlessly, vorbis has audible problems. I suspect that the secret of vorbis good quality at low bitrate is the cause of wrong things that are happening above.



I never seriously tried to confirm or infirm my suspicions with blind test. I was more interested by high bitrate & transparent encodings, and because of pre-echo serious problems, vorbis wasn’t really interesting (except with Garf tuning). Now than Vorbis begins to infiltrate industrial manufacturers, the audio format looks for me more interesting at lower bitrate (I expect from modern format a good quality at 130-150 kbps]. Testing Quantum Knot tuning encoder appears like a good opportunity for me to oppose post-1.00 encoders to old RC3 library, in order to see if my suspicions are justified or simple rubbish.


I looked on old CD-R, and founded three RC3 builds, dated on: February 2002, March 2002 and April 2002. I was tempted to test two different builds, and not only one RC3 encoder. Why? In my souvenirs, a vorbis developer talked about changes happened within RC3 (can't remember or understand what - maybe something related to stereo model). I think it happened in springs (April or May). February and March encoders produce same output result (I did a bit to bit comparison, and files were the same, except tiny difference on the first samples). April encoder is a different beast. Encoding speed had improved (twice faster, if not more!), and output is different too. If problems occur with final library, the cause is maybe in this crucial moment of vorbis history. I'm maybe completely wrong in my suspicions, and maybe RC3 sounds worse than 1.00 encoder. The best thing to be sure about it is to test the different encoders.


The following test confronts four vorbis encoders:
• 2002.03.07
• 2002.04.06
• Official 1.01 oggenc – JohnV compile (2003.09.09)
• GordianKnot tuning QK2 (January 2004, based on 1.01)

Setting for all encoder is the same: -q4 (VBR, 128 kbps nominal)

II. SAMPLES

In order to make the 1.01 & QK2 comparison (primary goal of the test) useful, I’ve selected sample with transients. I didn’t include well-known killer samples: positive reports for QK2 were done with this kind of samples, and for 128 kbps encodings, I guess that common musical samples are difficult enough for all encoders and more representative of real usage too. All samples are coming from my own library. I’ve deliberately chosen very short ones, so upload will be easier.

Short description:
harpsichord.wav: solo harpsichord, maybe too reverberated but very sharp and nice recording. Encoders usually suffer from pre-echo and from heavy distortions
erhu10.wav: erhu (Chinese string instrument) with percussive instruments in accompaniment.
Arche I.wav: orchestral extract from Penderecki first Symphony. Very sharp and loud attack, followed by something like a rattle [not Simon]
Mandolins.wav: extract from a famous Vivaldi concerto. This sample (reduced to 5 seconds) is one of my favorite, because many encoders failed to encode this properly at ~130 kbps.
Transfiguration.wav: part of an orchestral work of Olivier Messiaen. Two different problems should occur: distortions with cymbals (and with Vorbis 1.00 family: flattened noisy sound) and pre-echo/blurred brass instruments.
La Spagna.wav: percussions & wind instruments from Renaissance playing concertedly. Pre-echo reduction is expected from GK2, and maybe noise problems may occurs with wind instruments.
Mars.wav: beginning of the first Planet[/U], from Gustav Holst. Naturally quiet (violin pizzicatos and threatening winds, this sample is very quiet due to weird mastering of the CD layer [rip from a SACD]. This sample is comparable to orchestral lace, and I listened it without harm at very high volume in order to magnify all possible problems (pre-echo, ATH issues, background changes...)
Brahms6.wav: piano is a percussive instruments, potentially affected by pre-echo. This recording from the 6th Hungarian Dance of Johannes Brahms is really sharp and well-recorded.


In other words, the samples tested are probably not favorable to pre-Final libraries. For measuring vorbis noise problems at ~128 kbps, tonal sample are more interesting. But even here, interesting things happened during the test.



III. RESULTS




QK2 modifications have a positive effect on 5 samples. The benefits of this encoder concern pre-echo and sharpness only. On low-volume transients, GK2 is inoperative (Mars.wav). On sharp and detailed micro-attacks (Arche I.wav second part, and Transfiguration.wav), QK2 sounds identical to official 1.01. But when attacks are clearly defined (Arche I.wav first part and Harpsichord.wav), the difference is appreciable. Benefits are audible too when transients are not excessively strong (piano, percussions on La Spagna.wav), but difference is then mildly audible. I’m disappointed by the few differences with Mandolins.wav, and surprised by the identical scores I obtained with Erhu10.wav.
Vorbis library needs tweaking for mid bitrate encodings, and QK2 is a good answer for the pre-echo problem.

• The official post final “1.01” encoder sounded worse than RC3 libraries to my ears on 7 samples. Harpsichord is the only exception, with less distortion (vibrating effect). But most often, the noise issues (louder, unstable, affecting micro-dynamic of some instruments and definition/contours of others) was more discriminating than pre-echo variations. With Erhu10.wav, Transfiguration.wav and Arche I.wav (second part), difference between RC3 family and 1.01 is really big (whereas progress made with QK2 from 1.01 had a much more limited impact).
Interesting thing to note: official 1.01 never reached the 3.0 (= “slightly annoying”) notation. Notation is of course a very imprecise thing, but this ranking isn’t totally meaningless. 1.01 produces non transparent and barely acceptable sound with natural instruments, whereas the RC3 branch, as all different audio format, is able to reach near-transparency with non-difficult samples. To the 8 samples of this test I could add the results of the ~30 other samples I’ve tested in the past with vorbis: final vorbis results were each time around 3/5. In other words, the changes introduced with 1.00 are negative at –q 4 with a consequent part of the editorial production (at least for my ears).

• Other thing to note: there wasn’t any rupture between March 2002 [m2k2] and April 2002 [a2k2]. Library from m2k2 is not the crystal clear one, and a2k2 the Hoover© sounding library. M2k2 RC3 was better than a2k2 four time; a2k2 was better (but by a slightly margin) the four other time. Nevertheless, if April 2002 wasn’t an historic moment for Vorbis (except for encoding speed), it was surely the beginning of tendency: noise increased with a2k2 and reached an alarming level. The accident occurred few months later…
Other point: pre-echo issues were reduced with a2k2 library (compared to the previous one). But comparing RC3s to Finals libraries on pre-echo is not easy. Pre-echo is sometimes lower with RC3 (a2k2 or m2k2) : erhu10, Arche I, Mandolins and Brahms - and sometimes higher than with 1.01 output : Spagna, Holst.


IV. CONCLUSIONS


Hard to conclude anything with only eight samples. I nevertheless reached the following one: my suspicions weren’t unfounded, and there’s obviously something rotten in 1.00 – and following libraries. I haven’t test Vorbis with non-classical/over-compressed/electronic music, and therefore I can’t evaluate the possible benefits of 1.00 modifications. But with classical (or more generally, natural instruments recorded with fidelity principles), these changes have clearly a negative effect – with some local exceptions (like harpsichord).

I guess that without these changes, Vorbis will sound poorly at low bitrate (a quick try with RC3 at 64 kbps convinced me that 1.00 is much better). But isn’t RC3 a better basis for mid-bitrate tuning than 1.00? Or is it possible to disconnect Vorbis mid settings from low settings, in order to avoid all characteristic introduced by “final” version to contaminate all encoding area (at least, up to 5.99]?


Note: the eight samples (FLAC – 4.5 MB] are available on the uploading forum – or will follow very soon.

Vorbis quality – wrong direction?

Reply #1
I think you mean QuantumKnot. This is Gordian Knot.

Vorbis quality – wrong direction?

Reply #2
Files are uploaded.
I've just realized that I forgot to post the exact results.


EDIT > replaced Gordian Knot by Quantum Knot    Thanks
EDIT2 > replaced GK2 by QK2 

Vorbis quality – wrong direction?

Reply #3
Very interesting test, as usual.

Thank you, guruboolez.
Over thinking, over analyzing separates the body from the mind.

Vorbis quality – wrong direction?

Reply #4
I'm wondering: now that we have several different encoders (RC3, 1.0.1, GT3b1, QK2, aoTuV), it may be worthwhile to perform a full listening test comparing these encoders.

The MP3 encoder listening test has recently been completed, the AAC encoder test will soon begin, so it may make sense to do a Vorbis encoder test as well.

It probably won't be a good idea right now, since some of the tunings are still a work in progress, but I think it could be something to consider later this year.

I wonder if Roberto would be interested in setting up this listening test...

Edit: Technically, they're not really different "encoders" of course, just differently tuned revisions of the Vorbis encoder. So the terminology I used is not 100% correct.
Over thinking, over analyzing separates the body from the mind.

Vorbis quality – wrong direction?

Reply #5
what bitrate was the test run at? and i am also running a test and have noticed similar problems with vorbis. i would like to try an RC3 test and if it yields better results, it would be nice to have someone investigate the changes (Quantum, if you could).

im sure it would be a nice addition to the QK-tune.

good job with the test.

Vorbis quality – wrong direction?

Reply #6
I maybe forgot to precise: all tests were performed with -q4 setting (128 kbps nominal).

Vorbis quality – wrong direction?

Reply #7
Many thanks to guruboolez for being able to perform this detailed listening test.  It is essentially the feedback I was hoping for and it has brought about some fairly interesting points, esp. about RC3, which I haven't played with before.

I'll download the RC3 sources and have a look at the changes that were made.

Vorbis quality – wrong direction?

Reply #8


uncoupled -q4 was tested here. Bitrate is higher, but problem is considerably lowered (few hiss).
I did this test yesterday. Very quickly: I was very tired. I didn't test lower setting, except with few sample.
Apparently, coupling is THE problem.

Vorbis quality – wrong direction?

Reply #9
guruboolez, I'm somewhat confused as to what your graphs are explaining.  Considering I'm somewhat of a n00b, could you or someone else explain the graphic results for me?

Vorbis quality – wrong direction?

Reply #10
Quote
uncoupled -q4 was tested here. Bitrate is higher, but problem is considerably lowered (few hiss).
I did this test yesterday. Very quickly: I was very tired. I didn't test lower setting, except with few sample.
Apparently, coupling is THE problem.

Was this the uncoupled one I posted or the one nyaochi posted?  His one uses uncoupled stereo while mine used lossless stereo coupling (though at the time, I was under the impression it was uncoupled).

Looks like this hiss problem is affected by various factors, and stereo coupling is one of them.

On the HF boost front, Segher on vorbis-dev believes he knows a way of fixing this problem, which he has done in Lame.  Let us wait for the good news.

EDIT:  Gosh, I had a listen of "Arche I" in one of guruboolez' samples and Vorbis has a tonality problem on the first attack. 

Vorbis quality – wrong direction?

Reply #11
This morning i’ve quickly done the same test and can confirm what guruboolez has noticed.
It seems that channel coupling is the problem, moreover with totally uncoupled stereo mode the results are even more better. Don’t give too much attention to average, the scores are really significant only if compared each other.
Here is the table of results:
http://www.hydrogenaudio.org/forums/index....ic=18360&st=0&#

[span style='font-size:8pt;line-height:100%']edit: i've problems uploading with with my ftp.[/span]
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1

Vorbis quality – wrong direction?

Reply #12
Quote
guruboolez, I'm somewhat confused as to what your graphs are explaining.  Considering I'm somewhat of a n00b, could you or someone else explain the graphic results for me?

The numbers in the table show the score. Green cells indicate that the specified encoder won, red cells indicate the opposite.

The names you see on the left are samples (pieces of music), the names at the top are the encoders.

Vorbis quality – wrong direction?

Reply #13
Some clarifications. I've used QK vorbis_uc for my tests. I didn't test Nyaochi uncoupled recently. Is it the encoder known as "ModestTuning"? The same encoder released in december? [proxima] notations are really interesting - and for me, surprising. I've tested "Modest Tuning" when it was released, and though I heard real progress in noise reduction, I wasn't totally impressed. Was -q4 used for comparison? I've tested it at -q2 or -q3 (can't remember), following Nyaochi recommandations for a fair comparison.

Anyway, I'd like to thank you, [proxima], for your tests (and similar results)

Vorbis quality – wrong direction?

Reply #14
Wow!  I also find [proxima]'s listening result very interesting. If QuantumKnot and I change stereo mode correctly, the result shows lossless stereo is worse than uncoupled (dual) stereo. Where does the difference come from? Parameter difference in setup_44.h and setup_44u.h?

My uncooupled (dual stereo) oggenc is available in this thread. It is not my "Modest Tuning". 

Vorbis quality – wrong direction?

Reply #15
nyaochi> thanks for the clarification. I missed this encoder you provided some days ago. I'll test it as soon as I can.

Question: what about bitrate? Is it possible to lower the bitrate in order to have something similar between uncoupled and coupled stereo? Or does coupling save a lot of bits?

Vorbis quality – wrong direction?

Reply #16
Quote
Is it possible to lower the bitrate in order to have something similar between uncoupled and coupled stereo?

Unfortunately or fortunately, no. We must lower the quality of uncoupled one in order to get similar bitrates. Channel coupling actually saves alot of bits (with some expense of quality according to the listening results?)

Uncoupled stereo mode requires [number of channels] x [monoral bitrate]. Quality 4 requres around 80[kbps] for monoral samples, so dual-stereo quality 4 does around 160[kbps].

Vorbis quality – wrong direction?

Reply #17
Quote
Wow!  I also find [proxima]'s listening result very interesting. If QuantumKnot and I change stereo mode correctly, the result shows lossless stereo is worse than uncoupled (dual) stereo.

I 'm very surprised too !
Before this test, i was quite sceptical about the possibility of using an uncoupled stereo mode for a mid/low bitrate (128 kbps). Vorbis (in the current state) have to be an exception.

I want to advert you that i manly focused my attention with HF boost/hiss and not with other possible artifacts, moreover i'm not very familiar with classical samples so take my findings with care. Maybe Guruboolez, that certainly is more familiar with his samples, can test your encoder and eventually confirm the hiss reduction.
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1

Vorbis quality – wrong direction?

Reply #18
On a slightly off-topic note, I'm delighted to see that there has been so much Ogg Vorbis-related activity lately.

It feels to me like there the tuning of the format is finally starting to take off...

Over thinking, over analyzing separates the body from the mind.

Vorbis quality – wrong direction?

Reply #19
Here are some relevant links from Xiph.org's mailing list archive.

http://www.xiph.org/archives/vorbis-dev/200402/0009.html
Quote
You could test this effect on monoaural files.
The "HF boost" will still be present.
It's because of heavy quantization. The SNR is *so*
low at high frequencies in low bitrate modes, that the
quantization error increases energy noticably.


Test it yourself:
- generate some random numbers p_i (around -1..1)
- round them to the nearest int q_i:=round(p_i);
- compare the "energy":
  \sum{i} p_i^2
  \sum{i} q_i^2


The bad news:
I don't know how to avoid this in general.


Ghis!
Sebastian


http://www.xiph.org/archives/vorbis-dev/200402/0011.html
Quote
> You could test this effect on monoaural files.
> The "HF boost" will still be present.
> It's because of heavy quantization. The SNR is *so*
> low at high frequencies in low bitrate modes, that the
> quantization error increases energy noticably.


Ooh, if *this* causes that "HF boost" problem, I know how
to fix it (I fixed it for Lame a few years ago ;-) )


> The bad news:
> I don't know how to avoid this in general.


But I do :-)


Stay tuned...



Segher

Vorbis quality – wrong direction?

Reply #20
Quote
On a slightly off-topic note, I'm delighted to see that there has been so much Ogg Vorbis-related activity lately.

I´m delighted as well and would like to say "thank you" to all those working on vorbis.

Vorbis quality – wrong direction?

Reply #21
me as well, its good to know that vorbis may not be dead from this point forward!!

anyway. encoding a mono file still has HF boost i suppose. so it must be a combination of things. if the HF boost in vorbis was removed, it would do much better in listening tests i think. maybe if the problem can be pinned down by the multiformat test vorbis will score better than last time.

i am willing to do just about any amount of listening to fix this problem as i would love for vorbis to be the real competitor for AAC.

Vorbis quality – wrong direction?

Reply #22
Quote
Wow!  I also find [proxima]'s listening result very interesting. If QuantumKnot and I change stereo mode correctly, the result shows lossless stereo is worse than uncoupled (dual) stereo. Where does the difference come from? Parameter difference in setup_44.h and setup_44u.h?

My uncooupled (dual stereo) oggenc is available in this thread. It is not my "Modest Tuning". 

I have a feeling my lossless coupled Vorbis wasn't implemented correctly or optimally so its too early to make assumptions on it

Essentially what I did was replicate the adj_stereo of q 10 down to 5, 4, 3 and 2.  I'm not sure whether there are other parts that I need to change. 

So I wouldn't put that much attention in my rather quick hack

Vorbis quality – wrong direction?

Reply #23
Quote
On a slightly off-topic note, I'm delighted to see that there has been so much Ogg Vorbis-related activity lately.

It feels to me like there the tuning of the format is finally starting to take off...


As long as there are plenty of people willing to stick around, test our tunings, and give us feedback, then that's enough drive for us to continue tuning Vorbis.

Vorbis quality – wrong direction?

Reply #24
I encoded guruboolez' violin sample with the standard 1.0.1 coder and nyaochi's uncoupled stereo coder.  Then in CoolEdit, I did a mix paste and saved the differences in order to (informally) highlight the differences.

http://www.hydrogenaudio.org/forums/index....ndpost&p=183492

There seems to be a high pitched noise in the middle of the sample.  Is this the dreaded HF boost/hiss of Vorbis?


EDIT:  I've also done the same process with guruboolez' harpsichord sample (uploaded to the same thread). Looking at the spectrum (I know, its naughty  ), the HF difference is mostly above 8 kHz.  The spectrum of the violin sample also has most of this high frequency difference above 8 kHz.

Hopefully something useful can be gathered from this.


EDIT 2:  Did the same thing with mandolins sample where 1.0.1 got a low score and uncoupled got the highest in [proxima]'s test.  Same kind of noise.  Spectrum also mostly above 8 kHz.