After much discussion and misunderstanding (mostly on my part), I think it is about time we start testing the Vorbis encoders and eventually choose the best one to participate in Roberto's 128 kbps multiformat test, which will start on April 14. We should come to a decision by April 10 which gives us about 7 days, starting from now.

Due to a lack of time, we can't make this listening test as formal as the other ones, so it will be a self-prepared test, where each listener will prepare the coded test set and ABC/HR tests themselves. Ultimately that means it is possible to fudge the results but I have full confidence that it's not going to happen since there is barely any zealotry within the Vorbis community itself.

The Encoders

The Vorbis encoders and respective quality values, to be tested are:

1. Vorbis 1.0.1 CVS at q 4.25: http://www.rarewares.org/files/ogg/oggenc2.3CVSMinGW.zip
2. aoTuV 20040402 at q 4: http://www.geocities.jp/aoyoume/aotuv/test.html
3. Modest Tuning beta 3 at q 4: http://homepage3.nifty.com/nyaochi/soft/dist/oggencmtb3.zip
4. QKTune beta 3.2 at q 4.25: http://www.rarewares.org/quantumknot/oggencqk32.exe

Optional: GT3b2 with HF reduction at q 4.25: http://www.rarewares.org/quantumknot/oggenchfr.exe (late addition to the list)

The above quality values have been determined by guruboolez to give approximately 128 kbps on normal music. Given the time constraints (and given how we only got these to go by for now), they will do for now.

The Test Samples

The test samples that will be used are from ff123's 64 kbps listening test. There are 12 samples which can be downloaded from http://ff123.net/samples.html (right at the bottom of the page).

Test Results

As for the results, tabular format, such as this one would be nice and you can upload or post it in this thread. Otherwise, you can e-mail your ABC/HR results to: s dot so at griffith dot edu dot au

If you e-mail your result, please write a small post here to let me know so that I can confirm the submission was successful.

The test will end on April 10. Hopefully I can inform Roberto of the result the next day.

If there is anything I've missed or forgot to mention, please tell me ASAP and I'll make the changes/additions here. I apologise for the lack of co-ordination or notice since things have come upon me suddenly. We should manage to get some decent results.

Many thanks,

QK

EDIT: Edited aoTuV link
EDIT 2: Changed to mingw32 compile of 1.0.1 CVS
EDIT 3: Added GT3b2 with hfr. If you are already well into the test before I made this addition and don't feel like redoing everything again, continue with the original 4.
EDIT 4: I messed up the GT3b2/hfr link which pointed to a QK3.2 and GT3b2 combo....replaced with proper GT3b2/hfr binary

aoTuV 20040402 A link place is different.
It is as follows correctly.
http://www.geocities.jp/aoyoume/aotuv/aotu...nt_20040402.zip

However, since you may not be able to carry out direct download, please choose "aoTuV experiment [20040402]" from the following pages.
aoTuV test page

Thanks

A small step-by-step HOWTO do the test, to make things easier:

1) download all 5 Encoders listed in QuantumKnot's Post and place/unpack them in the same folder
2) download FLAC and OGGDEC and extract flac.exe and oggdec.exe to the same folder
3) download the samples you want to test from ff123's 64 kbit/s listening test samples (right at the bottom of the page)
4) copy paste the following code to a *.txt file:

5) VERY IMPORTANT: if you live in the us or japan replace the "-q4,25" with "-q4.25", if you live in europe (dunno about britain) leave it as is, otherwise you will only encode with "-q4", which wouldnt be good
6) replace "layla.flac" with the actual name of the file you want to test
7) rename your *.txt to *.bat and execute it (now the listening samples will get prepared for testing)
8) download the ABC-HR listening test tool
9) open it and go to "file -> setup test", under "orig wav" open the sample.wav you have on your disc now, under "wav" the other .wav files (dont care about offset)
10) enjoy your listening test




note that with these encoders -q 4.25 will not use 4.25 but 4!
you have to use -q 4,25 if you are in europe!

edit: added a small how to

hmm....could be a Windows localisation problem.

On mine:

Regarding the aoTuV encoder, to encode at q 4:



Note there is no space between q and the value, which contrasts to the other oggenc encoders.

It's -q 4.25 here, too. IIRC comma is used in the EU while period is used in the United States and Japan. (I don't know other region.)

The same problem occurred with past Roberto's multiformat test. I hope your reminder will be considered with the following tests. The option of providing an "accepts only dots" version is another solution.

Garf suggested I add GT3b2 with HF reduction at q 4.25 to the list so I'll make it an optional encoder:

http://www.rarewares.org/quantumknot/oggenchfr.exe

If you're already well into the test and don't want to redo everything again, just continue with the original 4 encoders.

EDIT: I messed up The binary I originally had here (OggDropXPd) was a combo of QKTune b3.2 and GT3b2. I've updated the URL to the point to the proper GT3b2 with hfr binary. I apologise for this. It's 2.30 am here so my brain stopped a long time ago.

ok just a first small finding:

the tunings are significantly better than 1.0.1 here (on the layla sample, but still)!

so it definitely makes sense to join the test and help finding which tuning suits best!!!

That's what was done in the former 128kbps test. Case provided a compile that accepted only dots.

i added a small howto to my first post to make things easier for everyone

I can't comment on the specifics but in the UK the decimal point is "." not ",".

How this effects vorbis I know not.

Fairy

That was my wrong. Thanks for correction.

OK guys, I've finished. Raw results are also available here.

[QUOTE=harashin,Apr 4 2004, 10:48 AM] OK guys, I've finished. Raw results are also available here.


for harashin

gt3b2+hfr and qktune are clear winners
mtb3 and 101cvs are clear losers

Harashin: Thank you for the results. And thanks for taking the trouble to tabulate them as well

ff123: Was that analysis done using the tool you describe here? It looks perfect for choosing which encoder. So after we receive say 10 results from 10 different people, do we run this on each listener's results or on the whole group?

Yes. That's the only analytical tool for listening test results I know of that is publicly available.



You must first create result tables for each sample. One column for each encoder tested, and one line for each listener.

So, you'll end up with 12 tables. Run each one of these tables through Friedman, and you'll get something similar to this:

(from the latest aac test)

Now, notice the line I put a <==== above. Take that same line from each sample result, and create a final table. Run this table through Friedman, and you'll get your final ranking.

Thanks Roberto.

Forgot to mention: You can use Phong's wonderful Chunky to parse the abc/hr result files into tables that can be fed to friedman. It reduces 2 hours of work to 5 seconds :B

ok i now finished all 12 samples too

i will not post my results till april 10 to avoid that anyone gets biased, but only that much:
1.0.1 is definitely last, lying 1.19 points behind the first one (funny its the same for harashin )

Nice. That definitely agrees well with the other listening tests.

Because I don't have too much time, I tested Waiting only. It's 12:03 after midnight as well... I'm not sure if the results below are valid or not, because I didn't test all the files, but since I took the time to do the test, I will post them anyway. Below, you can see that I'm not too sensitive or picky about Vorbis artifacts.

-------------------------------------------------------------------
ABC/HR Version 0.9b, 30 August 2002
Testname: Waiting - ogg

1R = C:\My Music\test_samples\waiting\QK32.wav
2L = C:\My Music\test_samples\waiting\1.0.1CVS.wav
3R = C:\My Music\test_samples\waiting\GT3b2hfr.wav
4L = C:\My Music\test_samples\waiting\MTb3.wav
5L = C:\My Music\test_samples\waiting\aoTuV20040402.wav

---------------------------------------
General Comments:
all abxable, but they are all not really annoying.
---------------------------------------
1R File: C:\My Music\test_samples\waiting\QK32.wav
1R Rating: 3.9
1R Comment:
---------------------------------------
2L File: C:\My Music\test_samples\waiting\1.0.1CVS.wav
2L Rating: 4.5
2L Comment:
---------------------------------------
3R File: C:\My Music\test_samples\waiting\GT3b2hfr.wav
3R Rating: 3.9
3R Comment:
---------------------------------------
4L File: C:\My Music\test_samples\waiting\MTb3.wav
4L Rating: 3.9
4L Comment:
---------------------------------------
5L File: C:\My Music\test_samples\waiting\aoTuV20040402.wav
5L Rating: 4.5
5L Comment:
---------------------------------------
ABX Results:
Original vs C:\My Music\test_samples\waiting\QK32.wav
7 out of 7, pval = 0.008
Original vs C:\My Music\test_samples\waiting\1.0.1CVS.wav
7 out of 7, pval = 0.008
Original vs C:\My Music\test_samples\waiting\GT3b2hfr.wav
7 out of 7, pval = 0.008
Original vs C:\My Music\test_samples\waiting\MTb3.wav
7 out of 7, pval = 0.008
Original vs C:\My Music\test_samples\waiting\aoTuV20040402.wav
7 out of 7, pval = 0.008

Thanks for the result. I'm not too sure whether it is statistically ok to include the result for just one sample. Can someone offer some advice here?

Of course it is. Just put it together with the other results you receive.

I get one-sample participants all the time

Finished too... My results are very different from harashin's. I'm the first surprised by my results. I could publish them, but it's maybe better to wait the end of the test.

To help anyone wanting to join the fray instantly, here's a little help.


I made two small scripts for this on Windows. Place them together with the samples, the encoders, flac.exe and oggdec.exe in a directory.

Call up Notepad, and copy/paste the first block. Save as blah.bat .



Save the following as listentest1.bat :


the version of listentest1.bat for the american folks :

This encoder crashs here. Most likely a problem related to SSE not supported by my processor (Athlon 1.3GHz - not AthlonXP); ICL compile from rarewares works though.

John33...?!

ok time for my results


note that the confidence intervals are surely not correct (i simply copy/pasted my results in an old table from rjamorim)



and the result files can be found here

The bars are pretty close to what they should be (each one in your particular case should be 0.535 long, or plus/minus 0.2675 from the average)

The Fisher's LSD is the length of each error bar.

ff123

ok thanks

so basically my vote would go to aotuv, qkt or gt3b2+hfr

dunno which one is the most interesting one technical-wise?

OK, time for mine (more negative against Quantum Knot tuning):


GREEN= best encodings
ORANGE= worse encoding


log files are available here:
http://membres.lycos.fr/guruboolez/AUDIO/vorbis128/

QK tuning are suffering from the opposite effect of CVS: low-energy/powerless/hollow sound. And sometimes there's noise and powerless issues at the same time.

If you average the scores across the 3 listeners who submitted full results, the table looks like this:



Which would yield:



ff123

Edit: fixed average table
Edit2: Addition of LoFiYo's results for "waiting" don't significantly affect the outcome
Edit3: Addition of Mac's results for "waiting" don't significantly affect the outcome

I only have one sample to contribute to your test, the others all ended up with each codec getting 5/5.. I'll try again if you have another Vorbis pre-test before proposed dialup test



Comments

I couldn't distinguish between any of the encodes except QK (sorry mate! ) - they all had a slight echo on the words that wasn't in the originals. QK was worse because one of the actual words sounded.. almost as though it had an incredibly mild distortion effect applied to it. This was the only sample I managed to spot any codecs on

Hope this helps a little, and thanks to everyone for working on these encoders

Many thanks to ff123 for analysing the results and also to those submitting.

It looks like there is a tie between aoTuV, GT3b2+hfr, and QKTune beta 3.2. aoTuV got the highest rating (3.76) out of the three so that means it was judged the best Vorbis encoder. Therefore, I suggest we submit aoTuV to the 128 kbps multiformat test.

Aoyumi: I suggest you make an oggenc.exe of aoTuV to make it easier for people to encode.

i have nothing against aotuv

No matter what encoder is chosen: Please provide me a compile that ignores user's local settings, and always uses either , or . for decimal separation.

I've PM'd aoyumi about providing an oggenc compile.

Is pre-echo code tuning (easily) pluggable for AoTuV? I've just read the short comments I've wrote during my tests, and it seems that AoTuV was more affected by pre-echo than other files (it's not a strong conclusion, and this supposition need specific tests for confirmation).

Anyway, it's sad to see that only three full sets of results were posted /

Yes, it should be quite easy to add pre-echo tunings to aoTuV though it is not going to be able to make the multiformat test. Aoyumi made a merge with Garf's tunings with a previous version of aoTuV so I'm sure he'll consider doing the same in the near future.

With regards to the results, it exceeded my expectations in a way since:

1. The test was organised and announced with very short notice
2. Vorbis has lost a lot of interest and attention, compared with codecs like AAC which enjoy active development and publicity.

nyaochi did PM me saying he would have his results in soon, though I assume he has been quite busy lately.

Thanks for the answer.



hmmm, not so sure. Lot of vorbis users are open-source supporters. The AAC agitation shouldn't really affect them. And by reading other generalist forums, I could easily conclude that vorbis is still more popular than AAC. But aac is certainly the new HA heartthrob; and if HA just precede (and make?) the public opinion, yes, I suppose that vorbis would lose popularity and interest in the next time.

Wait and see...

Welcome to the world of listening test conduction...

This is as a result of my test.
Please look at here. Listening Test


Since it was the test version, it did not plan to have released oggenc.exe etc. Since it seems that the so so result is obtained, I think that a plan will be changed. Please wait a little.


Yes, it is easy if it is pre-echo tuning like GT3 b1 and QKTune3.2. However, since there is a case which the bit rate increases superfluously, about this problem, it is deferment.


The portion which is likely to be concerned with pre-echo will not change with the present place 1.0.1.

I'd probably worried about pre-echo with AoTuV because other problems audible with CVS encoder were removed or lowered. Good new, and thanks for this clarification (and of course, for all your job)

OggEnc and the source code were uploaded to the test page.
In use, Please don't encode by the sampling frequency 30kHz or less. problem remain.

aoTuV test page

Aoyumi, your oggenc doesnt recognize the , / . separation rjamorim asked for

I don't think it's necessary, because -q4 should be OK with this AoTuV encoder (but more bitrate tests are needed to be sure)

Yes, I would be grateful if people could run bitrate tests. I am on my parents' Pentium 166, so encoding large batches of files is not really an option (specially considering they have a 2Gb HDD, and less than 200Mb are free)

I am sorry, an understanding was lacking.

By the way, is there any easy method of solving this?
Would you let me know, if some people know simple good solution?

Bitrate of aoTuV -q 4 definately isn´t too high on my samples... looks OK IMO.

I've also tested on some classical tracks: average is inferior to 128 kbps, which is common with vorbis and classical music (at least with all CVS version, from 1.00 to 1.02). I suppose that with other musical genres (especially louder music), bitrate should be really close to 128 kbps.