the is vorbis dead thread (quite a long read) has given me an idea. although i lack golden ears, i want to do a listening test anyway. i know that there is a 128 MP3 test going on, and that there will be an AAC test soon, but i want to compare codecs that i can download and use without paying anything up front (as in free legal software, i don't care about the patent issue thing).

those include, AFAIK, QT-AAC, psytel aac, faac, NT??-AAC, and maybe a few others on the AAC end. also, there is vorbis: Quantum knot's low bitrate tune, garf's tune, that AoTuV tune, and the official 1.01.

i know there are a lot of samples out there, and im looking to compare on a few properties (this is a test at approx 128 for portables - ipod and the vorbis players). overall sound, pre-echo and the like, filesize, and encoding speed are my primary concerns for each codec.

i know that MP3 still runs strong, and that MPC is very good, and that WMA is the most supported on portables after MP3, however, i wish to exclude those (128 tests have been run on them and we all know what they can do. now that i think about it, WMA sounds like a good idea:its also free and is undergoing development. can someone tell me where i can find a WMA9pro encoder or just WMA9 if pro costs money.

so i am asking for good samples to test as well as others interested in performing this test with me (the more opinions the better).

Just a quick note...

If you're serious about running a listening test, you should know that the standards for that thing are pretty high around here with all the work that ff123 and roberto have done. If you want to offer something comparable, you should realize that it's going to take a fairly large amount of work.

Second, if you're going to bother to hold a test, you need to be very clear from the start about what exactly it is that you're testing for. The more open ended and unfocused the test, the less useful it will be.

If you still think this is something that you can/want to do, you should read through the previous threads on this sort of testing, then go familiarize yourself with the abc/hr software. You should be able to find most of the rest of the information you'd need in doing so.

wma9 pro is not supported on portables!

Nice. What bitrates are you going to focus on?

EDIT: It would be nice to include the recently merged GT3b2 too so we can judge its quality. If it comes out with flying colours (ie. it sounds no worse than GT3b1 or 1.0.1), then finally we have the best of both worlds in a single binary.

What if to do 2 tests?
1. Low bitrate
2. For transparent quality, taking GT3b1 q5,0 (and its 180Kbs) as anchor.
What do you think of it?

I think it is good to include MP3, with 128 kbit and 192 kbit. Those two sizes are widely spread and is still the most used format.

dude, choose one bit rate for the test you have to fix all of the variables except one and that one is the one that you wish to identify differences in. So for example multiple codec, all at a single fixed bit rate with the same samples in this differences in quality will be down to the variable codecs.

Good luck. Go for the 128k or 160k but not both. For these sorts of CODECs you don't want to test higher than 160k (VBR/ABR) coz theirs no need.

Fairy

i was thinking mostly towards portables and bitrates that are most often used on them. at this point, i feel that portables greatly influence the choice of portable you buy, and once you own that portable you will want to encode in that format (MP3 is still #1 popularity wise but i feel that it has reached the end, and that the newer codecs will surpass it).

2 tests is an option, but it would be a lot to handle. one test at a time sounds like the best idea. I was thinking of doing a 128 test using aac, ogg, and WMA9 (i don't fully understand the concept of an anchor, please explain) on both hard samples as well as some specific genre's in order to compare performance under general use. the main focus would be which file was the better sounding encode (duh). im not sure what sub-sets of that exist, but i know that i've heard some pretty horrible 128 encodes and know the difference between tolerable and horrible.

i am very interested in doing some formal testing of the new vorbis tunes.

the definites are QT-AAC, Psytel, FAAC, and vorbis 1.01 (since GT3b1 was tuned for q5 and higher i doubt it would be useful to include in a 128 test, but the QK tune and the combined GT3b2 are very likely to be tested. aoTuV AFAIK was tuned for very low bit rates, though i could be wrong).

are there any free AAC encoders that i have overlooked? thank you for your input, i would really like to run this test.

ok, to make this easy im going to use the samples in the 128 MP3 test.

I plan on using psyTEL, FAAC (i need settings for 128), NCTU-AAC, QT-AAC, WMA9, WMA9Pro, vorbis 1.01, and the other low tuned vorbis encoders (QK, and the GT3b2 combo). that seems like a lot, so depending on the quality (FAAC looks pretty poor to me, so it may end up as anchor or i may remove it. also psyTEL is no longer being developed AFAIK, and therefore it might be best to remove it. opinions needed).

it will be a 128 test, and if it goes well there will be a follow up 180-210 range test.

EDIT: while encoding the files, i ran into some trouble with WMA. i can't seem to find a batch feature or a decent program to encode with. where can i find a good encoder for this?

In my opinion, it might be okay to remove psytel since, as you say, it is not being developed anymore, whereas FAAC is.

The other issues to sort out are the VBR/CBR issues. Essentially, are you going to select quality levels in Vorbis so that the average bitrate is about 128 kbps, use -managed -b, or just use -q 4?

on the AAC encodes I used -b to get around 128. for vorbis, i used -q 4. of course, some of the vorbis encoders went over by 20 or more, but thats ok i suppose.

the line up is (minus WMA):
aoTuV alpha 3
QK tune beta 2
Xiph 1.01
FAAC 1.20.1 (no compiled versions of 1.23.1)
psyTEL AAC (might remove because this makes for a lot of samples)
QT-AAC 6.5
NCTU-AAC 1.0 RC1

they are all encoded around 128. i'll start my own testing soon, and if i like the way it goes, i'll open it to the public and get a good set of results.

if i have screwed up somewhere so far, please give me a heads up. thanks.

imho forget wma, if you remove psytel then 6 samples are still plenty for such test.

You should also add MP3 192 kbit as an anchor, because so many songs has been encoded in that format. It is very usefull to see the differendes between mp3 192 and the others, because I think much people want at least mp3 192 kbit quality.

well, i started doing some of the samples. after you find out what just isn't 'right' in the sample, it is easier to ABX, but it takes a while to discover it. so far i have been doing the vorbis encodes, which have all featured either high frequency annoyances or a dulling of the background noise.

i have yet to discover the major flaws in the AAC encodes because i am not really familiar with the artifacts found in them. i'm going to look over the comments on the 128 AAC test and check out my samples.

after i have done some testing myself, i will set up a way to distribute the file so that others can participate if they so choose.

So what happened here?

Maybe an open-source audio codec listening test would be more meaningful.

If no one is planning this, I might sit down and work out a simple Open Source listening test and make it fairly quick and easy. Ogg Vorbis vs. FAAC vs. Lame. @ 96kbps

We should wait until Roberto's test is complete first though.

Or should this just get a new topic?

Roberto is postponing his test indefinitely due to health concerns. So you can start it anytime.

I was under the impression that he was postponing the listening test until his own personal test results came back. At that point the decision to indefinitely postpone his listening test will be made.

In any event, I need to start reviewing the opinions of tuned Vorbis codecs. My initial thought was to just test 3 codecs/programs: FAAC, OggEnc-1.0.1, and Lame, but part of me thinks that the QK-tuned and the aoTuV-tuned codecs need to be at least considered and debated.

I will postpone until my biopsy results arrive. They should arrive early next week. If it's only a cyst, the test will start on April 21, or April 28 the latest.

I would be more inclined to replace 1.0.1 with one of the tuned encoders. 1.0.1 has been tested many times in various tests and nothing much has happened since then. These tuned encoders are all "open source". Plus I'm confident that by chance or by reverse osmosis, one of the folks from Xiph.Org will notice it and perhaps consider it for future inclusion in the official binaries.

In a quick browse of this thread, I haven't noticed anyone mention the iTunes encoder which is free. Seems it should be included in a test of free AAC encoders vs. Vorbis encoders.

Well, right now, my biggest concern with testing the tuned versions is the question of bitrate/quality accuracy. Even with GT3b1, the bitrates went up by 20kbps. Upping the bitrate by 20kbps would not be fair in a 96kbps competition.

Otherwise we could test the sthayashi-tuned encoder where there's the added line:And we could watch Vorbis blow away the competition (presumably).


The test I'm interested in isn't about free, but about open source. If Roberto's test is conduced, I suspect that iTunes will once again beat out Vorbis, so there will not be anything new to test in this area.

However, if there's a worthwhile free AAC encoder, I'll be happy to consider it for testing.

Well, you could simply encode a lot of different music with different quality levels and select the level that averages at 96kbps (I believe this was also done for the multiformat test). If the bitrate is a bit higher on the test samples that probably means Vorbis is good at detecting hard to encode music.

From my experience with aoTuV at least, the bitrates seem quite well behaved. The tunings are different from the ones in GT3b1.

Are there any other open source perceptual coders out there that we could include? I guess MPC can be considered open source since the source is there but does it sound acceptable at 96 kbps?

Yep Can faac be used in VBR mode? If so, then there shouldn't be any problem with CBR versus VBR issues in this sort of test since all the encoders will operate in VBR and the q settings will be determined in a consistent way (average 96 kbps on lots of different music).

Just out of curiosity: What would be this test's merit? Deciding what is the best Vorbis tuning at a particular bitrate?

My original idea was to find the best open source perceptual encoder available. We already had the Vorbis listening test to find the best tuning. This contrasts to MGuti's original idea of testing free encoders (can be downloaded in full without any cost). I guess I should've made a separate thread for it but I didn't at the time since it was just a suggestion

@sthayashi, well I was talking about MGuti test, and not yours as iTunes conforms to exactly the criteria that was stated at the begining for inclusion in that test (i.e. a codec that can be downloaded for free).

I agree it is more interesting.

An open source-only codecs test would only appeal to people not using Windows. A free codec test would appeal to people without money and too moral to use pirate software - a broader audience, IMO.

Besides, people not using windows are often open-source enthusiasts that would only use vorbis anyway. Or they are Mac enthusiasts that would mostly use iTunes AAC anyway.

One thing iTunes doesn't perform well with is movie-audio encoding tools. Although granted it is possible to use iTunes for this (I think), there is very little information about it and most people who encode movies generally don't use iTunes.

Also, from a pseudo-scientific point of view, I'd like to refrain from doing tests that have more or less already been done. iTunes vs Vorbis has already been done (along with many other codecs) and will be done again if rjamorim's test is good to go. I don't think there will be anything new to be shown by including iTunes in such a test. And given everyone's time constraints, I'd prefer to keep the number of codecs as small as possible. 3 or 4 at the most.

I concur. Nearly all the codecs in the 128 kbps multiformat test (iTunes, LAME, MPC, WMA Std) are free to download anyway so that kinda overlaps with this test.