EDIT of Jan 9 2007:

Finally 192 kbit/s listening test has finished with the following results:

1. Ogg Vorbis, AAC+, AAC LC and WMA 9.1 showed the best audio quality
2. MP3 and MPC are second
3. AAC LC from iTunes is last due to the bug

See 192 kbit/s page for details. SoundExpert says “thank you” for participation and “sorry” for such long period of testing which was necessary for experiments with reliability parameter of ratings. Next tests will be substantially shorter. Once again - Thank you!



EDIT of July 16:

192 kbit/s listening test is opened. Final list of contenders looks as follows:

1. aac VBR@193.3 (NeroRef 1002)
2. aac VBR@197.8 (iTunes 6.0)
3. aac+ CBR@196.8 (Winamp 5.24)
4. mp3 VBR@200.5 (Lame 3.97b2)
5. mpc VBR@193.8 (1.15v)
6. ogg VBR@192.8 (-aoTuV-b4.51)
7. wma 9.1 CBR@198.0 (WMPlayer10)

First results (not reliable though) will appear immediately after first grades is returned by participants. Each rating needs about 300 grades. It might seem that a lot of work is required but fortunately a single testing session (one test file) is short and simple. So it won’t be too hard to test 3-5 test files at a time. You are welcome … and thank you in advance!

To participate in this listening test, please, download and grade a test file.


New 192 kbit/s listening test at SoundExpert is scheduled to start July 15. Here is my proposal of codec contenders:

1. mp3: Lame 3.97b2 [-V1 --vbr-new –noreplaygain] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR
4. wma 9.1 std: WMEncoder [bitrate VBR, 192 kbit/s] 188.0 kbit/s FBR
4a. wma 9.1 std: WMP10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6. mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR

Not sure about #4. Need your comments and other suggestions. Thank you in advance.

I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

Also, not trolling here, but MPC is more or less obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO. It is just unuseable outside of the PC, and even there its horrible seeking lag makes it a chore.

HE-AAC also, not needed.

About WMA.. 1 would be enough.

Instead of 2 WMA why not 2 vorbis, one tuned and one vanilla. Or add lame/FhG 192 CBR, now that would be interesting if properly done once .)

Is this a typo? Does HE-AAC even work at 192k? IIRC most encoders don't allow it above 96kbps. Even if it technically works, it seems like a pointless test.

It gives 173.1 kbit/s for SE test samples. I'm not sure ...


Even if it shows better results?


BTW, people from Nullsoft ask money for the use of this encoder above 128 kbit/s.


There will be single wma (at least for now). The question is which one - cbr from WMP10 or vbr from WMEncoder?

No, this is reality. It is new CT High Bitrate HE-AAC Encoder for bitrates 128 - 320. It showed surprisingly good results at 320 kbit/s and I decided to test it at other bitrates in order to check whether it is realy good or there is a flaw in testing methodology.

Those results are clearly wrong. At 320k MP2 will sound awesome. So will MP3 and AAC. Theres nothing to test at those bitrates, so theres clearly a problem in their methodology. Looking at their setup, I'd say the most obvious problem is that they ask people to answer which is the most "degraded" sample without telling them what the original sounds like, which sort of calls into question what exactly the results mean.

My guess would be "nothing at all".

Format selection looks good, even MPC since the last encoder update was 2005, same year as ogg aoTuV 4.51 and Lame 3.97b2.

No test of WMA Pro or iTunes 6.04 AAC? Might be good to include iTunes AAC as it has not had a test against the recent Nero AAC 1.0.0.2. Which one would edge out the other this time...

Can't wait for the results on this test and the 128kbps test.

- Gow

Suggest you use known codec killer samples because at these bitrates transparency will be the norm.

192 test and you want to use LAME -V1?? For my music, V1 is almost always 250 kbps or more - not even close to 192. I suggest -V2 or even -V3 if you want a fair comparison.

Why not throw the iTunes AAC encoder in the mix as well? It is probably the 3rd most popular encoder out there - behind LAME MP3 and WMA (referring to the greater population, not the HA population)

Just my personal opinion... but the last 128kbps test show that is almost imposible to tell the diference using new codecs even at that bitrate... is just insane that people can tell the diference at 192kbps... only a few "bat ears" people are going to take the challenge...

Personaly for me Ogg Vorbis reach transparency at 128kbps and only if is a dificult sample I can tell the diference with real care using ABX because I probably never will going to hear the diference if I hear the original first and the encoded later... at 160kbps Ogg Vorbis is just over kill (for me), I will never can tell the diference of the original and the encoded one...

I personaly would like to see a test at 96kbps...

I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.

Keep with the tests with the two versions of WMA is a good idea,look how is going the improvements in wma.But why not 128 this bitrate is the most used in internet and is a "high compression" when 192 is an ''average compression"

Please show me the results of an ABX test which back up this statement!
If not, please read TOS #8 once more...

Back on topic:
I see no reason to perform an ABX comparison test at a bitrate above 130-150kbps. Cause the 128kbps test show that 90-something % of all test samples are already transparent. So I can't imagine how hard it would be to differ lossy and source at a bitrate 50% above this. Good luck!

Oh I'm sorry about that.

Sorry, this is my fault – there is no popular explanation of SoundExpert (SE) testing methodology on the site still (except two boring ones in pro zone). So I have to explain over and over again: SE testing uses sound artifacts amplification technique (SARTAMP) which helps to evaluate perceived audio quality of devices and compression algorithms with extremely low artifacts and distortions, not audible in “normal” cases. Thus, all SE ratings above 5.00 show amount of perception margin of human auditory system for those tested devices (only codecs now). The absence of reference sample in SE test setup is not essential as the artifacts are clearly audible to more or less extent. The actual aim of SE listening session is to grade the annoyance of those artifacts.

Since 2001 a fixed set of reference (critical) sound samples (from SQAM disc mostly) is used for SE listening tests. All average bit rates are calculated on basis of these samples and called “file based bit rate” (FBR). As the sample set consist of different sound pieces including solo instruments, voices and speech the resulting bit rates (in VBR mode) are slightly lower than in case of pop/rock music (quite close to the bit rates for classical music though).

For now I’m going to add only one coder per format. For AAC I would like to test (and promote) Nero encoder because it is new, good and free. For WMA I tend to choose 192 cbr variant from WMP10 because it represents more real life scenario than WMEncoder with 192 bitrate VBR. The more so, as there is no problem to add other contenders later (say, in the autumn) including new Lame, Ogg, iTunes, Winamp and Vista ones.

I'm not sure that test is very useful either.

It rates transparent encoders by how far their distortions/artifacts are below the masking curve. What matters is how reliably they reach transparency, not how far 'beyond' they get I'd say.

What is the purpose of this? Isn't a listening test supposed to test how well a codec performs under normal conditions?

In general right you are. But if you want to apply some sound enhancements during playback (EQ, wide stereo for example) or transcode to lower bitrate those artifacts could (and definitely will) suddenly appear. Also don’t forget that any listening test is performed only with finite number (10-20 at most) of test samples and then the results are generalized to all music. That’s why the more perception margin - the better. Most people understand this intuitively and use slightly higher bitrates than usually claimed to be transparent (128 – according to last test).

Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons. I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...

Well, fortunately or unfortunately “the real human listening” is far from conditions supposed by musicians, producers and sound engineers. Are you brave enough, Ivan, to say to Nero aac codec users that after encoding their music @128, all those EQ, SRS and other cool things have to be used with great caution

Yeah, and I want to be a millionaire. You didn't follow the 48 kbps listening test discussion and the WMA 10 / WMA 9.2 problem, did you?

Yes, I did. But I though it should better to add an interesting codec.

Ok, but have you shown that the results are relevent to anything? Because saying that 320k AAC+ SBR is the best encoder REALLY makes me doubt the relevence of the results.

In addition, unless I am missing something, you have not addressed the other objection I raised to the methodology.

Methedology aside, I would like to add my input on the formats being tested.

Since Microsoft is moving to WMP10, I suggest sticking with it and only using one WMA setting with WMP10. I would also suggest throwing in iTunes AAC. You really can't have WMA unless you have iTunes AAC as iTunes AAC is Apple's answer to compete with WMA and other lossy formats.

So I suggest including Lame 3.97b2, iTunes AAC, the latest free Nero AAC, WMA 9.1 (with WMP10), ogg, and mpc. I really see no need for testing HE-AAC at such high bitrates especially when there is no hardware support for HE-AAC. Granted, the hardware support for MPC is zero to none but it is a more "establilshed" format that is on its last leg. The only reason I could see testing HE-AAC would be to show that Nero AAC and iTunes AAC (which are LC-AAC encoders) can perform equally well and that HE-AAC encoding above 64kbps is just not needed.

As for Lame, I too suggest going with -V 2 --vbr-new (nothing else) even if the bitrate is a little low. For my music (metal), when I used -V 2, the average bitrates I experienced were about 250kbps. -V 1 was just too high as it would produce average bitrates of 270kbps with metal.

although this is probably the test i would be must interested in, i doubt you will get many different results that are outside the margin or error

In order to prove relevance of the methodology a big thoroughly organized standard listening test has to be performed. Sorry, for now I haven’t such possibility. All I can do is to test as many different codecs as possible and to watch how SE ratings correlate with results of other listening tests. That’s what I’m doing.

Concerning CT High Bitrate HE-AAC Encoder @320 see this HA discussion. I will add this codec to 192 and other SE groups to investigate the situation further.


If you mean absence of reference sample see my post #16, end of first paragraph.

As usually discussion on codec contenders turned into discussion on methodology flaws. May be it’s a good idea to open special topic, say, “SoundExpert testing methodology criticism”?



Now I plan to add only one codec per format. Others could be added later. What AAC coder you prefer to include in this case – Free Nero or iTunes. I would prefer Nero. I’m pretty sure that Apple’s codec will have lower score just because it produces very audible artifact with one of SE test sample (glockenspiel) – a bug definitely.


But -V2 produces only 173.1 kbit/s on SE samples. Other contenders are 191-198 kbit/s. Will it be honest competition?

Unless these ones are killersamples, -V2 is transparent to me (didn't ABX lately, but I had problems with V6 and V5, so I just assume this). When using SARTAMP, all encoded samples might have audible artefacts.
BTW, I still don't get the pourpose of SARTAMP.

Without SARTAMP all codecs above 160 kbit/s seem to be equal. This is not so indeed and the technology helps to rank them above 160 where standard listening tests become unreliable. In some circumstances greater quality margin is useful. In any case it is useful to preserve as much sound quality as possible at given bitrate.

Those people, who don#t understand the way soundexpert tests work, should keep quiet and just watch at the moment, and make a criticismn topic besides this topic.

As example, I recommend to have a look at the 128k soundexpert test, which reveals, that these results confirm some usual knowledge about the qualities of the different encoders.

So, see the soundexpert tests as additional special tool, to collect informations about the encoders.

btw., everybody who thinks, that 128 vbr with modern codec, is transparent HiFi, should visit an ear doctor and avoid listening to 10 €/$ pc speakers/ear-/headphones... and avoid listening to live concerts with unprotected ears...

...like HE-AAC absolute superiority at 128 and 320 kbps.
http://www.soundexpert.info/coders128.jsp
http://www.soundexpert.info/coders320.jsp

AFAIK, it goes against all studies, listening tests and developers' experience.


Who the hell are you to give such stupid advice to people? Modern encoders are most often very good. Collective listening tests performed by Roberto, Sebastian and Gabriel have proved it four consecutive time. I'm a trained listener; I can ABX lossy encoders at ~200 kbps but I must also admit that ~128 VBR encoders are often damn good and even transparent, even on critical listening conditions. Just count the 5.0 mark on this test. And several people here have confirmed how difficult ABXing has become at ~130 kbps and how satisfied they are by these encodings. I personaly find your opinion as very haughty - especially for someone who hadn't brought any ABX test at higher bitrate on this board in the four last years (or did I miss them?).

Sorry, but I consider criticism to be very much on-topic. You don't have to worry about me repeating "issues", though -- I've already said what I had to say. But let me link to a thread where Ivan, Garf and myself are questioning this testing methodology here.

After pressing "preview" and seeing that guruboolez already commented on your last paragraph (I totally agree) I'm not going to say anything else to that one.

Sebastian

I'm new to this forum. I don't understand why he-aac and mpc are going to be in the test when the only thing they work on is a computer. It would be good if the itunes aac encoder is there because its proven to be a compedetive codec in the past and because I use it. Why don't you use the wma encoder from WMP11. Finally what is the difference with LAME V1, V2, and V3.

The only reason I suggest throwing in iTunes AAC is that it has yet to be compared against the new Nero AAC encoder. Yes, I think it is pertinant to test the new Nero AAC encoder since I don't think anyone has tested it yet but I think it would be worth while to stack it up against a very popular AAC encoder, the iTunes AAC encoder. I would much rather see Nero AAC (LC-AAC) and iTunes AAC (LC-AAC) and take out the WinAmp HE-AAC encoder especially since hardware support for HE-AAC is zero to none. Maybe HE-AAC could be added later on and keep iTunes AAC and Nero AAC in the initial runs.

As for the -V 2 having a ~20kbps lower bitrate, this might not be fair but I think it should still be tested especially since so many people use -V 2 over -V 1. Again, my claim on thhis might not hold any merit as I have had little experience on setting up proper ABX tests.

I think I am going to continue to push iTunes AAC though, that way it can be directly compared with Nero AAC just to see which one is better. Besides, iTunes AAC has almost always retained a very slight lead (not statistically higher but still) over Nero AAC. It would be nice to see the tides turn towards Nero AAC if those are the true results of the test.

Welcome back to the topic. I think, instead of fitting codecs and settings to SE groups, better to fit the groups to widely used codecs/settings. Frankly talking, those groups are created for user convenience only. SE system allows to group codecs according to different criteria. For example: CBR/VBR/ABR or according to quality ratings or actual bitrate as for now. Present division by bitrate is quite reasonable but it is not a strict guideline for testing. While theoretical efficiency of compression algorithms is interesting mostly for developers and advanced users, practical efficiency of widely used codecs and settings is interesting for majority of average users. Taking all this into account I suggest revised list of contenders:

1. mp3: Lame 3.97b2 [--preset standard] 177.1 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6. mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

Sorry, I have to include #7 for research purposes.


... because it is beta (not finished/tuned yet) and because it’s illegal to publish results of testing according to WMPlayer 11 End User License Agreement (EULA).


They are quality settings of Lame encoder which affect resulting audio quality and actual size of output mp3 files.

Since everyone is now using --preset fast standard (aka -V2 -vbr-new) instead of the vbr-old setting, it would probably make more sense to test the fast version. As a historical comparison (and for die-hard 3.90.3 fans ), additionally testing the legacy Lame 3.90.3 at the --alt-preset standard setting might be an option if you still have room for another contender.

Nearly all codecs are transparent to most people at that bitrate, so artificially amplifying artifacts that are inaudible in normal listening setups only serves to determine which codec is "best" to transcode from, and transcoding from lossy sources shouldn't be done in the first place.

Lossy codecs aim at transparent perceived sound at the lowest possible bitrate. As gaekwad2 already said, this methodology rates how far a codec goes "over the top". When two codecs produce transparent results at a certain bitrate, it only shows which one might sound better at lower bitrates.

Since the test is most probably done regardless of this criticism anyway, I suggest waiting for aoTuV Vorbis Beta 5.

@ Silversight and everyone else:

Stop talking about the test procedure. SoundExpert has been doing tests like this for some time now. It's not necessary to take part or accept the results themselves. If it is really neccesary, i would ask for separating this conversation to another thread.

@ Serge :

If you talk about using standard settings for LAME, shouldn't this be applicable to Nero, Vorbis and Musepack aswell? and if you use 3.97b2, it would be better to use --vbr-new, as already noted.

Edit:
suggestions:
vorbis -q6.25 or -q6
nero : -q0.625 or -q0.6
mpc : -q5.5 (?)

Ok. Done.

1. mp3: Lame 3.97b2 [--preset fast standard] 173.1 kbit/s FBR



The only reason of choosing “unusual” quality settings was to achieve target bitrate as close as possible. Also AFAIC these codecs are equally tuned for all q-parameters. But if there are some “popular” values I’m ready to make changes.

Perfect. Varied selection, iTunes vs. Nero, etc.

Will be interesting to see the results.

- Gow

min = 177.7 kbps & max = 198 kbps [iTunes, WMA]. Difference is higher than 10% (and 20 kbps).
Moreover, it seems that you decided to use very precise quality level (0.626, 6.37 & 5.49), probably to match a precise bitrate - but a different criterion (popularity) is used for LAME. This second criterion leads to considerably lower the bitrate and therfore handicap the format. It's likely that for most people there wouldn't be any perceptual difference between V1 and V2 in normal listening conditions but isn't your methodology supposed to amplify the distortions?

I'd use the same criterion for all encoder and try either to follow "popular" settings" or to match the same approximate bitrate for all. You said that:
“1. mp3: Lame 3.97b2 [-V1 --vbr-new –noreplaygain] 200.5 kbit/s FBR”
Then use V1. The deviation would be much lower (min = 191 kbps and max 200 kbps => ~4...5%) and even perfectible (by increasing MPC bitrate).

I rounded q-parameters for Nero, Ogg, MPC:

1. mp3: Lame 3.97b2 [--preset fast standard or –V1] 173.1 or 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

Now I see the only point to be decided - what setting to choose for Lame: “fast standard” or V1? If there is more interest in testing “fast standard”, it could be moved safely to 160 kbit/s group at SE after this test finished.

Coherency is the most important point IMO. If a majority of people would rather see HE-AAC at 64 kbps instead of LC at 192, would you include the first one among all ~200 kbps encodings?
-V2 is clearly too low bitrate according to your methodology; V1 is marginaly higher. The second one is the best choice if you want to keep the comparison fair.

Other solution: try to build a bitrate table based on various albums. Values may significantly change (and would correspond to a real/daily usage of codecs)

Why not? Especially if they are comparable in quality. I know at least one ISO listening test with different codecs at different bitrates. So, what - "coherency" or practical usefulness? And last but not the least: more interesting contenders - more participants.


Unfortunately this approach is not ideal as well because actual figures will depend on type of albums chosen for bitrate calculations. Add more classical music albums and you’ll get lower values, more hard/metal – higher values. Such figures will be “just for reference” in any case like SE ones.

You call it a '192 kbps' test, so you should choose the setting that is closest to that bitrate. If -V1 --vbr-new is closer then you should choose that one. You seem to have chosen the setting for every other codec closest to 192 kbps, so why not for LAME?


No it can't. IMO 173.1 is neither close to 160 (8% diff) or 192 (10% diff), both differences are too big. If you want to call your test '~ xxx kbps-test' thats fine, but at least let all contenders respect that bitrate. Otherwise the name of your test is completely irrelevant to what you are actually testing.

Ok. You win, guys:

mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR

The next test will be heterogeneous for sure and will include most popular codecs/settings. I have to think out a proper name for it or make it completely nameless in order to simplify discussion.

This set seems to be final now:

1. mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

I seriously doubt you've looked closely at those results. Do so now, and then try and find me a test showing 128k AAC-HE beating all other codecs. Unless I've missed something, you cannot. The ranking of AAC-LC and Ogg encoders also looks contrary to previous results.



Yes, but what information? Given that they do not correlate well with double blind listening tests, what exactly is being measured?

No one seems to be able to answer that.



I'm guessing this was meant to be a clever way to stay inside TOS#8 without having to support your assertion, or really even provide an arguement beyon "I'm right, you're wrong".

IMHO, you should change the test target to 198 kbps and tweak Vorbis, Muspack and Nero to produce exactly 198 kbps. Then all contenders would be within +- 2.6 kbps of the target. (195.4 - 200.5)

Why is WMA the only codec using CBR? why not use VBR on WMA as well?

Out of curiosity, I tested LAME, Vorbis, Nero and Musepack bitrates with the same set of 25 test tracks that I have used previously [1] [2].

At first I encoded the files with LAME 3.97b2 @ -V2 --vbr-new. The average bitrate was 196 kbps.

Then I tweaked the other codecs to produce exactly the same average bitrate. After a few encoding rounds I settled with these final Q settings :

aoTuV 4.51b : -q 6.20 (196 kbps)
Nero (2006-05-26): -q 0.552 (196 kbps)
Musepack 1.15v: --quality 5.75 (196 kbps)

I think that Q settings like these would be more comparable and fairer.

I used Mr QuestionMan for bitrate measurements. The MrQ reports are available here: mrq.zip