http://www.soundexpert.info/

After a quick look 3 things stand out:

1) There's no error analysis whatsoever on the results.

2) There's no reference when grading, so you're supposed to already know in advance what artifacts the codec produces, and then look for them, rather than identifying a difference with the original.

3) The rating scale is very crude, and unfit for small differences.

Other than these three flaws, it looks pretty reasonable to me. The first two are 'fatal' though.

--
GCP

Funny way to produce a 64 kbps vorbis-file... Maybe I should use '-q5 --resample 22500 song.wav' for best quality in future...

<edit>
More nonsens:

Why not use a decoder, which is known to work good (MAD/lame/mpg123)?
</edit>

dev0

Technically, this choice is actually quite defensible, but in reality -q0@44khz has had more tuning than -q1.75@32khz of course, and -q0@44kHz is also the recommended usage.

The idea isn't bad, he just didn't do his homework



Is there any reason to believe the one used is no good?

--
GCP

No reason besides the millions of bad mp3decoders out there
I didn't say that this decoder can't be good for whatever reason; I just tried to say, that for a public blind-listening test it might be a better ideo to use a decoder which has actually proven to decode the mp3s flawlessy. (Decoding an lame encoded MP3 with lame -decode sounds quite sensible to me)

dev0

Thomson decoder is good. It uses the FhG engine - the same as WinAmp.

And lame decoder == mpg123, BTW.

> og1 - OggVorbis audio format.
> CODER: OggEnc v1.0 (libvorbis 1.0) (Quality controlled VBR)
> - usage (32.7) kbps: oggenc -q 0 --resample 22050 test.wav
> - usage (64.1) kbps: oggenc -q 1,75 --resample 32000 test.wav
> - usage (93.9) kbps: oggenc -q 2,99 test.wav
> DECODER: OggDec 1.0 from OggVorbis package.
> - usage: oggdec test.ogg

Funny way to produce *all* vorbis-files tests:

For the 32kbps sample, maybe -q-1 --resample 32000 test.wav would do better.
For the 64kbps I wouldn't add anything that was already took into consideration.
...and for the 96kbps sample, why didn't they used the -q3 setting? In fact, that is the nominal 96kbps setting...

...just thoughts...

I don't see that as being a problem though, becuase to be honest I don't really know what artefacts each of the codecs produce. If I don't know specifically where to look for differences, then I probably won't see them, so to me, the coded file is good! In this sense, that would mean the coder is a good one, because at the end of the day, I want music that sounds the same as the original in small file sizes. If it's doing something really wrong, but it's not something I can detect, what's the problem?


And tbh, I like their idea, a public blind test. I don't really trust their methods though, if it can rate mp3 as being the 2nd best at 32kb, I think something is wrong.

What I would like to see, is a test like this done in collaberation by all the boffins at HydrogenAudio. You know the reliable and *honest* way to test, and what settings to use for each codec. I would like to see it stretched to 128+ bitrates as well, seeing as thats what most people use for music. Suppose it's harder to test though, so we would need to find some hard to enocode samples of music (that aren't fatboy & castanets!!!) Is that a possibility?

Hi, HydrogenAudio folks,

Thank you for all your discussions on SoundExpert project. Sorry for my not active participation in them (promise to change this). Nevertheless, I've read all your opinions about the project and indeed many of them were taken into account while improving it.

The next thing I'm sorry for is a lack of technical information about the project on the site. In fact only functional part of SoundExpert (test engine) was done, as I was not sure if the project survives and I could attract enough participants for testing. But in recent two months the traffic has tripled and that kind of uncertainty has gone. So in the nearest future I plan to prepare all necessary technical information and make the site more complete. While this important part of the project is under construction I've just put some tech. details in THEORY section in the form of FAQ. I would like also to put here the answer (and proposal) to the most frequent question:

Q. Funny way to produce a 64|32|96 kbps vorbis-files...
A. For command line codecs, especially with VBR, several (sometimes too many) alternative sets of settings are available for a given bitrate. Some of them, for example posted by participants of Hydrogen Audio Forum for OggVorbis codec are definitely better than SoundExpert ones. Probably, the developers of this codec could propose even better settings. I'm ready to replace all OggVorbis test files with the new ones if there will be some consensus among the forum participants concerning the right settings for each bitrate. OggVorbis developers are invited specially. All proposed settings will be tested for FBBR compliance, as it is ideologically important for SoundExpert testing (see FAQ on the site). FBBR values for proposed settings will be posted here. Some of them are below:
-q -1 -resample 32000 => OggVorbis ABR=33.0, SoundExpert FBBR=33.3
-q 0 => OggVorbis ABR=48.5, SoundExpert FBBR=48.9
-q 2.99 => OggVorbis ABR=93.6, SoundExpert FBBR=93.9
-q 3 => OggVorbis ABR=102.8, SoundExpert FBBR=103.1
It seems that "-q 2.99" is the only setting suitable for SoundExpert 96.0 kbps testing.


PsyTEL® MPEG-4 AAC Encoder settings are also discussible. Other codecs are incorporated into end-user applications, which give not much control over compression parameters. They seem to be OK.

And finally, several betas of MS Windows Media Player 9 with "9 series" audio and video codecs are available already. Any thoughts about testing?


WBR, Serge.
www.soundexpert.info