Multiformat@128 public listening test - CANCELLED Options

[rjamorim]

Hello.

I'd like to announce the opening of my Multiformat at 128kbps public listening test.

The test consists of comparing how 6 encoders: iTunes AAC, Ogg Vorbis, Lame MP3, Musepack, WMA Std. and Atrac3 behave compressing 18 different samples representing a wide spectrum of musical styles.

People interested in participating are invited to visit this page:
http://www.rjamorim.com/test/multiformat12...esentation.html

The test will end on May 16th and results will be posted soon after.

Best regards;
Roberto.

[kheops]

i'm in

downloading from bittorrent

first time, will try to do my best...

+

[Mindstorm]

I'd like to help but, unbelievable as it might seem, the last time I checked I couldn't distinguish a l3enc 1.0 encode from MPC Q8 :/ It's quite sad...

Anyway rjamorim thanks and congrats, I'm looking forward to the results

rgd

[rjamorim]

i'm in

downloading from bittorrent

first time, will try to do my best... 

Nice

If anyone is curious about the torrent stats:
http://www.rarewares.org/tracker/mystats.php

[kanuac]

Let's download!

This post has been edited by kanuac: May 6 2004, 01:03

[caligae]

Just for the next test:

Would be nice to include a shell script/Makefile for decoding on the non-windows platforms. Wouldn't be too much work and sure is pretty easy.

Doing the decoding by hand is error-prone too.

[ErikS]

Is saving/loading sessions disabled for this test?

If it's not, consider this a bug report:

using MacOS X 10.3.3
java version "1.4.2_03" JRE Standard Edition (build 1.4.2_03-117.1)

apple.awt.EventQueueExceptionHandler Caught Throwable :
java.lang.NullPointerException
at abchr.gui.modulecontrols.m.a(Unknown Source)
at abchr.gui.cb.c(Unknown Source)
at abchr.gui.bq.actionPerformed(Unknown Source)
[...]

[ErikS]

Just for the next test:

Would be nice to include a shell script/Makefile for decoding on the non-windows platforms. Wouldn't be too much work and sure is pretty easy.

Doing the decoding by hand is error-prone too.

http://www.comp.nus.edu.sg/~s0300325/decodeall.txt

Just don't take this as an example of how one should make shell scripts... It worked for me...

[bond]

great!

btw roberto you write "Ogg Vorbis auTuV tuning", it should say aoTuV

[sPeziFisH]

Don't be too sensitive on my reply, I just want to get an impression of peoples usage/utilisation of 128kbps-files.
Are you using 128kbps as it is a passable weighting of quality and diskusage with the focus on using transportable players or to have results which are comparable with older listening tests or just because it's too hard to rate files with higher bitrates (for example 192 kbps).
To go lossy I only use LAME's aps (sometimes musepack, but mp3 is still more usable with (cheap) everyday-life-hardware like Standalones (SA) or discmen, also my friends are more happy to get stuff in a format they get along with ) and this mostly on my Desktop-PC, so I guess (I don't really know, that's why I am asking) I am not in the target-group this test is started for or ?

...and now I have to laugh, just read your words at your page (stupid as I am I first started writing as my brain gave me a *ping* '128kbps? - so where's the need if you want to have fun with your music?'). Oh man, things become better, at least this is my hope.
As I spend time with writing this stuff I spend these lyrics to outta space... don't argue about useless, spacewasting words.

Cheers

[Luke_A_P]

Roberto,

Regarding bit torrents.
I have a suggestion to make, I'm not saying you should change it now but it might be something to think about next time.

People may find it easier to keep torrents open if you could bundle all the samples together in one archive for the bit torrent release.

Personally I use the default windows bit torrent client which opens one window for each torrent. I have 7 of them open for upload now and will keep them open for the next couple of days. They are swamping my start bar as it is, to open 18 would make it a little tricky to use my computer normally.


Thanks,

Luke

[Luke_A_P]

Is it permitted to discuss here which areas of samples to inspect for weaknesses?

Sometimes I find it can take quite a bit of listening to pick up a flaw which then become obvious in most of the sample as soon as you have detected it.

It might save people listening time and to share these points.

I also understand if you think it is best to have no discussion during the test.

Luke

[cabbagerat]

To go lossy I only use LAME's aps (sometimes musepack, but mp3 is still more usable with (cheap) everyday-life-hardware like Standalones (SA) or discmen, also my friends are more happy to get stuff in a format they get along with ) and this mostly on my Desktop-PC, so I guess (I don't really know, that's why I am asking) I am not in the target-group this test is started for or ?

I think you are missing the point of this test. 128kbps is a good bitrange to listen to for two main reasons (as far as I can see):

1. People can actually do it. If we had a test of lame standard and it's equivalents then we would struggle to get meaningful results with real world samples. I am sure I am in the majority when I say that at 190kbps LAME, Musepack, Vorbis and the rest sound identical to me (except for some known 'evil' samples). What this means is that except for a small number of talented listeners, all we would get is 5.0 ratings for everything.

2. It's a useful bitrate. For streaming. For small (solid state) portables. For short samples on download sites. 128kpbs is a nice medium between 'listenable' and 'huge'.

Anyways, I am busy downloading. This is the first test I will post results for, so hopefully I do ok.

[PoisonDan]

Personally I use the default windows bit torrent client which opens one window for each torrent. I have 7 of them open for upload now and will keep them open for the next couple of days. They are swamping my start bar as it is, to open 18 would make it a little tricky to use my computer normally.

Have you considered using ABC yet? This client is very convenient when you want to download several torrents simultaneously. And it has a global upload limit setting, so the uploads won't slow down your downloads too much.

[Luke_A_P]

PoisonDan, that is a nice client, I have set it up now, thanks.

I've never had any need for anything apart from the standard client before now but this does give you more control.

(I think I would still prefere everything to be in one torrent in the future but it is a minor point.)

Luke

[Zurman]

Ok for the first time I take part in a listening test.
I've begun with "hongroise", and at fisrt I couldnt make any difference between files. But then .... I noticed the length of some tracks slightly differ from others : it's then obvious to chose the original file, even if there is no hearable difference.

But there is something more serious : at the beginig of the original track, there is a VERY DISTURBING "clic".... And this click disappears in some tracks !

What must I do ? Put a bad mark (2-3) to those files because I can easily distinguish them from the original, or put a good mark (4-5) because they are even more pleasant than the original?

Edit : to be sure I made an ABX test... Here are the results

ABX log
ABC/HR for Java Version 0.4b4
May 6, 2004 2:59:50 PM

Sample A: Hongroise.wav
Sample B: Hongroise_XXX.wav
Playback Range: 00.000 to 30.000
2:55:17 PM p 1/1 pval = 0.5
2:55:21 PM p 2/2 pval = 0.25
2:55:25 PM p 3/3 pval = 0.125
2:55:29 PM p 4/4 pval = 0.062
2:55:31 PM p 5/5 pval = 0.031
2:55:34 PM p 6/6 pval = 0.015
2:55:36 PM p 7/7 pval = 0.0070
2:55:39 PM p 8/8 pval = 0.0030
2:55:41 PM p 9/9 pval = 0.0010
2:55:43 PM p 10/10 pval < 0.001
2:55:45 PM p 11/11 pval < 0.001
2:55:55 PM p 12/12 pval < 0.001
2:57:48 PM p 13/13 pval < 0.001
2:57:52 PM p 14/14 pval < 0.001
2:57:54 PM p 15/15 pval < 0.001
2:58:00 PM p 16/16 pval < 0.001
2:58:03 PM p 17/17 pval < 0.001
2:58:07 PM p 18/18 pval < 0.001
2:58:09 PM p 19/19 pval < 0.001
2:58:11 PM p 20/20 pval < 0.001
2:58:21 PM p 21/21 pval < 0.001
2:58:24 PM p 22/22 pval < 0.001
2:58:25 PM p 23/23 pval < 0.001
2:58:27 PM p 24/24 pval < 0.001
2:58:31 PM p 25/25 pval < 0.001
2:58:33 PM p 26/26 pval < 0.001
2:58:35 PM p 27/27 pval < 0.001
2:58:41 PM p 28/28 pval < 0.001
2:58:44 PM p 29/29 pval < 0.001
2:58:47 PM p 30/30 pval < 0.001
2:58:49 PM p 31/31 pval < 0.001
2:58:59 PM p 32/32 pval < 0.001
2:59:01 PM p 33/33 pval < 0.001
2:59:03 PM p 34/34 pval < 0.001
2:59:06 PM p 35/35 pval < 0.001
2:59:10 PM p 36/36 pval < 0.001
2:59:13 PM p 37/37 pval < 0.001
2:59:15 PM p 38/38 pval < 0.001
2:59:17 PM p 39/39 pval < 0.001
2:59:18 PM p 40/40 pval < 0.001
2:59:20 PM p 41/41 pval < 0.001
2:59:22 PM p 42/42 pval < 0.001
2:59:24 PM p 43/43 pval < 0.001
2:59:26 PM p 44/44 pval < 0.001
2:59:30 PM p 45/45 pval < 0.001
2:59:32 PM p 46/46 pval < 0.001
2:59:36 PM p 47/47 pval < 0.001
2:59:38 PM p 48/48 pval < 0.001
2:59:40 PM p 49/49 pval < 0.001
2:59:42 PM p 50/50 pval < 0.001

---------
Total: 50 out of 50, p < 0.001

O fault, and each time less than 4 seconds to find the good track
(I've hidden the track)

This post has been edited by Zurman: May 6 2004, 14:37

[Halcyon]

Is it permitted to discuss here which areas of samples to inspect for weaknesses?

I think this is somewhat similar to "is practise allowed"?

I hope so. Otherwise the test would have to be carried out with completely naive (as in domain knowledge/skills naive) testers.

I suspect that is not the purpose of this test, considering the forum in which it was announced

[Aoyumi]

I found the problem.
The version of vorbis used by the test was old. The ogg file was not created by aoTuV beta2. (It's aoTuV experiment [20040402]) If aoTuV test page to "aoTuV beta2 Win32 OggEnc-locale fix" (file name is oggenc_aotuv_b2m.zip) downloads and a suitable file is encoded, it will output the file which is clearly different.

vendor name is
aoTuV experiment [20040402] = AO; aoTuV b2 test [20040402] (based on Xiph.Org's 1.0.1)
aoTuV beta2 [20040420] = AO; aoTuV b2 [20040420] (based on Xiph.Org's 1.0.1)
...

[EDIT]
Correction of an expression

This post has been edited by Aoyumi: May 6 2004, 14:51

[2Bdecided]

I have the same problem as Zurman - this time with the Kraftwork sample

The .wavs are fine, but when played in ABCHR, there's a click at the start of the original, but not all the coded versions (2 out of six have clicks).

Maybe ABCHR is reading some data in the wav file as audio?

This is a very serious problem.

I'm using WinXP, audiophile 2496, just installed Java.


Also, are the files being ReplayGained in ABCHR? If so, why?! If not, why is playback so quiet?

Cheers,
David.

[ff123]

Ok for the first time I take part in a listening test.
I've begun with "hongroise", and at fisrt I couldnt make any difference between files. But then .... I noticed the length of some tracks slightly differ from others : it's then obvious to chose the original file, even if there is no hearable difference.

But there is something more serious : at the beginig of the original track, there is a VERY DISTURBING "clic".... And this click disappears in some tracks !

Possibly the clicking noises you're hearing are the same type that were reported for the last test, and unfortunately are associated with the abchr-java program. You can try increasing the buffer size under Options->Settings->Playback (default is 500 ms). Also sometimes just moving the starting position of the playback range using the selection bar down at the bottom will eliminate the clicking.

The short answer is that you should not downrate quality because of this clicking noise. Also, you should not downrate quality because you can tell the difference in files because of a difference in length.

ff123

[ff123]

I have the same problem as Zurman - this time with the Kraftwork sample

The .wavs are fine, but when played in ABCHR, there's a click at the start of the original, but not all the coded versions (2 out of six have clicks).

Maybe ABCHR is reading some data in the wav file as audio?

This is a very serious problem.

I'm using WinXP, audiophile 2496, just installed Java.


Also, are the files being ReplayGained in ABCHR? If so, why?! If not, why is playback so quiet?

The java library lowers the volume, for some unknown reason. Hopefully this is not too serious of a problem, since this occurs for all samples equally.

The clicking is serious, though, I agree. But there is not much one can do about it.

ff123

[PoisonDan]

The version of vorbis used by the test was old. The ogg file was not created by aoTuV beta2. (It's aoTuV experiment [20040402])  If aoTuV test page to "aoTuV beta2 Win32 OggEnc-locale fix" (file name is oggenc_aotuv_b2m.zip) downloads and a suitable file is encoded, it will output the file which is clearly different.

Oh no!

This is bad...

This post has been edited by PoisonDan: May 6 2004, 15:06

[ff123]

Is it permitted to discuss here which areas of samples to inspect for weaknesses?

Sometimes I find it can take quite a bit of listening to pick up a flaw which then become obvious in most of the sample as soon as you have detected it.

It might save people listening time and to share these points.

I also understand if you think it is best to have no discussion during the test.

Luke

I think the answer to this would have to be no.

Discussion would bias the results.

ff123

[DeeZi]

The version of vorbis used by the test was old. The ogg file was not created by aoTuV beta2. (It's aoTuV experiment [20040402])  If aoTuV test page to "aoTuV beta2 Win32 OggEnc-locale fix" (file name is oggenc_aotuv_b2m.zip) downloads and a suitable file is encoded, it will output the file which is clearly different.

Oh no!

This is bad...

Oh my god

[Zurman]

Ok for the first time I take part in a listening test.
I've begun with "hongroise", and at fisrt I couldnt make any difference between files. But then .... I noticed the length of some tracks slightly differ from others : it's then obvious to chose the original file, even if there is no hearable difference.

But there is something more serious : at the beginig of the original track, there is a VERY DISTURBING "clic".... And this click disappears in some tracks !

Possibly the clicking noises you're hearing are the same type that were reported for the last test, and unfortunately are associated with the abchr-java program. You can try increasing the buffer size under Options->Settings->Playback (default is 500 ms). Also sometimes just moving the starting position of the playback range using the selection bar down at the bottom will eliminate the clicking.

The short answer is that you should not downrate quality because of this clicking noise. Also, you should not downrate quality because you can tell the difference in files because of a difference in length.

ff123

I agree, I shouldn't.

But I will for sure be influenced by this clic or even the length by the tracks

By the way even with the buffer @ 2000 ms, the noise is still here

This post has been edited by Zurman: May 6 2004, 15:14