Public, Multiformat Listening Test @ 64 kbps - OPEN Options

[vinnie97]

Sebastian, can you tell me if I rated any references? Results should've arrived from vince dot hart at gmail. I fear the answer.

[Sebastian Mares]

The key will be available after the test is over so you can check yourself.

People, about two hours left!

[Sebastian Mares]

Hmm... I could use a bit more results for the last 10 samples. I am extending the test one last time until 15th since I expected a bit more results to arrive shortly before the test ends. Maybe Guru can test, too (BTW, received two mails from you - thanks!).

[naylor83]

Come on people! If you feel geeky doing the tests, remember it's all in the name of science!

[Alex B]

Also, even if you are not personally interested about this bitrate range this a good opportunity to practice with relatively easy samples. A future higher bitrate test will be much more difficult for the testers. We have recently tested 48 kbps and now 64 kbps. Possibly the next multiformat test will be 96 kbps. I also look forward to see the discussed 128 kbps MP3 test happening sometime soon.

[Whelkman]

Also, even if you are not personally interested about this bitrate range this a good opportunity to practice with relatively easy samples. A future higher bitrate test will be much more difficult for the testers. We have recently tested 48 kbps and now 64 kbps. Possibly the next multiformat test will be 96 kbps. I also look forward to see the discussed 128 kbps MP3 test happening sometime soon.

This is why I joined. 48 kbps is unrealistically low and thus uninteresting to me right now, but I'm having problems as early as 96 kbps. A new 128 kbps test may be more challenging than many are expecting. 64 kbps is just the right spot where you might be fooled some of the time, making it interesting for testing purposes.

[Sebastian Mares]

Well, in case anyone is interested, the next tests will be:

MP3 at 128 kbps
AAC at 80 kbps (featuring both LC and HE-AAC)
Multiformat at 80 kbps with winner of the AAC test vs. other codecs
And finally maybe a 32 kbps and a 96 kbps test, but these have very low priority

This post has been edited by Sebastian Mares: Aug 12 2007, 19:01

[Alex B]

Well, in case anyone is interested, the next tests will be:

MP3 at 128 kbps
AAC at 80 kbps (featuring both LC and HE-AAC)
Multiformat at 80 kbps with winner of the AAC test vs. other codecs
And finally maybe a 32 kbps and a 96 kbps test, but these have very low priority

It is good to see that you have plans for several tests since no one else has been interested about conducting listening tests.

The 128 kbps MP3 test will be interesting because LAME 3.98 should be soon ready and FhG has a new version too.

Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test, but I suppose you have carefully thought about the options and followed the ongoing HA discussions more keenly than I.

It would be good to keep the tests flowing in a steady stream and not allow as long breaks as we had now. I think more testers would be willing to participate if the test discussions are constantly kept alive. For example, if a test is not possible in the scheduled time frame because of codec development reasons you could start the next test from the list instead.

Also, I wonder if the test presentations, instructions/help/faq and the results presentations could be developed further. It would be nice to have an attractive and carefully organized web site where you could browse info about the ongoing test, the test schedule and the results easily. Perhaps naylor83 would be interested in helping with this. He did excellent work with the rarewares site.

[kennedyb4]

I too would be more interested in a multi-format at 96 with the AAC candidates being Nero and Itunes VBR.

This bitrate would be of more interest for those using portable devices I think.

[IgorC]

Come on people! If you feel geeky doing the tests, remember it's all in the name of science!

And science should be in the name of better life

[vinnie97]

Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test, but I suppose you have carefully thought about the options and followed the ongoing HA discussions more keenly than I.

I disagree. 48 --> 64 --> 80 = 16 kbps between each test iteration. The difference between Vorbis at 64 and 80 kbps has me very interested in this test. 80 kbps is the portable sweet spot for me when it comes to Vorby...I'd like to see if it's still #1 or if there's a similar transparency being reached across all formats.

Sebastian, you're killing me....I need to know how many references I ranked.

This post has been edited by vinnie97: Aug 13 2007, 05:50

[rockcake]

I'm also with vinnie97 wholeheartedly as regards 80 kbps tests - I'd also be interested in the results & suspect quite a few participants would be pleasantly surprised with the fidelity of the sounds; I for one was a little surprised in this test at the fidelity in some of the non-problem/general-music samples e.g. "Big Yellow" & "Les Voleurs" come to mind as being mostly quite listenable to my ears/head/brain IIRC.
Can't wait for the overall results!

Edit P.S.: I also dread my contributions being thrown out for whatever reason, so I tend to be a bit conservative and listen/rank like beto wrote above.
P.P.S.: Blimey, the next 128 mp3 test will be ruddy hard, I reckon!

This post has been edited by rockcake: Aug 13 2007, 06:35

[muaddib]

Hmm... I could use a bit more results for the last 10 samples. I am extending the test one last time until 15th since I expected a bit more results to arrive shortly before the test ends. Maybe Guru can test, too (BTW, received two mails from you - thanks!).

It would be really great if people with golden ears would help us with this test. I guess for them it would be very easy and they could do it in 2-3 hours.

[kdo]

people with golden ears would help us with this test

But the golden ears would have to rate all these samples between annoying and very annoying.

Whereas people with average hearing can use the full scale for the ratings. Like I did.

Hmm... I could use a bit more results for the last 10 samples.

I have a question: When somebody ranks the reference, do you disqualify the whole test for this sample or only the result for this particular encoder within the test?

Would it make any sense for those of us who ranked the references to repeat the discarded tests more carefully and re-submit?

[muaddib]

people with golden ears would help us with this test

But the golden ears would have to rate all these samples between annoying and very annoying.
Whereas people with average hearing can use the full scale for the ratings. Like I did.

Giving low grades wouldn't make any problems.
And because someone can hear very well and find those small differences it doesn't mean that he/she will grade those differences as annoying.

[guruboolez]

It would be really great if people with golden ears would help us with this test. I guess for them it would be very easy and they could do it in 2-3 hours.

I'll do it today or maybe tomorrow for all 18 samples - Sebastian will therefore have one more result for each one.

[Alex B]

I think the amount of ranked references in the future tests could be lessened with better instructions. ff123's practise page is very useful, but an updated page (with the JAVA version screenshots) inside the test site would be better.


The following things should be said very clearly:

- If the difference is not obvious, you should do an ABX test.
- If you can't succesfully ABX you should not try to guess. Don't pull either of the rating sliders down.
- If you didn't ABX test an obvious sample it would be good to listen to it once again and verify that you moved the correct slider.


Perhaps the instructions could have some step by step examples with screenshots. For example:

[Screenshot 1, the initial view] & a speech bubble:
"Hmmm... the sample on the left has less overall clarity and the symbals are a bit diminished, could it be the lossy file? Though, I am not sure, better go to ABX."

[Screenshot 2, ABX view, configuring the test]

[Screenshot 3, ABX view after a succesful ABX test] & a speech bubble:
"Oh dear, it was the other way around. This encoder has increased the volume of the higher frequencies slightly and at first sight it sounded better because the reference is not very bright."

and so on...

[naylor83]

"Oh dear, it was the other way around. This encoder has increased the volume of the higher frequencies slightly and at first sight it sounded better because the reference is not very bright."

"... Ah, that must mean this is the WMA sample! I'll bring it down an extra notch just for spite."



Edit: Seriously though... Good suggestions!

This post has been edited by naylor83: Aug 13 2007, 13:09

[muaddib]

All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

I am willing to help in building instructions for future listening tests (for this one it is to late I am afraid). Maybe someone could start this in Hydrogenaudio Wiki (if that is a good place).

[kennedyb4]

All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

I am willing to help in building instructions for future listening tests (for this one it is to late I am afraid). Maybe someone could start this in Hydrogenaudio Wiki (if that is a good place).

That's a fantastic idea. I downloaded the program and a few samples but got bogged down in the mechanics of things. Wasn't really sure what I was doing.

A single clear " how to " manual might have helped.

[kdo]

All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

In my case it would mean practically re-doing the whole test from scratcg, because most of the time I didn't do abx.
I did go through abx a few times when I was in doubt, and I did manage to abx successfully every time (or almost every, cannot remember). But there is still a chance I could have ranked a reference somtimes.

It would be nice if Sebastian could notify us by e-mail which of our tests are discarded and which we should re-do and re-submit. (Of course, if such thing is allowed by the test protocol)

[guruboolez]

The four first samples are tested on my side.
In order to avoid some mistakes I begin the procedure by ABXing all files and I start ranking them after this ABX procedure. Advantage: it's impossible to accidentaly mark the reference (only one slider can be lowered after a successful ABX test).

N.B. It seems that one competitor tends to make higher frequencies brighter (more noise/energy) which wouldn't be unpleasant if there wasn't additional artefacts. After lossy encoders, maybe the first "glossy" codec...

This post has been edited by guruboolez: Aug 13 2007, 15:40

[muaddib]

The four first samples are tested on my side.

I hope you will not get bored until you get to samples 7.-18.

I don't understand why so little results for sample 8. It is easy after short 'getting used to it'. However I will not send my results because we at Nero don't want to have influence on the results.

[Whelkman]

Add to the list explanation of ABX testing behavior. It was far from obvious to me, and I had to redo several samples after experimentation (at the time I thought using the testing mode to eliminate the reference was required in all cases, but apparently this isn't true--this should be covered, too).

Not obvious is that the listener gets only one shot at testing mode unless he wishes to redo all samples within the test or load from a good session. The program also has some strange ideas of what qualifies as a commit; if I begin testing, reset my trials, then decide to come back later, I'll find the test was marked as a failed commit and will not be able to try again without starting over or loading from a successful session save. At least twice I successfully ABXed only to have my results discarded and replaced with a failed commit.

The listener should also know that he needs to run at least eight trials in testing mode. Early on I tried finalizing when ABC/HR first displayed a number (6/6, I believe), but results came up in red and the reference was not discarded, which is not helpful, especially given the retesting issue.

I experienced numerous bugs while using ABC/HR Java including crashes, disappearing menu options, nonfunctional sliders, failed saves, and phantom ABX commits. The program becomes especially unstable after a session save. Are these issues known?

Had I not kept notes on each sample, I probably would have quit due to the above frustrations.

[[JAZ]]

I experienced numerous bugs while using ABC/HR Java including crashes, disappearing menu options, nonfunctional sliders, failed saves, and phantom ABX commits. The program becomes especially unstable after a session save. Are these issues known?

Had I not kept notes on each sample, I probably would have quit due to the above frustrations.

You play hard, it seems...

I've done three listening tests so far, and the only bugs i found were the non working audio (that gets fixed manually selecting the output audiocard), and the clicks when pressing play (which doesn't happen to me if i use the soundcard directly instead of java sound engine)

But if you say it happens *After* saving session... i've never saved session. (don't use to get interrupted in the middle of a test)

This post has been edited by [JAZ]: Aug 13 2007, 16:55