Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 64 kbit/s Test has started! (Read 17240 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

64 kbit/s Test has started!

Reply #1 –
FYI, I have plans to implement a cross-platform (mostly Linux) version of ABC/HR, using wxWindows for GUI, and libsndfile+libao for wave input and output. Unless I encounter unsurmountable problems, a semi-usable version (I have no ambition to implement the ABX dialog at first, because we don't need it for this test ) should be ready by next weekend. If any of you are interested in this, motivate me by letting me know. If any of you have already started something like this, please let me know and save me the work.

Cheers,

Carsten Haese.

64 kbit/s Test has started!

Reply #2 –
Quote
Originally posted by c_haese
FYI, I have plans to implement a cross-platform (mostly Linux) version of ABC/HR, using wxWindows for GUI, and libsndfile+libao for wave input and output.

Sounds like a very good idea - wxWindows seems to be one of the most cross platform GUI toolkits around (although I've seen comments saying that wxWindows apps are quite slow to start up).

Count me interested

64 kbit/s Test has started!

Reply #3 –
Quote
Originally posted by c_haese
FYI, I have plans to implement a cross-platform (mostly Linux) version of ABC/HR, using wxWindows for GUI, and libsndfile+libao for wave input and output. Unless I encounter unsurmountable problems, a semi-usable version (I have no ambition to implement the ABX dialog at first, because we don't need it for this test ) should be ready by next weekend. If any of you are interested in this, motivate me by letting me know. If any of you have already started something like this, please let me know and save me the work.


Cool!  I wouldn't assume that ABX isn't needed for the 64 kbit/s test, though.  People seem to be using it.

ff123

64 kbit/s Test has started!

Reply #4 –
'tis true, no need to ABX really.
Yet it's usable to compare the best two similar samples to eachother
There were a few times where I needed to compare a few samples together so the feature proved itself quite useful

David

64 kbit/s Test has started!

Reply #5 –
Quote
Originally posted by MTRH
'tis true, no need to ABX really.
Yet it's usable to compare the best two similar samples to eachother
There were a few times where I needed to compare a few samples together so the feature proved itself quite useful


It depends on the listener.  Some are using ABX to distinguish from the original on some samples.  Not everybody is equally sensitive to artifacting.

ff123

64 kbit/s Test has started!

Reply #6 –
IMO it's not good to give the impression that ABX isn't needed at all in this test. It may only discourage people who may not necessary hear any difference at once.
Even though it wouldn't be needed, it's good to test how the ABX works in ff123's software. I tested it myself with the piano clip (LisztBMinor) which was relatively best handled by few codecs out of the samples I've tested so far. ABX always helps me to find more problems than just simple AB comparison. Maybe it's because of the test situation..
Juha Laaksonheimo

64 kbit/s Test has started!

Reply #7 –
Hmm... I've only done a couple samples so far but so far it seems that WMA and AAC really suck at these bitrates

Anyone else feel this way?

64 kbit/s Test has started!

Reply #8 –
I am FLOORED by my results. :eek: But then again, this is not the kind of music I normally listen to...I don't have any experience with picking the best encoder for classical sources. Surprises, surprises.

1L = BachS1007BachS1007_wma.wav
2R = BachS1007BachS1007_mp3pro.wav
3R = BachS1007BachS1007_ogg64.wav
4L = BachS1007BachS1007_oggq0.wav
5L = BachS1007BachS1007_aac.wav

---------------------------------------
1L File: BachS1007BachS1007_wma.wav
1L Rating: 4.6
1L Comment: Just a small introduction of noise. Very close to the original. Could not ABX.
---------------------------------------
2R File: BachS1007BachS1007_mp3pro.wav
2R Rating: 4.2
2R Comment: Slight to moderate stereo collapse, lacks hall ambiance, but it's not too different from the original.
---------------------------------------
3R File: BachS1007BachS1007_ogg64.wav
3R Rating: 2.1
3R Comment: Echo-ey artifacts, cello sounds a little harsh and unnatural, moderate stereo collapse
---------------------------------------
4L File: BachS1007BachS1007_oggq0.wav
4L Rating: 1.5
4L Comment: Watery, gargling noises, sounds like listening through a tunnel, moderate to strong stereo collapse, worst of the bunch
---------------------------------------
5L File: BachS1007BachS1007_aac.wav
5L Rating: 2.8
5L Comment: Echo-ey artifacts, though they are better controlled than #3 and #4, slight stereo collapse
---------------------------------------

ABX Results:
Original vs BachS1007BachS1007_wma.wav
    9 out of 16, pval  0.402
Original vs BachS1007BachS1007_mp3pro.wav
    15 out of 20, pval  0.021
Original vs BachS1007BachS1007_ogg64.wav
    8 out of 8, pval  0.004
Original vs BachS1007BachS1007_oggq0.wav
    8 out of 8, pval  0.004
Original vs BachS1007BachS1007_aac.wav
    8 out of 8, pval  0.004

64 kbit/s Test has started!

Reply #9 –
Mith,

Thanks for your results.  Can you email future ones to me?  On the last (128 kbit/s) test, I missed one result which had been PM'd to me on the Winamp forum.

ff123

64 kbit/s Test has started!

Reply #10 –
Hmm...that's truly surprising. Yeah, there have been declinations in the past in the views of different listeners, but such totally contrary critiscism seems weird to me. To one WMA is almost perfect and to the other sucks bad ??? That goes beyond the typical (and usual) dealing with the explanation "different ears are sensitive to different kind of artifacts".
 

64 kbit/s Test has started!

Reply #11 –
Quote
Originally posted by bluewer than blue
Hmm...that's truly surprising. Yeah, there have been declinations in the past in the views of different listeners, but such totally contrary critiscism seems weird to me. To one WMA is almost perfect and to the other sucks bad ??? That goes beyond the typical (and usual) dealing with the explanation "different ears are sensitive to different kind of artifacts".
I think, it's more a question of samples than of ears. I've done a few test samples already, and I ranked WMA first and last, depending on the file. WMA seems to be quite good at encoding classical music (I always had that impression).

More will be revealed to us, when this test is finished.

64 kbit/s Test has started!

Reply #12 –
Quote
Originally posted by Volcano
When's the deadline for submission of results? I'll download the large file in increments, I just need a little time


I don't think there's any big hurry.  I'd like to get at least 10 listeners per sample, and I'm averaging perhaps 5 at the moment.

ff123

64 kbit/s Test has started!

Reply #13 –
the q0 ogg file size is like a 48 kbps file (looking at his file size)

but the q64 one is 63kbps , so it's why the b64 is better than the q0

it's strange (it's because of the vbr routines of ogg vorbis i guess)

64 kbit/s Test has started!

Reply #14 –
Quote
Originally posted by ff123
I don't think there's any big hurry.  I'd like to get at least 10 listeners per sample, and I'm averaging perhaps 5 at the moment.

To hell with it, I'm downloading the complete file now.

(I hadn't refreshed the thread, so I didn't notice anyone had replied already when I deleted my post...)

64 kbit/s Test has started!

Reply #15 –
Quote
Originally posted by mphilamp
the q0 ogg file size is like a 48 kbps file (looking at his file size)

That's why I thought it's best to find a q setting that yields around 64 kbps. But Most people, including Monty, thought it wasn't necessary.

Without doing that, we don't know how ogg will perform with this test sample when it takes the full advantage of 64 kbps.

As it is, we have two sub-optimum ogg samples in the contest -- for this particular test sample. One is similar in bitrate with others, but not in ogg's best condition. The other is using the right mode, but uses only 48 kbps.

It's too easy for a regular viewer to look at the result and come to the conclusion that "ogg sucks in classical music" or "ogg sucks at 64 kbps when used in classical music". Why? Because everyone says ogg -q0 is about 64 kbps.

It would be better if the actual average bitrate of each file is reported along with test scores. But so far no one does that. The only way to find that out is to go to ff123's page and dl the files. But how many casual readers would do that?
Quote
but the q64 one is 63kbps , so it's why the b64 is better than the q0

it's strange (it's because of the vbr routines of ogg vorbis i guess)

It's not that strange. Monty explained it in this thread.
tw101

64 kbit/s Test has started!

Reply #16 –
I submitted 3 results so far, and will do more. All of them were surprising!

Thanks for holding the test ff123.

--
GCP

64 kbit/s Test has started!

Reply #17 –
Quote
Originally posted by tw101
That's why I thought it's best to find a q setting that yields around 64 kbps. But Most people, including Monty, thought it wasn't necessary. 

Without doing that, we don't know how ogg will perform with this test sample when it takes the full advantage of 64 kbps. 
Then again, some ogg files use more than 64 kbps at -q 0. But I'm sure ff123 will publish the average bitrates with the (final) results. (Knowing them before the test doesn't really make sense anyway)

Quote
It's too easy for a regular viewer to look at the result and come to the conclusion that "ogg sucks in classical music" or "ogg sucks at 64 kbps when used in classical music". Why? Because everyone says ogg -q0 is about 64 kbps.
This is true. -q 0 produces files with less than 64 average for classical music. But I hope the "-b 64 --managed" should compensate for that.

64 kbit/s Test has started!

Reply #18 –
Quote
Originally posted by tw101
As it is, we have two sub-optimum ogg samples in the contest -- for this particular test sample. One is similar in bitrate with others, but not in ogg's best condition. The other is using the right mode, but uses only 48 kbps.
Then again -q0 can use clearly higher bitrate than 64kbps. Isn't it unfair towards the other competitors?
In my opinion Vorbis is favored enough already when it participates with 2 settings to this test. If after this, the -q values would be even tweaked in favor of vorbis, it would be too obivously favoring Vorbis IMO.
-q0 is a quality setting. If it gives relatively worse quality results with classical music compared to others, then it's an issue of Vorbis' psychoacoustics.
Juha Laaksonheimo

64 kbit/s Test has started!

Reply #19 –
IMHO, We can surely speculate that Vorbis is mostly developed and improved for VBR ( -q )  modes, therefore consider that if WMA tends to align to 64 kbit and its commonly well made for CBR mode I think we should consider Vorbis -q with = bitrate
"Taking a jazz approach and concentrating on live playing, I wanted to use several different rhythm sections and vintage instruments and amps to create a timeless sound that's geared more around musicality and vibe than sonic perfection. The key was to write with specific rhythm sections in mind, yet leave open spaces for soloing." Lee Ritenour

64 kbit/s Test has started!

Reply #20 –
I have a request: can you guys try to find inexperienced listeners and ask them to do the test? I asked my girlfriend to do this test (she'd never done something like this before). After showing how the program and the ABX test worked I let her play with it till she was done. The results were quite surprising, and I'm interested if other inexperienced listeners get comparable results.

So, get your girlfriend/wife/sister/little brother/uncle/aunt to take the test! The less they know about listenings tests/audio, the better!

Don't help them, make sure they understand how the ABX test works, and remember that 'bad' results are good data points anyway.

(and don't forget to indicate it when sending in the results)

--
GCP

64 kbit/s Test has started!

Reply #21 –
Quote
Originally posted by JohnV
IMO it's not good to give the impression that ABX isn't needed at all in this test. It may only discourage people who may not necessary hear any difference at once. 
Even though it wouldn't be needed, it's good to test how the ABX works in ff123's software. I tested it myself with the piano clip (LisztBMinor) which was relatively best handled by few codecs out of the samples I've tested so far. ABX always helps me to find more problems than just simple AB comparison. Maybe it's because of the test situation..


I agree, I shouldn't have given the impression that ABX is not necessary at all for this test. What I was trying to say was that my first priority is to get the main window in a Linux version, so that Linux users like me can participate at all. Adding the ABX dialog is not too much of a problem, but it does take time, and the test is already going on. I'm just hoping it'll go on for a little while longer so that maybe some participants will actually get a chance to use the Linux version (once it sees the light of day ) for this test.

Cheers,

Carsten Haese.

64 kbit/s Test has started!

Reply #22 –
so .... up

64 kbit/s Test has started!

Reply #23 –
Quote
Originally posted by JohnV
Then again -q0 can use clearly higher bitrate than 64kbps. Isn't it unfair towards the other competitors?

Of course. If a test sample shows ogg -q0 beats wma 64 kbps hands down, but it turns out ogg -q0 averages 80 kbps for that file, I don't think the comparison means anything. (Just imagine when you tell somebody "ogg is really neat, because a ogg file at 80 kbps beats wma at 64 kbps".)

The same is true when comparing ogg -q mode and -b --managed mode. It is only meaningful when we adjust the -q value so that the encoded file is about the same size as the -b --managed bitrates -- e.g., 64 kbps.

If we test only -q0 vs. -b --managed 64 k, and the test samples give us:

sample 1:

-q 0  (42 kbps) < -b --managed 64 kbps

sample 2:

-q 0 (80 kbps) > -b --managed 64 kbps

What does that tell us? Tests of this sort is meaningful only if there's surprise result, like

sample 1:

-q 0  (42 kbps) > -b --managed 64 kbps

or

sample 2:

-q 0 (80 kbps) < -b --managed 64 kbps

For in the 1st case, we can say "Ah, ha, so -q mode is really good, for it beats the -b --managed mode even with 22 less kbps."

Likewise, if the 2nd case happens, we can say "Ah, ha, so -q 0 is really not good, for it got beaten by -b --managed even with 16 more kbps."

But if the result is the "normal" one (higher bitrates win), then we still don't know which mode is better.

Note: I know it's not enough to draw any generalized conclusion with only one test sample, or only one tester. What I mean is that, with certain test designs, you can't get meaningful conclusion even with repeated  and consistent results.

Quote
In my opinion Vorbis is favored enough already when it participates with 2 settings to this test.

I'm not sure I understand this. Why does having two test samples have anything to do with being favored? (Why should we favor ogg?) There're two test samples only because it has two modes: true VBR and managed bitrate.

If it's a highbitrate contest and LAME is in there, I would think at least two test samples are needed, too (alt-preset VBR and alt-preset CBR), or maybe even 3 (+ ABR).
Quote
If after this, the -q values would be even tweaked in favor of vorbis, it would be too obivously favoring Vorbis IMO.

I don't understand this, either. It's about leveling the playing field. Who's favored? If -q 0 gives us a test sample of 80 kbps, it's still not fair and should be adjusted to get about 64 kbps (to go against wma and mp3pro at 64 kbps). Is wma or mp3pro favored in that case? Certainly not, it's still about leveling playing field.

Quote
-q0 is a quality setting. If it gives relatively worse quality results with classical music compared to others, then it's an issue of Vorbis' psychoacoustics.

But nobody has to use a particular -q setting. No matter what kind of music I want to encode, what I want to know is "at around 64kbps (or 128, or 192...), who will give me the best result."

Again, using the example above about -q mode vs. -b --managed mode, but let's change oqq -b --managed to wma 64 kbps.

If today we get the following results:

ogg -q 0, when the encoded file is around 80 kbps, beats wma 64 kbps.

ogg -q 0, when the encoded file is around 42 kbps, looses to wma 64 kbps.

Do you think it has much informational value?
tw101

64 kbit/s Test has started!

Reply #24 –
Quote
Originally posted by tw101

If today we get the following results:

ogg -q 0, when the encoded file is around 80 kbps, beats wma 64 kbps.

ogg -q 0, when the encoded file is around 42 kbps, looses to wma 64 kbps.

Do you think it has much informational value?


This might be a plausible expected result.  In fact, out of the dozen samples, I would not be surprised at all if the results came out that way.  The ideal would be to test enough samples to show that on average -q 0 can be said to be better (or worse) than wma8 at 64 kbit/s.

But I still don't think it's fair to tweak a VBR codec to match the bitrate of a CBR codec using short samples.  The only fair way to tweak it is to use a lot of diverse albums to match up the overall bitrates.

ff123