wah, the same for me (and i already wondered about the results)



you can have a look at the coments on the samples in the aac listening test (pretty much the same as now) here
there you can find hints on what you should listen in each sample

first time i tried it i couldnt also hear many differences, but now i listened to 5 samples and i could hear differences in every sample (voted 5 only twice till now)

thanks but is this the same cvs version rjamorim used?

he used "Ogg Vorbis 1.0 post-CVS". what does "post-cvs" mean?


edit:
according to the filesize case's oggenc produces exactly the same output as the original oggenc with -q 4!

->
i would recommend changing the *.bat files by hand if "," is used as a comma in your country!

I suppose so, except library version string isn't modified like john33 did.


I think it should be "post 1.0 CVS".

some one here who has access to the presentation page to update it with this (not unimportant) info?

Should this test to be called Medium quality Extension test? If all samples are'nt close enough at 128kbps. Yes i've read all comments, but still this is not the right way i think. If MP3, AAC and WMA are forced to be near as possible to 128kbps and ogg and mpc can be what ever they want to bee...

And what would be the right way in your opinion? Forcing vbr samples separately to 128kbps? I hope you understand that in that case you'd have 12 different settings for one vbr codec, so what would the results tell then about the codec's quality at certain setting - nothing...
Simply put: VBR is used with average bitrate basis here, that's the whole idea and the only sain thing to do. I'm sorry if you don't understand that, but that is the most fair way to test vbr.
Lame MP3 is ABR because it performs better than Lame MP3 vbr near the 128kbps. Quicktime is CBR because there's no VBR/ABR setting for it.

Bummer! If only someone had spotted this before the articles were submitted to those large sites...

I guess I'm lucky that at least on one of the two samples I've submitted results for so far, the Vorbis encoded version was totally transparent to me even though it was only encoded at -q 4, so I won't have to re-test that one.

So far, I am very surprised by the test... on the samples I've tested so far, the differences were very hard to detect and ABX most of the time (the two MP3 competitors - OK, Blade isn't really a "competitor" - aside). Nice to see how the encoders have improved over time.



I can still vaguely remember your last 128kbps test more than 1 1/2 years ago. Yeah, that was indeed a *lot* easier.

I suggest that Roberto verifies with each person who submits results that their ogg ratings weren't affected by the comma bug (only a potential problem if they use the regional comma setting *and* happened to rate ogg files worse than the reference).

Meanwhile, I guess I'll append a note to my Usenet postings.

ff123

I'm a bit afraid to see that everybody could check immediatly after a test their results. Names are now explicit. It's easy to 'correct' some results, in order why not to give to our favorite format the best place... It's stupid I know, but there is a disease here called zealotry, and we can fear some cheating...

It's time for encrypted result log file...

i would say that this would be the best way:

upload as fast as possible a new abc-hr_bin.zip package instead (!) of the old one

the following changes should be done in the package:

1) exchange the old oggenc.exe with a version that isnt "," or "." sensitive like the one from case (although i am not sure if his version produces really 4.25 output in every case)
2) change the sampleXX.bat files in a way that the .ogg.wav files created with this updated package have a different name than the original ones (for example append a "_2" to the ogg filename so that rjamorim knows that there is the possibility that this file (and the rating) is flawed
-> than rjamorim can verify with the person who sent this package if the comma bug was present (so he doesnt have to check it with every person)
3) it will be than also possible to give the files "anonymous" names (like in the aac test) like guruboolez suggested


but the most important part will be that this happens as fast as possbile!!!
were is rjamorim or does anbody else has access to upload an updated package?

Okay, if required to, I can redo three tests I've sent. Got the bug.

The batch file would need to be unreadable and have no output.
Which isn't possible, of course.

/EDIT\
Oh well, it is... with a tool called Bat2Exe.
Now, how to suppress the output of all encoders?

Of course, you'll need to use anonymous names for all flacs and output files.
\EDIT/


You might want to write a dedicated program for this.

Encryption would be nice, but it would require modified version of ABC/HR.
And lots of work!

Unfortunately, I don't really have time anymore to modify ABC/HR. Although the source is open, in practice, only the original authors seem to dare to modify such programs.

A program to modify the batch files should not be hard to gin up using something like python. Let's see what Roberto wants to do.

ff123

Reread my post, please.
Modifying the batches won't help - how Roberto will know which file is which?

It has to be set earlier.
Additionally, encoding/decoding order should be 'randomized' (in the batch files, of course).

If I am reading this, some encoders can output upto 190Kbps? based on quality settings. In a fair world each sample would be compressed by hand until the final output fitted just under the filesize for a 128Kbps CBR file.

Jesus.. How many times this has to be explained in the same thread???
http://www.hydrogenaudio.org/forums/index....ndpost&p=117956
http://www.hydrogenaudio.org/forums/index....ndpost&p=117779
http://www.hydrogenaudio.org/forums/index....ndpost&p=117895

I'd understand if this was at slashdot.org..

Next same kind of message goes straight to off-topic.

Here is the sample batch - you need to make an executable from it!
/EDIT\
Okay, final, working version!
\EDIT/

Well Jesus...I have already read those and didn't agree with the messages, so being as this is a free and open message board I used my democratic right to post what I wanted. Or is it because you a Moderator have replied to it, then that is gospel and no one else can reply?

Pretty much yeah, because it has been explained many times now, and frankly this is an issue which should be pretty much clear that you are getting no interpretable results about vbr codec's quality with the method you are suggesting..

So for example regarding Vorbis (just example, not real):
sample 1: -q2
sample 2: -q6
sample 3: -q3
sample 4: -q2.5
sample 5: -q3.3
sample 6: -q1.8
sample 7: -q4
sample 8: -q4.2
sample 9: -q5
sample10: -q6.5
sample11: -q2.3
sample12: -q3.5

Now, tell me what can you say overall about the average 128kbps quality (certain quality setting) based on results from those? Or actually any useful result.. I'd like to know...

And posting here is not any "democratic right". It's a priviledge which is possible to be taken away.

In a fair world, you had to cut each track of each album in small 5 seconds parts, encode them manually with proper setting in order to reach x kbps, then merge all small parts into a complete track again. By doing that you will obtain VBR file, with each difficult or easy part encoded at x kbps. In other words, CBR with bit reservoir...

If some people doesn't like the kbps amplitude phenomenon, they just had to avoid VBR, and chose CBR encodings. Quality is worse, but bitrate will be constant... You can't expect choosing VBR for its quality and fighting against its own logical : more bitrate when needed. Is it so hard to understand ?

so perhaps we will see mpc as winner (or on a top position) just because it's smart enough to "cheat" the comparison by using as much bitrate as possible to reach great quality (and everybody just wanted to add it for "academical reasons")

"hey, i always had the opinion that mpc was the best codec @128kbps and the test also came to the same conlcusion" (although mpc hardly ever really used 128kbps)



in a fair world the whole song would have been encoded at an average bitrate around 128 and then the 30sec problematic part would have been cut out...

Read this:
http://www.hydrogenaudio.org/forums/index....showtopic=11134

The reason why mpc and vorbis give high bitrates is because some of the samples are short and hard to encode in order to hear some difference between the codec qualities. I don't understand why vbr codecs should be punished, just because the test samples are short and have been chosen to be such that listeners may have easier job to distinguish differences..

The encoder doesn't cheat.. it's encoding algorithms are such that when more bits are needed, more bits will be used. It is pointless to discriminate against one encoder by cutting out "problematic" parts of a song. This 128kbs test is for the very purpose of seeing which encoder can produce the best sound while using a relatively (depending on who) low bitrate. If it takes --quality 4 or whatever is being used to attain ~128 (ABOUT 128kbs), so be it.

Encoding a whole album in order to reach 128 kbps, and let the same setting for a microscopic sample, is something fair. This is what Roberto asked us to do, by comparing mpc, ogg and wma at different settings on our favorite discs. Musepack --radio is close to 128 kbps, and for oggenc it's -b 4,25.

If you're familiar with your encoder, it's possible to anticipate some of its behaviour, and to chose different setting according to the disc you had to encode.
If I had a piano disc to encode with mpc at ~130 kbps, I will set the encoder at --quality 4.5 without trying --quality 4. For an harpsichord disc, I won't go beyond --quality 3. For orchestral, --quality 4 have the most chances to be close to the targeted bitrate...

Still have my priviledge



The reason VBR exists is that a codec can be advanced enough to lower its bitrate and up its bit rate, but for this to be a fair test - especially with different sample types - for all we know on the harpsicord: ogg will go to an average of 200Kbps, whilst WMA goes to 128Kbps. Now you could say, tough luck WMA for not matching Ogg when it goes to 200Kbps, but you could also say that WMA is better programmed because it stays within its quality range and does not vary wildly. *** these codecs and numbers are totally made up ***

I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.

My batch file is finished!
Now I'm going to check if Bat2Exe works correctly with it.

/EDIT\
It does! Of course there are still ways to detect which file is which, of course...
But these aren't that easy.
\EDIT/

In testing different vbr codecs nothing is ideal, but the current method is hell of lot better than tweaking short hard-to-encode samples forcefully to 128kbps and then try to make some statistical conclusions about that vbr codec's quality which should potray in someway the average quality relative to other codecs.

Nice idea, to test Vorbis ABR, completely untweaked (I suppose), completely unused (I'm quite sure), slower and probably worse quality...

Or how doing a listening quality test by forcing the best encoders to be worse than they are...


EDIT :


PS : it's mabe time to split this thread. I'm a bit sad to see whole Roberto's work contested here. It's his test, he worked for it - we worked too for him. It's isn't time anymore to contest the methodology. There were a PRE-TEST DISCUSSION for it.

Sorry, I thought you were talking about a batch file to fix the comma bug. As far as making the encoding process silent, I wonder if it's worthwhile to insert that in at this point. What's the point of randomizing the encoding process?

ff123

The 128Kbps title of this test is slightly missleading, we will stick with the Harpsichord...for sake of argument if Ogg went to a final Average of 200Kbps whilst all the other Codecs were 130Kbps, it would be fair to say that Ogg would 'win' that one, but what are you telling people? if you had 10 samples to be tested then oggs result would IMHO be 1/10th higher than it should, unless the whole world listens to harpsichord music, which I don't think it does


Edit: Indeed, please split the thread, I have the upmost respect for Roberto

one time the vbr codecs are discriminated and the other time the abr codecs (the old thing with apples and oranges)...

but i wouldnt know what to think if mpc will be voted on 1-3 place

let's wait for the results

I suppose the reason ABR was not used where applicable is because not all codecs do ABR. Hence, VBR was used since all the formats tested are able to encode in it.

Discussions like this are always necessary for the interpretation.

I already finished three samples, with quite suprising results for me, for example that a encoder that is inperceptable for me at one file completly sucks at another.
But I have a problem now: The files at audiocoding.com are not downloadable.
They are not down, but server does not respond.
Because I have unlimited traffic and 10MB free, very fast, space I uploaded the first file. If you don´t have it yet and audiocoding doesn´t work for you, too, download here:
Sample01.zip (41_30sec) (6,93MB)
It would be great if someone else could upload sample02, 03 and 04.
Also Quicktime AAC? I thought it´s CBR only.

I had some problems too with the first four packages last hours. My download software can now DL them. Problem seems to be corrected.

@guru, JohnV and others

I appreciate your response, but please don't just curse and link to threads I have already read multiple times. It achieves nothing.

I completely understand the logic behind the choice of codecs settings etc, and I completely understand the use of quality settings in the real world, etc. You should credit the average member of HA with a bit more intelligence.

I also feel that Roberto has done an excellent job organising a complex public test like this one. My query about bitrate was in no way a reflection on his outstanding efforts and attention to detail.

But at the same time, here we are promoting this as a public, open test, slashdotting here, putting other notices up there, but when the results are interpreted, to many readers the results will be bogus because of the huge descrepancy in the bit rates for a test titles "128k Extension Test". 197k is not 128k in any one's language.

OK, so we should proceed and see how the results turn out, but I can just see that if mpc gets up as the best encoder for example, the vorbis zealots will have an absolute field day attacking the validity of the results, and vice versa.

There are some ways that this concern can be alleviated. First, place a statement clearly explaining that the codecs are VBR, and for some samples, some codecs will exceed 128 k by some margin. Make sure that it is publicly known that the samples are not all the same size. Also, as part of the test results, show the average bitrate for each codec across all the samples, so that at least when people read through the results, they can conclude that a certain encoder may be better for their own needs, but it uses additional bitrate as required, etc. so at least the bitrate bloat for some of these encoders at this setting is not hidden.

Den.

In the ideal, here is how I think a VBR test suite would be constructed:

There would be a mix of difficult and easy samples, and one encoding setting is used for the entire test suite such that when the average bitrate was calculated for all the samples, it worked out to 128 kbit/s. Of course, this would be after verifying that multiple album encodes yielded 128 kbit/s on average too, also using that same encoding setting. Also, we would do the "slicing" the sample out after encoding an entire song/album routine. I think this would get rid of some of the criticism we're seeing here.

Real world:

Probably to achieve the average bitrate of 128 kbit/s, we would have to encode many more easy samples than difficult samples, and the test suite would grow very large while not being significantly better at telling us which codec is better (the idea is that difficult samples are best for discriminating quality differences).

The assumption being made is that the relative codec ranking stays the same with the easy samples as it does with the difficult samples, or at least that all the codecs are transparent if the samples are easy to encode.

But I think that making this assumption is far more reasonable than limiting the VBR codecs to 128 kbit/s by changing the encoding settings per sample. That's not how the codecs are used in real life, and it's nullifying the purpose of using VBR.


Regarding slicing:

The problem with slicing out samples after encoding is that we'd have to assemble another test suite, where the entire songs are available. This is actually possible to do, though, so it's just a problem of practicality for this test.

ff123

P.S. I agree that this whole discussion must be made very clear in the results page, because if there is argument and criticism here, it will be ten times worse on other forums.

Well, I was under the impression that using alphas wouldn't be very safe.



The joys of VBR...



Thanks for the idea. I'll do that now



Blah. And since when this is HydrogenAudio's test?

Sorry about being anal here, but I'd rather not have my tests directly connected to any community.



I don't deserve this >_<



Right. Quality shouldn't be much worse either.



Thanks a lot, Case.

I'll add it to the abc-hr_bin package and reupload it. No point in asking people to retake the test though, IMO.

So, it now just accepts either dot or comma as separator?

True. Fixed that on the page.



Just FYI, people with access to the rarewares account are John33, Dibrom, Ruairi (rc55) and me. You can PM one of them if you need something to be corrected urgently while I'm away.


About the oggenc comma issue:

People, it's only 0.25 (or 0,25 ) difference in the encoded file. I can't be arsed to believe 2-3kbps will make a huge difference.

And as JohnV pointed out, I'm using nominal 128 anyway.


The updated abc-hr_bin package is being uploaded already though.

I spent the whole day with my GF. It was about time I gave her some attention, didn't do that since I returned from my trip.



One of them is located at Audiocoding. Menno has access to that account.

The other one is being hosted by Paul Harris (verloren). Also, John33 has access to that account.

*cough* I happen to have access too..

This fix isn't as advanced as the one I gave to oggenc author when the issue was originally found some months ago, I now just removed locale changing for the encoder and it only accepts dot.

I just got started. Poor blade, it's like a grizzly bear trying to do ballet. :x I hope nobody comes to my apartment and sees me locked in the bathroom with extension cords running under the door. Unfortunately I can still hear the cheetah in my Windoze box seeking, even through the wall. >_<

I agree an explanation about the VBR up front (before results are compiled) will be key to appeasing the skeptics. All the codecs used settings that produce an average of about 128kbps over a large number of albums (and you've got the data to back that up.) VBR codecs can't be punished for spending those bits where they'll do the most good. Unfortunately, with the festering hole Slashdot has become, there will still be plenty of people who won't accept the results no matter how carefully the test is conducted. The recent thread on the AAC test was almost entirely flaming by people who either didn't read or didn't understand the test.

@JohnV: Sorry, I forgot you got root access.



Audiocoding is actually sourceforge. Their servers aren't very reliable, but I would be a bastard if I complained about something I get for free. Please, just try again later.



Ok, here's my take on the "I think your test is wrong" I expect some critics to come up with. As you guys know, I'm a very blunt person

I did the test. To the best of my knowledge and capacity. If you dislike it, either run your own test or f*ck off.

Of course, I accept constructive criticism. But I will really ask people that come to me saying my test sucks without providing a better way to do it to go fuck themselves. I actually learned to be blunt from my experiences with the AAC test. :B



Almost finished

I was about to post, saying thank you for responding to my initial concern a little more calmly than some others...



Excellent. I think that should do it. Time to get on with the testing. B)

Thanks again Roberto.

Den.

I added this text to the presentation page:

"I noticed that, on some files, bitrate goes very high, nearing 200kbps sometimes. Is that right?

Yes, and that's the beauty of VBR encoding. It will simply ignore bitrate limitations (whenever possible), using as much bits as needed to encode a problematic sample.
Although that raises issues of fairness, it's the best way to compare modern codecs that shine the most in VBR mode, like Musepack and Vorbis. Trying to force a VBR setting to match as best as possible a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that an user just stick to a quality setting, not caring much about a specific bitrate"

Any remarks?

Well, that wasn't targeted at you. I was just giving people a sample of what awaits clueless critics.



I thank you guys for all the help




edit: Just uploaded new abc-hr_bin.zip packages using Case's oggenc (thanks!), all should work as planned now. :B

Sweet. Explains the situation without apologising for it. B)

I know it's early days yet, but some comments about the same, and reported bitrates would be useful when the test results are published too, but I'm sure you're already onto that!





Den.

I would add that the quality settings chosen for the VBR codecs were chosen because they average out to about 128kbps over a number of encoded albums. And to pick a nit, I might say "fairer in one sense" instead of just "fairer' and would add that it would be unfair to tie the hands of VBR codecs and punish them for being smart about where to spend what turns out to be the same number of bits over the long run.

BTW: Let me also say thank you for running this test. You rule.

Added your comments (actually, did some copy-pasting hope you don't mind).
Thanks a lot for your remarks, and your kind words.

About this VBR issue: a similar thing happened with the 64 kbps test. Some people said that that was not the way of testing codecs, but it was just because they didn't understand the issue.

Test designers are not responsible for people being dumb. You can explain why VBR codecs have been used this way. If some people still don't agree with explanation, it's their problem. The only "but" I can accept is that for the type of music someone listens, the average VBR bitrate is higher than 128 kbps. People complaining should provide their VBR average bitrate to support their complaints.
Even when that can happen in some particular cases, it's a particular problem impossible to overcome with a general test, which is still much better than nothing.