Just a note for people who have just installed ubuntu (like me). The default gnu java is compatible with sun java jre 1.4, but abchr-java needs jre 1.5 or later (also called version 5). I decided to go for broke and install sun java version 6:

sudo apt-get install sun-java6-jre sun-java6-plugin

and then change the configuration to use this:

sudo update-alternatives --config java

and finally from within the abchr application I changed the playback device to be the Java Sound Audio Engine.

It seems to work fine!

Geez, I could party that I finally got rid of Ubuntu Live CD and others install it voluntarily...

OT warning:

Hehe, actually for all its supposed user-friendliness, I did have to tweak it quite a bit. I didn't like the fonts, which seemed big and fuzzy, so I found out how to install/configure the Microsoft ones. Samba sharing (using cifs and not the deprecated smbfs) took me quite a while to get right, so that I could get my XP Share to mount/unmount as it came online or went offline. Using cifs without having to explicitly use the IP address of the share took more tweaking (installing winbind), and did I mention Feisty has some sort of shutdown bug when using cifs? Had to fix that too. I wouldn't have had to use samba to browse my kids' XP laptop in the first place if the GnomeFS was actually able to stream files properly over a Microsoft network.

Getting my wireless network to connect on bootup was kind of a pain. No, it's not obvious how to do that and why, but it wasn't as easy as just playing with the Network Manager settings inside a GUI of some sort.

MPlayer configuration took a lot of tweaking (it doesn't even use X11 by default).

Wanted to write to ntfs volumes -- more fiddling.

And of course the java thing.

Good things: my streaming video media plays faster, but maybe that's only because I have not installed a software firewall; I could have uninstalled zonealarm on my Win2k boot and I might have seen the same speed improvement. There's no resident anti-virus program sucking up resources. Openoffice works well enough, and it does most things (I don't play games) I need to do as well as my Win2k installation does.

/OT warning

Oh, and BTW, I have 4 more samples to test, and I'll email them off to you.

Yes, PowerPC. That was my intention originally. But given that we can't even redistribute the binaries (as I've read in the license), I don't think that I'm going to package them. It is a pity for those of us with multiple platforms.



I do and I'm talking about this with Mike. I already told him by private e-mail what my plans are here with the layout of the repository and this would make mirroring the repository much easier.

Anyway, back to the topic of the listening test, this will be my first time participating on the test. I'm having a really hard time (as others) doing the listenings.

I think that it is easier to recognize problems with genres of music that I'm familiar with (like Metal and Harpsichord), but with others, it is quite hard even to detect a low pass (say, with Techno).

One thing that would be nice to catalog in the Knowledge Base of HA would be the types of artifacts that usually arise with different encoders/formats.

It is really strange here that I have Blind Guardian's "Nightfall in Middle Earth" encoded with Apple's latest version of Quicktime at 128CBR and I can listen to artifacts that are here, but that I'm having a hard time in participating of this test...

I think that I may have a "killer" sample to contribute...


Regards, Rogério.

Well, I have only had the chance (you, know, that thing called "Real Life" actually takes our time) to try 3 samples. With much difficulty, I should add.

Regards, Rogério.

I'm now working on sample 11, having worked backwards from 18 (to even out the number of results for the different samples ) . Another 10 to go, in other words.

Thanks for all submissions so far and good luck to all of you who are still testing.

So far I've noticed: lowpass (of course), mushy transients, wheezy voices, narrowing of stereo, fluttering or a type of low-frequency noise in which the music doesn't sound "clean." In one case, I had a "wtf!?" experience, in which an entire instrument virtually disappeared. That was kind of cool; I've never heard that one before. I should add some of these to my artifact training page, but I'm so lazy these days. The HA knowledge base would be good too.

ff123

I thought that was cancelled for poor ratings.

Hey, I just noticed this test is running!
Aren't we suppose to receive a mail from Roberto for those tests? I thought he had a ML dedicated to this.

Yeah, Roberto is so unreliable these days...

Extending the test until August 12th.

Click, but don't hurt me!

Interesting, really interesting. So far I haven't been aware of something like this. Once again I learned something new.

Tried a few sample....

GUYS, WOULD YOU PLEASE STOP IMPROVING YOUR ENCODERS (as LAME now seems quite lame)

I'll throw in an LOL @ that.



Great! That should give me enough time to do all the samples.

Edit: This strategy of setting a short date and then extending it is probably a good one, because if you had said the 12 of August from the very start, people (me) might have put it off and not finished in time all the same.

Did somebody succeed finding (abx-ing) all encoded samples except maybe high anchor?

I've found basically all compressed samples so far (samples 9-18) including high anchor. Possibly the odd 5.0, I can't remember now.

But I often find, when ABXing, that I can get the first 3 or 4 attempts right, but then suddenly, I can't hear the difference any more... then I have to rest my ears for little while and continue... In other words: not really artifacts you would notice during normal listening!

BTW: When should you set a score between 4.0 (perceptible) and 5.0 (imperceptible)? How do you define something that is between perceptible and imperceptible? I'm ending up giving nearly all my samples 4.0. Some slightly lower, and the odd just above 4.0 - when I have to twist my brain round and swivel my eyes into my head to hear any difference.

I'm also spending a lot of time trying to spot. There is important difference between 48 and 64 kbit test.

I'm also noticing when I ABX in test mode and I suddenly lack feedback, I become unsure at first, and then just say "oh well" and click through the trials so fast that I have little time to consciously decide whether I heard the artifact or not. I'm quite surprised afterwards when I find I scored something like 14/15, which is what usually happens.

I heard before that very fast decisions can be better than those you had time to think about, maybe this is a similar effect

Well, sometimes I can do 10 or so abx choices quickly in a row, but it has to be a very clear artifact, or at least I have to be able to define it very clearly in my head. (I.e. "the cymbal on beat 3/8, to the left, sounds mushier than it should".)

Gah! Blame my teacher that wants me to code a huge Java program by yesterday.

Sorry about that. I wonder if it's still worth announcing?



FYAD

Uhm... I'd say you could call it ... perceptible?

When ABXing, I use that range to indicate how difficult is for me to spot the problem. If i have to concentrate a lot, and might even fail at times, it's almost imperceptible, so 4.8, 4.9....
You get it?

Perceptible (4.0) means that i hear the difference, but if i didn't have the original to compare, i wouldn't think that's an artifact, or wouldn't notice it at all. (say, some lowpass filter that doens't affect the sound quality, an artifact that makes the noise more noisy without affecting the tonal parts...)

Well, yeah. But my kind of anal point is that if you can tell the difference, then you can't set a score higher than 4.0. I know, I know, I perhaps shouldn't take the scale so literally.

Well, you said yourself that sometimes you can hear a difference and then suddenly it's gone after you ABXed a few times. So, it's between perceptible and imperceptible.

Ok, I'll buy that.

Off-topic,

Since the original thread was closed, and there is still a sense of doubt regarding the legality of disclosing the results of which the MS codec WMA 10 pro would be included, I would suggest that after the test is finished, the results be disclosed, but do not include the rating of WMA 10 pro yet.

If according to this test, WMA 10 pro be either superior/inferior to any of the other codecs, then in the next discussions, the moderator and other members may state that:

"IMHO, codec X is better/worse than WMA 10 pro"

without being thrown with TOS#8. Just don't disclose the results online (but hey, you can PM anytime! just pass the message).

It's better this way than being thrown with a subpoena or court orders or warrants of arrest...

Let the MS guys worry about the hidden result, and let us just have our final announcement of the results without WMA like saying (for example):

"HE-AAC came out No. 1. WMA 10 pro is included in the test, but we are not going to disclose its score. HE-AAC is still No. 1. Don't use WMA 10 Pro--it sucks" or if it's favourable to them, just keep a mum.

They won't stand a chance with that one! Har Har Har!

I must have missed that discussion... why would publishing the results of this listening test be illegal? Because Microsoft pwnz the internets?

Edit: I've just looked through the pre-test thread and couldn't find any such discussion... could you fill me in?

There appears to be a recording/ripping "click" artifact in "Sample02 - Bachpsichord" just after 4 seconds. While its effect is largely nullified since it appears in all samples, I am able to "cheat" in ABXing since this short burst of random noise sounds noticeably different when encoded. I might skip this sample because I'm ABXing based on a technicality. Any advice for this situation?

Well, you could zoom in on some other part of the track and ignore the click.

That's true, but, personality-wise, it's not easy for me to give up advantages when I know they exist. It's like pretending to be shocked at a movie's end when you solved it in the first act. Now that this artifact is known, we cannot be confident others aren't also ABXing based on it, consciously or unconsciously. I'll sleep on it, but I'll probably declare this one a five-way tie (by not submitting anything).

So you say that you can rate a codec based on one click? It's like saying Car A is better than Car B because of one screw that doesn't even have a "vital" function. I understand that some people focus on certain parts of the music and rate a coded depending how that specific part sounds like, but rating a codec based on something that isn't even intended to be part of the music (and wouldn't disturb your listening experience) is weird.

Edit: Bear in mind that you don't have to ABX only, but also rank each codec.



Pardon? I would also like to know where you got that information from regarding this listening test in 2007. There are no such claims that wouldn't allow me to disclose the results in the WMP 11 final package. And even if, in Germany, EULAs have no legal meaning.

Were I evaluating eighteen "pristine" cars and one of the cars had a noticeable scratch, I'd remove it from testing because the other seventeen lacked obvious physical defects, just like I am doing with this sample set.


5-way tie. Honestly, it's not far from it in any case. I had a more difficult time with that particular sample than others.

I'm with you on this one. If there is a CD-style click in one of the samples (haven't got to nr 2 myself yet) then I don't understand why it has been included...



Not so weird if you ask me. Simple human nature to pick the easy way out. The question is, why was this sample included, if it contains a ripping error? (Does it?)

I didn't notice there was an issue and the sample was available for use in listening tests. And as I said - you are supposed to rate how the audio files sound, it's not all about ABXing. Yes, you may "cheat" now by successfully ABXing and therefore making ABC/HR disable the reference sliders, but then do you rate the codecs based on a few milliseconds? On the other hand, you can test for pre-echo in this way or how encoders deal with vinyl clicks / ripping errors. Personally, I don't see this as a huge issue.

Well, I may be doing things wrong, but that's basically what I do. I find a position that I can ABX and rate the samples at this position.

If I would just listen to the whole samples and rate them overall, they would all be 5.0 except the low anchor.

Focusing on something like the first couple of seconds is OK in my opinion, but focusing on one single click to rate a sample that has several seconds - well, this is not the way it should be done. That click has only several milliseconds - even it the click was 1 second long, it would only make out 4% of the sample. And like I said, what if the sample was ripped off a vinyl?

Edit: And BTW, if you think all 5 tracks sound the same, why not submit the results even if it's a 5-way-tie?

Because I want to prove to myself that I can tell the difference. That's half the reason we're all doing this, right?

Edit: Also, I wan't to help find the best codec. And if I don't zoom in on the details, how will I know which is best?

OK, your POV. As I said, I don't see this as a problem. I have no idea if that rip is from a vinyl or not, so maybe it's not clipping or an extraction error at all. You can rate that click if you want and focus on the click alone like you are trying to teach me that it's the best way, or you can think about whether or not that click is really so important for your overall listening experience / pleasure. Personally, I would either ignore it or rate it together with the rest of the music (or at least some seconds if not the whole sample) and deal with possible artifacts like with any other sample (microattacks, preecho...). I would also like to end the debate about sample 2 at this point since I see no point in continuing such a discussion.

No need to get all upset or angry. I'm not trying to teach anyone anything, I was merely joining in a discussion I thought was interesting, which is what I thought forum boards were for. I won't rate sample 2 from that single click if it is an obvious fault of some kind.

And yes, I also listen to a couple of seconds when rating the samples. (Although I would say that the deciding sound is usually only some fraction of a second - i.e. a snare, a cymbal, etc.

Edit: Perhaps I should explain further: I listen to the whole sample, but focus on the "weak spots".

Finding the best codec is a question of averages, and discarding worst-case scenarios. ABXing is only useful to detect differences, not quality.

I think rating this sample taking how the encoders encode the artifact is useful, just like how we rate preecho on some hard-samples. What cannot be done is rating the whole sample only by this, if there are other artifacts present aswell. (don't know, haven't done it yet).

I'm the 'author' of this sample, ripped several years ago.
If you listen closely to this sample you can ear several clicks that occur on the whole sample - and on the whole album if you got it. It's not a ripping issue nor a random noise: it's a mechanical sound. Like most instruments on earth harpsichord sound isn't electronically generated and the process required to create a note from the instrument isn't completely noise-free. This kind of annoyance are very common with solo instrumental music and are often magnified by the recording conditions (microphones close to the performer/instrument): fingers on the flute, "breath" of the organ, fingers on the bow of guitars/luth, etc...

Ah yes, these small details may cause annoying artefacts on some situations like low-bitrate encoding.

Yes, and everyone, including myself, do that too. However, our views of "weak spots" seem to differ.



Hey, you're alive! Long time since I last saw you online here.

Umm... ok...? BTW, now that we know the mysterious "click" is part of the recording, I'd say it's quite fair to use it as part of the rating. I wouldn't have otherwise, though.

That's the thing I don't understand - why wouldn't you? Even if it was vinyl clicking or clipping or extraction errors - why not? Maybe it was intentionally there to see how codecs behave when fed with such material? In my first test we even had a transcoded sample.

Oh, right. I misunderstood you. I thought you wouldn't either... hmm.. well, I guess I feel a sound codec shouldn't have to perform well on such things as ripping errors. Anyway, no big deal. I guess, in general, a codec which copes well with a click will also cope well with certain types of sounds in music, so I guess you're right

Thanks. Upon further investigation I discovered a few more of these clicks. I had suspicion they existed in the source but couldn't pass judgment without original material. I am familiar with orchestral noise and pitfalls of "one take" recordings, but the clicks in this case seem magnified.


I agree. Now that the artifacts are known to be part of the original recording and not extraction errors, its testing use is acceptable.


Correct me if I misunderstand, but the goal of this test is to see how well these codecs perform on typical well mastered, well ripped general consumer material. Certainly the bulk of the samples within reflect that viewpoint; is anything in here recorded from tape, vinyl, or anything that deviates significantly from the Redbook format? Analog and extraction artifacts force the listener to focus on outlier cases not typical of modern mastering processes. If it is indeed your intention to gauge low bitrate performance on mastering errors, then I misunderstood the goal. I brought the issue up because I believed I found a flaw in the test.

In any case, I disagree with the suggestion to ignore the clicks and trudge through as if they don't exist. Doing so forces the listener to be "honest", which blind testing seeks to avoid, while simultaneously handcuffing the tester by forcing him to use the second best detection method. As it stands, Id' say this particular sample is valid though not ideal for generic testing. I believe the average (well mastered) orchestral track to be less noisy but am not an orchestral aficionado so I will defer to the experts on this one.

To clear up some things.

I did not notice those clicks just until you pointed out that there is one after the first 4 seconds. Then I listened carefully and noticed it indeed. I did not include the sample because of those clicks, but simply because it features a solo instrument (harpsichord) and because the sample is also quite old.

Even if the click was an extraction error, I fail to see why that would be a huge problem in the test. The only advantage one might have (if he even notices those clicks) is that he can, more or less, successfully ABX the original and all samples making ABC/HR disable the reference sliders. Then it's up to the user again to rate and I highly doubt that he can give fair ratings by considering only a couple of milliseconds of a 25 seconds track, and those milliseconds not even containing anything especially related to music. What would you have done if I included a sample taken from a vinyl? Aren't we allowed to use lossy codecs for that task? Personally, I copied most of my audio tapes to 80 kbps MP3 because it's suitable for my ears.



Really? No where did I mention well mastered and well ripped general consumer material. Red Hot Chili Pepper's "Californication" is everything else than well mastered, but that didn't keep me from encoding it to MP3, even though all tracks clip like hell.

So - all in all it's your choice if you rate the click or not, but not submitting the results for the track only because of some clicks is something I don't understand. I also wouldn't even give much importance to those damn clicks, even if they were extraction errors (which they are not thanks to guru's clarification), simply because it doesn't change my listening experience if a certain click suffers from pre-echo or not. The annoying thing is the click itself that is also present in the original and not some artifact that changes the sound of a simple click if the rest of the file is transparent.

@Sebastian

I'm sorry I got confused regarding the legality of the disclosure of the results of this test using WMA Pro 10. I just picked up from where the other (closed) thread left off. At any rate, since there would be any issue regarding it...

then ROCK AND ROLL!!!

Can't wait to see the results. Much as I would like to participate, I don't have the minimum equipment.

You might be surprised. I'm plowing through with $35 Sennheiser PX-100 headphones and a noisy air conditioner a few feet away.

IMO there is no need for high quality equipment for this test. Headphones (<$30) and descent integrated soundcard should be enough. Noisy environment is bigger problem IMO.