Hello.

I'd like to announce the opening of my MP3 at 128kbps public listening test.

The test consists of comparing how 6 MP3 encoders behave compressing 12 different samples representing a wide spectrum of musical styles.

People interested in participating are invited to visit this page:
http://www.rjamorim.com/test/mp3-128/presentation.html

The test will end on January 25th and results will be posted soon after.

UPDATE! - The test will be extended and it'll end on January 31st.

Best regards;

Roberto Amorim.

/me grabbing headphones

Why LAME 3.95? Why not 3.95.1 ?

From the changelog, .1 didn't introduce any modification to the ABR mode.

Testing 3.95(.1) seems like a very good decision to me, since this version needs the testing much rather than the proven 3.90.3, which is almost exclusively used with the VBR presets.

Excellent
First time for me, will try to do my best.

once again
thanks a lot rjamorim for all your efforts to make the audio codec world more transparent

and i want to have that everyone who reads this threads joins the test and spreads the word about it!!!

any info why #_6 (Audioactive I gues) is upsampled to 48 KHz?

i don't mean to sound stupid...but i really can't tell the difference between them.

Am I listening wrong or am i just blessed enough to enjoy low quality sound.

maybe you could give me an example in one of the tests to listen for so i know how to pick out the differences.

Reading this, "not good" came to my mind. Java Sound doesn't support any bitrates higher than 44.1kHz, and sure enough ABC/HR-Java chokes when playing this sample.
That's not good at all. What's worse I probably won't be able to do anything about it. I'll think about this at the weekend but I really don't think there's much I could do to fix this.

you don't sound stupid, I have picked 5 random test cases and 4 of them were totally transparent on all encodes to me (WinXP, so no possible Java playback problem).

In only one of the tests I was able, after many replays, to find slight errors. Of course not saying which one
But imho there are too many ABR encodes which make the test very hard.
128CBR is much more used in wildlife and I would have preferred one or 2 more true CBR samples instead of "fit-to-128" abr/vbr.
Oh well, going to the next 7 now Kudos roberto ofc, I forgot

WTF? I can't believe the encoder 6 is resampling the output.

Is it happening on all sample packages?

I can verify upsampling on _6 for sample packages 1,2,10, and 12. That's all that I have downloaded at this point.

It does for me under XP, etc.

Edit: I mean it does resample to 48khz.

--

Quick question. How am i supposed to be listening? I just did a test now and picked out a part with noticable high frequecy distortion in EVERY sample. In fact all but one sounded exactly the same at this point. I rated the one that sounded slightly better higher, and all the others the same. Was this correct? Or should i listen more and attempt to locate other artifacts and take those into account as well?

AAAARRRGHHHHH!

F**king shitty Audioactive. I hope it dies a painful death!



@Mike: You should consider the sample as a whole, and not only one artifact. So yes, you should try searching for other artifacts in there.

Optimized for Creative cards?

Just wondering, is the headphone requirement really a must? I have a pair lying around, but the cable's broken, so I don't get any stereo

headphones make a big difference!

But not big enough for me to hear the difference!

Big_Berny

This is very exciting. I can't wait until the results come out. I'm particularly interested in how bad Xing is going to be rated, considering it used to be my favourite encoder So Blade is not going to be tested?

you need to practise
take some time, listen often (3 times or so for 1 sample is not enough) and carefully to the clips (or only parts of the clips) in a silent room (with headphones)

and then you will hear how crap most encoders really are

I would bet a small amount that it won't finish last place in this test. Another one is yet worse.

Yeah, that's the issue that is worrying me. :/

People will whine Audioactive will be favoured in sound cards with crappy resamplers.


@danchr: It's better if you have them, but if you are able to detect the artifacts using only speakers... go ahead.

Probably not. The original will suffer crappy resampling as well as all mp3s except Audioactive. Therefore - if the differences in resampling are audible - Audioactive might sound 'better than the original' but the others will sound closer to the original.

AFAIK bad resampling efects most on high frequensies and with 128kbps MP3 there most likely is'nt much high frequensies left. But the important thing is that all settings are default...

This stuff tells me I need something better than Philips backphones and Creative soundcard.
I actually managed to ABX a sample file against the reference using cues due to offset differences. They're very small, but noticeable enough. It may NOT be offset differences, but well, it didn't seem to be artifacting at any rate.

Could you tell me where you found this problem? I'd like to check whether there really is a problem with the offsets. Maybe you could send me a personal message specifying the sample where you've encountered this?

The approach I suggest is to back off of this test for the moment. Start with the 64kb test (now closed) and/or some of the classic problem samples. Once you are familiar with the sorts of problems to look for, it will be easier to find more subtle versions of the same faults in the 128 test. Then you will curse us for ruining your enjoyment of what were formerly transparent bitrates.

good plan, where can I find the 64kb test?

What about taking into account the crappy sound cards / amps being used in tests. Soundcard may react very differently to playback between them. I have an old SB pro 16 that has so much hiss it's not funny. My LIVE 128 is no better. My Vibra is crap. MY Crystal one is not too bad. My AWE DSP 24/96 is great! especialy using MAD plugin in 24 bit dither mode which make most all .mp3 sound better than anything on any on the other 16 bit sound cards. Must be due to the accurate float point decode to 24 bit that MAD can do when your hardware supports it.

All I'm saying is unless a standard is given to listen with like a good pair of headphones (type N) and a (Type N) sound card, all tests will differ from place to place. Oh, but what about the digital out! Well that all depends on the Digital to analogue converter at the other end doesn't it. I have a nice Yammy with 24 bit Burr banks in it. That's different to my CRs Digital speakers with amp. Amps only Class a/b so digital or not, it's not as true as say headphone on the soundcard.

just a thought.

I tried to run the test in ABC/HR for Java, but the configuration files use \ as directory delimiter and the application doesn't convert it. I wanted to try fixing it myself, but my own compilation keeps throwing errors and exceptions...

http://audio.ciara.us/test/64test/presentation.html
I don't know if the samples are still up, though.

no, they aren't unfortunately.

alas, i'll just have to wait and see what the results are.

@MGuti: You can do artifact training here:
http://ff123.net/training/training.html


@SHADES:
The hardware doesn't matter. How good or bad each sample sounds doesn't make much difference, what matters is how they compare to each other.

So, as long as the sound card applies the same amount of noise to each one of the samples, the final results shouldn't come out too altered. After all, that's real world usage. I'm not trying to evaluate how compressors sound inside an acoustic lab, I'm trying to evaluate how they sound in the average person's PC

rjamorim is right, it is very much about the average person with average equip. I have access to an OK soundcard and sennheisers. But ill do this at work with an onboard card and old labtec phones (with no foam covers left - not good for extended listening!) I can quite happily/quickly A/B 4 of them with this (shame that the ABX function doesnt automatically update the front page if you've managed to get the probabilty to above 95%. Ive done this with one sample but the problem is i feel a bit pressured to get it right with one go on the front page - the difference is slight, but audible and repeatable.) Guess ill just spend a bit mroe time on it or break out the sennheisers.

As noted above, you can't run the test in ABC/HR for Java since one of the codecs resamples to 48kHz, which Java Sound doesn't support. Anyway, for the directory delimiter issue you might want to try this fixed version.
I also uploaded an updated sources package, which includes an Apache Ant build file to make it easy compiling the sources yourself if you want/need to (to compile just use "ant compile" or "ant dist" on a command prompt). I never really supposed anyone would want to look at these sources , anyway thanks for your interest.

Now, that's embarrassing.

I was just preparing for a fruitless search of an alternative sound API for Java to fix this annoying 48kHz issue, and it turns out that Java Sound indeed supports 48kHz without a problem. For some reason I just shortsightedly assumed it didn't, which resulted in a buffer overrun when loading anything over 44.1kHz .
Guess, I should have looked a bit harder for the cause of the error in the first place .

Well, the good news is, Linux and Mac (and whatever else) users can download a fixed version here:

Binaries
Sources

I apologize for the inconvenience, especially to Roberto.

Btw, without going into advertising too much, I'd be really happy if some people could give ABC/HR for Java a try, it has some additional features I found quite useful, which Windows users might like, too (for example, you can save and resume sessions, turn quick switching and looping on/off, see the playback position, set "bookmarks" in the timeline, etc.).

oh, come on! You already did so much for my listening tests, there's absolutely no need to apologize



I agree, Java ABC/HR is amazingly featureful.

I foresee the day that I will only be able to use it for my tests, mostly because of encryption capabilities. I see no need to use encryption at the upcoming AAC test - I doubt there is any strong zealotry towards, say, Nero and against QuickTime

But encryption would surely be welcome at the second 128kbps multiformat test.

Also, I plan to make my AAC test (and the others coming after it) truly multiplatform, with bash scripts as alternative to .bat files and an option to download the samples in uncompressed form for those that can't run even FLAC on their system (of course, the download would be a very big .rar file. Not much can be done about it)

I'll update the Java ABC/HR version inside the start package as soon as possible. Thank-you for fixing it.

Best regards;

Roberto.

...it would help if you revised the version number of abchr-java, too...

i.e. 0.4b has had at least 3 lives that the public knows about...

perhaps 0.4c? 0.4b2? etc.


thanx

As usual I can't ABX a single sample, sorry Roberto

Here you go: abchr-java-0.4b2.zip
(No changes except for the version string)


To those who have problems hearing differences, I can only say: Practice, practice, practice... Really, I was very surprised how much difference practice can make at hearing artifacts. In the 64kbit/s test (which was the first listening test I took), when I started listening to those samples there were always at least two codecs, which seemed to be completely transparent to me. With the Chopin piano sample, only one codec had obvious artifacts for me, at first (...and I'm playing piano for ten years now, which again shows that "being a musician" doesn't necessarily make you a golden ear). But after an hour of intense listening, searching for cues, trying dozens of spots in search for an artifact, I was finally able to abx all of the codecs. After that I went back to one of the easier samples I tried before, and I could identify every single codec almost instantly by just listening to the first second, the differences suddenly were just glaringly obvious (which really surprised me, actually I was kind of expecting that the ability to hear artifacts is more or less a physical predisposition).
I tried one (easy) sample of this test so far, on my (pretty moderate) speakers, with my noisy PC running, and I could identify all of the codecs after some practice. So, if you really want to find a difference (and I can understand everyone who doesn't), practice will probably get you there.

For the people who has trouble identifying the artifacts, i think sample 12 has the most obvious flaws.
All samples should be easily identified.

At lame --alt preset standard i cant ABX it though.

Btw, what artist is performing on sample1 and 3?

Don't underestimate how fanatic some mac users are. Plus, some script kiddie might attack you simply because it's possible


I'd be willing to help in this endeavour! I have access to at least three different UNIX systems, and to be truly cross-platform, there are a lot of assumptions you simply cannot make. If you tell me what the shell script should do, I could probably come up with something

What a surprise



The samples are the same as the 64kbps test. The names are available here:
http://www.rjamorim.com/test/64test/results.html

So,
1 = Counting Crows
3 = OMD (New Wave rulz!)

LOL. You have a point there.



Well, this script kiddie probably won't know about standard deviation in listening tests. He probably doesn't even know about reference ranking. So I would probablyeasily screen it out.



Great!

For the AAC test, my only assumption will be that you have FAAD and FLAC somewhere in %PATH%. FAAD compiles well at least in Linux and MacOS, so it will hopefully work on BSD and Solaris. I don't know if there is any other Java-supporting platform worth caring about.

So, the script is at the ./bin folder, it takes the encoded samples at the ../SampleXX folder and decodes them to WAV. I usually do a cleanup after decoding (removing the samples, leaving only WAVs there), but that's not needed. I also like to avoid screen output (basic obfuscation). You can have an idea about how my .bat works by looking at one of them.

Thanks.

Regards;

Roberto.

A couple of initial thoughts:
a) The binaries ought to be compiled for x86 Linux and Mac OS X. I have PPC Linux installed, so I might as well do that too.
b) It can safely be assumed that anyone not running one of the aforementioned systems know what they're doing and thus know how to compile software
c) It ought to be possible to extra the sample number, etc., from the name of the executable script.
d) For Mac OS X, a double-clickable file should be created. ABC/HR should also be adapted to the Mac OS X style if possible (it is).

I have 9 days of vacation now, so for once there's a chance I actually get something done

(Random thought: Shouldn't schnofler be in the Developer group?)

In theory this is a matter of changing one line of code. You can just tell Java to use the platform's look and feel. However this sometimes messes the UI up badly if you use custom visual styles on Win XP (which I happen to do), and since I don't have a Mac (or a Linux installation) available for testing, I took the safe route and used a uniform L&F for all platforms. However, if you are willing to try it out just replace

with

in abchr.gui.Main.

My Java is a bit rusty, but I actually tried to do something pretty much like that. I just kept getting Java linkage errors

A little late, but oh well...

I've made .torrents for all of the sample packages. You can get them at:

http://www.foobar2000.org/Sample01.zip.torrent
http://www.foobar2000.org/Sample02.zip.torrent
http://www.foobar2000.org/Sample03.zip.torrent
http://www.foobar2000.org/Sample04.zip.torrent
http://www.foobar2000.org/Sample05.zip.torrent
http://www.foobar2000.org/Sample06.zip.torrent
http://www.foobar2000.org/Sample07.zip.torrent
http://www.foobar2000.org/Sample08.zip.torrent
http://www.foobar2000.org/Sample09.zip.torrent
http://www.foobar2000.org/Sample10.zip.torrent
http://www.foobar2000.org/Sample11.zip.torrent
http://www.foobar2000.org/Sample12.zip.torrent

Or for all of them together:

http://www.foobar2000.org/samples.rar.torrent

I'd appreciate it if the official links were updated to reflect this for the time being. For the moment they are using the foobar2000 BitTorrent tracker, but this will be changed as soon as I finish setting up a separate tracker for Rarewares, which should be soon.

If people with fast connections/fast servers could help seed these, I'd be grateful

Thank-you very much, Dibrom

Already added information about the torrents to the readme, and updated the Java ABC/HR version.