Castanets: ATRAC3plus hardware encoding, Hi-MD encoding vs. SonicStage 2.1 Options

[Cygnus X1]

This is a branch of another thread that compared ATRAC Type-R and ATRAC3plus to Vorbis, MPC, and MP3.

I was poking around at the minidisc.org mirror of the above thread, and noticed that somebody had claimed that a hardware-encoded 256kbps ATRAC3plus castanets sample sounds better than the software encoded sample from SonicStage. Thus, I've compared the two. (Note that my rankings have been adjusted to reflect the sound being "annoying;" I was too conservative when choosing rankings for the last test, due to my intense dislike of Likert scales )

ABC/HR for Java Version 0.4b, 05 September 2004
Testname: Castanets, SS2.1 vs Hardware

Tester: Angelo

1R = /Users/Angelo/Desktop/castanets_hi-sp_(nh700).wav
2R = /Users/Angelo/Desktop/samples/castanets hi-sp.wav

---------------------------------------
General Comments:
---------------------------------------
1R File: /Users/Angelo/Desktop/castanets_hi-sp_(nh700).wav
1R Rating: 1.7
1R Comment: Metallic distortion in middle of file. Awful.
---------------------------------------
2R File: /Users/Angelo/Desktop/samples/castanets hi-sp.wav
2R Rating: 1.0
2R Comment: Horrible!! Metallic, buzzing distortion in middle of file...even worse than last sample. Bad enough to be used as a low anchor!
---------------------------------------
ABX Results:
/Users/Angelo/Desktop/castanets_hi-sp_(nh700).wav vs /Users/Angelo/Desktop/samples/castanets hi-sp.wav
15 out of 20, pval = 0.02
Original vs /Users/Angelo/Desktop/castanets_hi-sp_(nh700).wav
20 out of 20, pval < 0.001
Original vs /Users/Angelo/Desktop/samples/castanets hi-sp.wav
20 out of 20, pval < 0.001

RESULTS: The hardware encoded sample is only a hair better than the SonicStage version; both exhibit the same type of artifacts. I've never heard a codec color the sound like that; I would compare the sound of the artifact to that of playing music from a speaker made of tin foil

(Edit: part of comment didn't copy; fixed).

This post has been edited by Cygnus X1: Sep 5 2004, 17:52

[Cygnus X1]

Have a look at the minidisc.org thread.

Looks like I've opened a Pandora's box by posting at MiniDisc.org; the site admin is already questioning my results as being absurd (not to mention asking about my headphones ). I have nicely tried to explain the importance of double-blind testing and the fact that these artifacts are not subtle by any means, but I'm not sure that anybody will buy it. It's a shame that objectivity is always seen by some as a liability.

(Note that I even toned down some of my comments from the ABX test (e.g., "horrible') as to not suggest a bias.)

This post has been edited by Cygnus X1: Sep 5 2004, 18:04

[MugFunky]

hehe. you gotta understand that ATRAC basec codecs are designed to only be played back on Sony headphones.

you wont be able to ABX on those babies! hehehe.*

*WARNING: claim not backed up by evidence.

[Cygnus X1]

Actually, IIRC, "ATRAC was the only codec designed specifically for music."

I think I've had enough ATRAC discussion for today. Every time I've attempted to bring up the quality of codecs at various MD boards, people act like I am some sort of audio terrorist and start going on the defensive. It's an inanimate disc, people, not something with blond hair and boobs!!! And I thought that I didn't have a life...

People are free to think whatever they want; I would rather think scientifically.

[ilikedirtthe2nd]

minidisk.org is not a very good place to start such a discussion...

[Cygnus X1]

minidisk.org is not a very good place to start such a discussion...

No shit, Sherlock I guess I'll have to sell my portable recorder on eBay now as opposed to their local classifieds

[Kouby]

Guys, come on...
I had no intention, by starting these tests, to create (yet?) another war between HA and MD boards such as minidisc.org...

Cygnus X1, guruboolez and myself have without a doubt rated ATRAC3+ as being a total loser on the castanets sample. The fact that others @ minidisc.org can't hear it shouldn't be a reason enough to flame them like this. We all have different ears...

[guruboolez]

full comparison between
- mpc --insane
- vorbis megamix II -q6
- lame 3.93.1 --preset extreme
- hardware atrac encoding at 292 kbps
- hardware atrac3plus encoding at 256 kbps
- software atrac3plus encoding at 256 kbps

CODE

ABC/HR Version 0.9b, 30 August 2002
Testname: ATRAC TYPE-R / ATRAC3+ Hi-SP / MP3 LAME / VORBIS / MPC Comparison

1L = castanets mpc.wav
2R = castanets ogg.wav
3R = castanets sp.wav
4L = castanets_hi-sp_(nh700).wav
5L = castanets hi-sp.wav
6R = castanets mp3.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: castanets mpc.wav
1L Rating: 4.0
1L Comment: I heard thousand times this "piou-piou" = mpc, without hesitation;-)
Slight artifact between 2.1 - 3.1
---------------------------------------
2L File: castanets.wav
2L Rating: 4.5
2L Comment: first guitar note is slightly smeared.
---------------------------------------
3R File: castanets sp.wav
3R Rating: 2.2
3R Comment: first guitar note is distorted. Attacks are dull, slightly smeared, but it's not offending. Sound is also synthetic and lifeless: maybe some background noise is missing.
---------------------------------------
4L File: castanets_hi-sp_(nh700).wav
4L Rating: 1.2
4L Comment: unsharp attacks, and buzzing -electrical- distortions in the middle. Guitar is smeared, and with a strange and electric noise too. Really awful.
---------------------------------------
5L File: castanets hi-sp.wav
5L Rating: 1.0
5L Comment: attacks are sharper (first four castanets), but are then slightly more distorted than previous file. Overall sound is also really synthetic
---------------------------------------
6R File: castanets mp3.wav
6R Rating: 2.0
6R Comment: unsharp on the beginning (first castanets and guitar). Rest is OK.
---------------------------------------
ABX Results:
Original vs castanets mpc.wav
   7 out of 8, pval = 0.035
Original vs castanets ogg.wav
   6 out of 8, pval = 0.145
Original vs castanets sp.wav
   8 out of 8, pval = 0.004
Original vs castanets_hi-sp_(nh700).wav
   8 out of 8, pval = 0.004
Original vs castanets hi-sp.wav
   8 out of 8, pval = 0.004
Original vs castanets mp3.wav
   8 out of 8, pval = 0.004

Few differences in my opinion between atrac3plus hardware and software encoders. The biggest artifact is slightly reduced with the hardware encoder, but it also introduce additional distortion (noise), audible on the first guitar note.
Both are rated near 1.0 but 0.0/5 is more realistic I think to maintain the real distance between other challengers - much better.

[Kouby]

@guru
The interesting thing here (at least what I noticed) is that this second time, you rated the MP3 sample to be inferior to the ATRAC sample. Though I do not think you used a different MP3 sample than the one you used the first time...

[guruboolez]

@guru
The interesting thing here (at least what I noticed) is that this second time, you rated the MP3 sample to be inferior to the ATRAC sample. Though I do not think you used a different MP3 sample than the one you used the first time...

What do you mean?

[Kouby]

@guru
The interesting thing here (at least what I noticed) is that this second time, you rated the MP3 sample to be inferior to the ATRAC sample. Though I do not think you used a different MP3 sample than the one you used the first time...

What do you mean?

Nothing, just wondering how that came up. First test you did, SP was inferior to MP3, now MP3 is inferior to SP.
I've not done many ABC tests, so I was just wondering if it could generally be "expected" to have two opposing results using the same samples...

[Cygnus X1]

Guys, come on...
I had no intention, by starting these tests, to create (yet?) another war between HA and MD boards such as minidisc.org...

Cygnus X1, guruboolez and myself have without a doubt rated ATRAC3+ as being a total loser on the castanets sample. The fact that others @ minidisc.org can't hear it shouldn't be a reason enough to flame them like this. We all have different ears...

Good point. I guess where I get aggravated is that we went through the trouble of trying to approach this objectively, and then have our results immediately questioned by those without anything to back up their assertions, save for their own subjective biases. This is exactly the kind of stuff we discuss every single day in my PhD program.

However, we need to be civil about the way we conduct ourselves on other boards; I need to be especially careful, given my aggressive Italian blood Of course, our arguments about assessment standards and Chi-square tests and educational funding tend to get a bit animated in my classes and practica as well.

This post has been edited by Cygnus X1: Sep 5 2004, 19:28

[guruboolez]

Nothing, just wondering how that came up. First test you did, SP was inferior to MP3, now MP3 is inferior to SP

Which test?

[Cygnus X1]

Nothing, just wondering how that came up. First test you did, SP was inferior to MP3, now MP3 is inferior to SP

Which test?

No, IIRC, Guru found ATRAC Type-R to be a bit better than MP3 (2.3 as opposed to 1.8, I think). Check the original thread, please.

[upNorth]

I found it quite easy to ABX 12/12 when comparing castanets_hi-sp_(nh700).wav to the original, with foobar's ABX component.

But, IMHO that is because the two samples are not in sync. Just to make sure, I also tried with ABC/HR, but it didn't make a difference (as expected i guess).

So, could someone fix the sync problem?

[Cygnus X1]

I found it quite easy to ABX 12/12 when comparing castanets_hi-sp_(nh700).wav to the original, with foobar's ABX component.

But, IMHO that is because the two samples are not in sync. Just to make sure, I also tried with ABC/HR, but it didn't make a difference (as expected i guess).

So, could someone fix the sync problem?

They weren't in sync for you? That's interesting. I didn't have that problem, though I only listened for the same 2.5 seconds in the middle of the file (where the noise was most apparent).

[Kouby]

Nothing, just wondering how that came up. First test you did, SP was inferior to MP3, now MP3 is inferior to SP

Which test?

No, IIRC, Guru found ATRAC Type-R to be a bit better than MP3 (2.3 as opposed to 1.8, I think). Check the original thread, please.

Nervermind, I assumed too much.
sorry about that...

[guruboolez]

No problem.
But between two identical tests, notation can change.
Here for exemple, I performed my first test with a different soundcard and headphone (Terratec / Beyerdynamic vs AC97 / Philips). Perception of artifact can slightly change for that reason (note that it wasn't the case).

But even on the same listening condition, different notation is possible. A notation is not only dependent on the PERCEPTION, but also on the EVALUATION of this perception. Lame and atrac have different quality issues: lame suffers from excessive pre-echo whereas atrac have problems related to noise, dull and lifeless sound (and also a small stereo issue). I've found today and yesterday that pre-echo was slightly more annoying than other flaws. But in two weeks maybe, I'll find pre-echo slightly more natural and less annoying than flaws audible with atrac.

Anyway, difference between lame and atrac (not atrac3plus) is very, very limited when compared to other competitors on this castanets sample. Both have serious problems, bot none exhibits awful artifacts. I find more interesting to note that vorbis and mpc are much better (and vorbis megamix superiority compared to mpc is also something new), and that atrac3plus is completely out of competition at this bitrate.