Hello.

As most of you already know, I'm planning an extension test for the AAC@128kbps test. It'll be the AAC winner (QuickTime) versus Musepack, Vorbis, Lame ABR, WMAv9 and maybe Atrac3 (more on that later). The test should start (if everything goes right) in a week (the 23rd)

There are some subjects where I'm still not sure how to proceed, so I would like to rely on the forum users' knowledge:

1 - Should I include Atrac3 (used on Sony MiniDiscs and Real Audio at mid-high bitrates) in this test? Is there anyone out there interested in this format? (besides Den ;) )

2 - I plan to use anchors in this test. For those that don't know, anchors are samples that suffer no processing from the original apart from lowpass. They are used to put the results into perspective and avoid participants rating the encoded samples too high and/or too low. The questions that arise are:
a) 2 anchors (high and low) are needed, or only one anchor is enough? (IMO, two anchors are more useful in low bitrate tests, and don't make much sense in high bitrate ones)
b) JohnV votes for one lowpass for each sample, else some samples might sound better lowpassed than their original versions. ff123 votes for one fixed lowpass for all samples, to put things into perspective. What's your personal opinion?
c) Anyone has any idea of good lowpass(es) for the test? It's a difficult choice since low lowpasses might sound too unpleasant to the fellas with good hearing and pro gear, but high lowpasses might sound too transparent to the fellas with not-so-good hearing or simple gear.

Thanks for your help.

Regards;

Roberto Amorim.

I'd go along with JohnV on this one, since I've had problems with samples sounding better lowpassed. As for the frequency, I used the PCABX Training Room to test my limits, and I'd say either the 15 or the 18 kHz, as those are the ones where I actually had to focus.

Edit: I'm a newbie, so don't take my opinion as being worth very much. Just an extra push one way or the other, if the opinions are split even. I'm sure some of the regulars could make much better arguments than I did (actually, I don't think I made any arguments).

I look forward to the test!

I was thinking of a 10 kHz lowpass, which should sound pretty bad to most people on most samples. The idea is to prevent a codec, which may sound pretty good if evaluated by itself, from being given unfairly poor ratings when compared with the other codecs. For example, lame might be predicted to be the loser of this test. But surely it doesn't deserve to be rated near 1.0, which might happen sometimes if there's nothing to keep things in perspective (what then about truly crappy codecs like blade?)

I understand JohnV's concern, though. Some types of music (for example solo piano) might still sound good with a 10 kHz lowpass.

Actually, another possible idea is to use blade as the anchor. I'm pretty sure this would always be rated the worst, and is likely to occupy the position near a rating of 1.0

I think only one anchor is needed (the low anchor). The hidden reference, which must be chosen from in each comparison, should be functioning as the high anchor.

ff123

Either 15 kHz or 18 kHz would be useless for me as a participant (and I presume for some others as well). Those lowpasses are transparent to me in music.

ff123

I would be interested in seeing how ATRAC stacks up (alliterative pun not intended) only because I have an MD player and find the sound pretty good. I would like to see how it rates against the other compression systems. Not a big deal; only if it is not a pain in the butt.

Someone should really create a Listening Tests Forum. All these tests are hard to find when they get old.

The following topics could be moved to the "Listening Tests" forum:

Link 1
Link 2




<edit: added links>

and boojum! Wow, I'm not alone!

In all seriousness. My first thought is to give ATRAC3 a miss because of the limited interest and usefulness. On the other hand, it would be interesting to see just how it does compare with others at this bitrate, not that your average Minidisc user can do anything about it. Mind you, there are the new Sony mp3/ATRAC3 CD Discmans (or is that Discmen ) now, so maybe it would be useful for somebody out there.

Roberto, how are you planning to make the ATRAC3 files and then present them for testing decoded into wavs? If you use the Sony software, there is no export as wav option. If you use RealProducer, my understanding is that this version of ATRAC3 is not as polished as the latest versions used in the Sony software...

Den.

Sounds like a job for TotalRecorder. It's a pain in the ass, but it can be done.

ff123

Does this approach leave any opportunities for noise being added to the file due to the recording step, or is it noiseless/lossless?

I don't want to seem picky, I'm just curious.

Den.

1. I'm also interested in how ATRAC performs compared to other alternatives for portable use like mp3 and aac.

2. Sounds like a good idea to use an anchor. And for an anchor to be useful it should be fixed, right? Anything below 12 kHz would be fine for me. Not too low though - if less than say 6 kHz I think the gap to the tested codecs will be too big to actually be able to relate their quality...

Even if atrac3 does not seems really interesting for us, I think that it should be included because Sony is trying to push it on portable devices instead of mp3.

It's amazing what you can find if you really, really put your mind to it. I've tried to find ways before of playing with ATRAC3 outside of the Sony software for my own educational purposes, but I always lucked out.

I just tried again, and stumbled across this...

ATRAC3.zip

It is an acm style ATRAC3 codec, and once installed you can encode and decode ATRAC3 in the three common bitrates using the windows audio tools. I quickly tried it with Audiograbber (it was handy), and hey presto it spewed out an ATRAC3 132 kbits file inside a wav wrapper, which I could then playback/replaygain/write to wav etc with trusty foobar. B)

I then encoded the same file with OpenMG, and tried to do a file comparison to look for differences, but the Sony version is an .omg file and there is a slight different in size ~4k in a 8.5 Mb track (@132 kbits). The Open MG generated file has tag data on the end when you check it in a hex editor. Someone else who know's what they are doing might want to look at this more closely and see if this is a reasonable way to generate some files for testing that are representative of what the Sony software churns out.

I also had a quick look at the decoded file in EAC. Has a radical haircut above 17500 Hz... :x

This should make things easier for you Roberto if you do decide to include ATRAC3 in the test. I'm also going to have a play with this further just for my own interest...It makes ABX tests with ATRAC a shed load easier.

Den.

The ACM ATRAC3 encoder is totally different from SonicStage's one.
Speed, first (on my Duron 800) :

ACM = x4
SonicStage = x16

Then, quality :

LP4 with atrac3 ACM is a joke. I rarely heard something worst. SonicStage encodings are totally different.
LP2 is more problematic. SonicStage encoding are not very good : lack of details, easy to ABX on classical instrument for exemple. It sounds a bit synthetic to me.
AMC LP2 encodings are crisp, more detailed and more natural : ABXing is therefore more difficult on non-critical parts. Nevertheless, I noticed that cymbals are suffering with this codec (ACM). SonicStage, in comparison, has less problems.


In my opinion, ATRAC3 is interesting to include in this test for one reason only : Minidiscs performance. If the codec doesn't match with MD device, it's useless. In other words : sonicstage (+ Total Recorder) or nothing.

i vote for atrac3 too (there is an interest from the ripping community side too)

Dammit... You really know how to let a guy down gently guru...

Shame about the ACM codec being not representative. I really had my hopes up there for a while. Oh well, back to TotalRecorder. I thought it was a bit suss when the codec file dates were 2001.

Thanks for the clarification guru.

Den.

PS: It would be nice to know what the differences are though...

I would like to see atrac3 in the test, too.
Why don´t you use helix-producer as encoder? You can download it for free.
Also it is easy to decode in GraphEdit when RealOne and Gabest RealMedia Splitter is installed:

atracfile.rm -> Real Media Splitter -> Real Audio Decoder -> WAV Dest -> out.wav

Real supports Atrac3 at these bitrates, all only at 44100Hz, Stereo:

66 kbps Lowpass: 12,4kHz (flavor 0)
94 kbps Lowpass: 15,1kHz (flavor 1)
105 kbps Lowpass: 13,7kHz (flavor 2)
132 kbps Lowpass: 16,5kHz (flavor 3)
132 kbps Lowpass: 16,5kHz (surround mode) (flavor 8)
146 kbps Lowpass: 16,5kHz (flavor 4)
146 kbps Lowpass: 16,5kHz (surround mode) (flavor 9)
176 kbps Lowpass: 19,2kHz (flavor 5)
176 kbps Lowpass: 19,2kHz (surround mode) (flavor 10)
264 kbps Lowpass: 22kHz (flavor 6)
264 kbps Lowpass: 22kHz (surround mode) (flavor 11)
352 kbps Lowpass: 22kHz (flavor 7)
352 kbps Lowpass: 22kHz (surround mode) (flavor 12)

This information is from the Codec Information Table by Real. I don´t know what the surround mode means, maybe a different (none?) channel coupling. Also I don´t know if it´s a mistake that the lowpass of 105kbps is lower than the of 94 kbps. "flavor" is the number you tell the encoder (write in the audience-file) on what mode/bitrate to encode.

When you'll be trying WMA9, be sure to test both Std and Pro. The latter one is very well able to compress 44khz stereo at 128kbit, and it seems to sound better. It's also a PC playable format, neglecting it would be disappointing.

I think if both WMA versions will be tested, Atrac must be dropped. Otherwise there are way too many codecs to rate. Imo 5codecs+anchor is already pretty much maximum.

Personally I'd be ready to drop MusePack from this low bitrate test. Imo MPC lowbitrate has no practical meaning to most people or even to MusePack users outside "nice to know"-factor. I don't believe many people will be using MusePack at this low bitrate even if it does well in the test.

My thoughts exactly.

I would also like to see ATRAC3 tested.

Most people around here are quality freaks so I don't see the point in testing the standard WMA9. Anybody who is after the best quality with WMA9 will use the Pro version, therefore only WMA9 Pro should be included in the test.

I would also like to see QT AAC at 128kbps up against Nero 6 -streaming.

Personally I'd rather include Musepack than ATRAC3, but I can understand that people, who already own a minidisc player, might be interested in the results.

Including both versions of WMA is pointless IMO. Most users won't be using Pro and it does not yet have the hardware- and softwaresupport of the earlier StreamVersion (AFAIK the streamversion is officially called WMA2), so its inclusion would be just as pointless as Musepacks.

dev0

Hello, people. Thanks a lot for your help.

About atrac3: Well, so I guess it's decided, atrac3 will be tested.

@Den: I already expected the ACM to be worse, I had it for years, so it can only be deprecated.

About SonicStage? Is it freely available or only comes with Sony gear? In this case, I might have to ask someone to encode-decode the samples for me.


About Musepack: I know the results might not interest much the HA community, but keep in mind they will be published on several boards I participate, at the front page of CD-rw.org and maybe even Slashdot.

Besides, it's known Musepack does well at this bitrate, but I still believe a comparision to other formats will be welcome. Specially since portable support isn't a dream anymore, it seems.


About WMA: Should I test WMA PRO (that has better quality but worse support) or WMA Standard (That has worse quality but much better support)? I know the ideal would be testing both, but IMO testing more than 6 formats, plus anchor, will be unbearable. (And these results will _really_ not interest much people, due to the bias against MS and whatnot)


About the anchor: Do people agree with ff123, about using Blade to create the anchor? (lol)


@kl33per: Nero will be tested against other formats later this year/early next year (maybe). Again, I would like to test it now, but few people would be inclined to test a multitude of samples.


About the samples: Guruboolez raised a question that there is no real classical sample (both classical samples in the 64kbps test were replaced with problem cases). So, do people agree with exchanging, say, LifeShatters (that is a bit redundant, stylewise, to BlackWater and Thear1) with Bachpsichord? Or people have another suggestion about a sample to be replaced?


Well, I guess these are enough issues raised.

Again, thank-you very much.

Regards;

Roberto.

I'd be very interested in seeing how the encoder from Nero 6 stacks up as well. It sounds as though this may not be in the spirit of this particular test, but I thought I'd throw it out there...

-raymond

I suggest to replace BeautySlept - harpsichord sound, but maybe synthetic - by a real instrument recording (as Bachpsichord, or many others).

Nevertheless, I think that this test should include a most common classical sound : orchestra, choral, piano... something familiar. Two at least are on ff123 website : Macabre.wav, and fossile.wav - this last one was used on ff123's 128 kbps test. Fossile is ABXable. So why not this one, as common classical sound, and a harpsichord, as a more problematic classical sample?

You mean we have to listen to the same samples once again? I got tired of them already in the first test Is it because it's a good test suite or because you want to be able to compare results between the tests? Anyway, I probably can't take this test, so I wonder why I even care...

I forgot this one :
SonicStage can be downloaded ; the software is big (70 MB), and I don't know any link for the moment. Recently, many were dead.

SonicStage produce .omg file : atrac3 + encryption. I can't sent to somebody an .omg file. DRM system of Sony forbid to anyone to read an .omg file that wasn't created on his PC.
Therefore, if someone want to upload you the files, they must be in PCM format (decoded + encoded losslessly). It's difficult for me to do it : I'm limited this summer to a 56K connection.

Keep in mind that we had to cheat with SonicStage in order to play the ATRAC3 file. If the original (.wav) file is still present on the system (same path, same name), this one will be played (and captured by TotalRecorder). It's easy to do a mistake, and to conclude to the superiority of Sony's format (I saw an occurence on a MD dedicated board, where the LP2 encoded was identical to the original).

I vote both for keeping Musepack and for using WMA9pro.

ff123

1. BTW, I verified that the time offset of WMA9pro using dbpoweramp to decode is 0.

2. The filesizes of the downloads will be much larger this time around, one reason is the higher bitrate, the other reason is the proprietary formats. You'll probably have to flac the wma9 and the atrac3. Are you going to batch encode the ogg, lame, and other encodeable files as well as batch decode? That would make things smaller.

3. On classical, fossiles.wav is a possibility, although Filburt once told me he couldn't stand to listen to it because he said it was recorded so badly (I personally am not annoyed by this clip). So I would vote to replace LifeShatters with fossiles.wav or macabre.wav, and also second guruboolez's suggestion to replace beautyslept.wav with bachpshichord.wav.

Just FYI: BeutySlept is one sample where MPC:s PNS screws up. Except a trumpet/horn from the SQAM collection, I haven't heard that as clearly as with this sample, so I think it would be nice to leave it in. Maybe the other "real" harpsichord is as tough though... I wouldn't know. Would you mind doing a quick test, Guruboolez? Could seem unfair to the other codecs to remove a problemsampe fro MPC...?

Ah, there's that harpsichord again. B) I agree with Guruboolez that there should be some classical sounds in the test. Choral would be great: voice, as we all know what that sounds like, and the mass of the symphony orchestra. How about something from Beethoven's 9th? Freude!

One 'problem' with bachpsichord (and with beautyslept too, but at lower degree), is the bitrate reached with --radio profile : near 200 kbps...

P.S. What to you mean by "screw up" ?

That it's a clear difference with and without PNS active.

Is PNS lowering sound quality ? Or is it an exemple of the benefit of Klemm PNS algorithm ?

With PNS sounds worse than without. For me I should add... But I don't always trust my own ears - will you try it?

if it is generally accepted that wma9 pro will produce better results than wma9 standard then i will vote for using wma9 pro (although hm i dont really want that the codec is getting pushed if it is really good compared to the others )

No problem. But for the moment, I must shut my computer down : thunder is more than menacing...

I'm not in favor of adding any other form of lowpassing other than the codecs use themselves. Most importantly we should be testing real world performance imo. Also expect alot of people who mistake hifi with lots of high frequencies to complain about overall dull sound. It will be a mess to explain this step to all those self appointed experts (especially outside of HA) who will disregard such a testing method. It all depends on what you want this test to be.

Another problem of artificial lowpassing would be readjusting the bitrate of the vbr codecs.

One valid reason for an artificial lowpass might be to reduce ogg Vorbis' hf boosting bug.

I'm not sure about adding mpc. People will extrapolate the results of the 128k test to other bitrates and this might shed a wrong light on mpc. Mpc doesn't suck at this bitrate, but I expect ogg or aac to sound better by quite a margin. Mpc is not intended to be used at 128 and probably noone will, so why bother testing? On the other hand, maybe mpc will present itself as a viable alternative and this is a good chance to find out.

Oh yeah, I think trance and celtic music is definately underrepresented in this test.

@Gecko: I think you didn't understand the purpose of the anchor. The anchor is only one file, and it suffers lowpass (or any other process to degrade quality) in order to put things into perspective and to avoid codecs being rated too low.

The samples encoded with the competing codecs suffer no preprocessing like lowpass.

About MPC: From some preliminary tests conduced by Den (and Guruboolez?), I would expect MPC to win this test, or at least come close to the winner. :B

Regards;

Roberto.

OK... unless someone comes with good reasons to use WMA std or to sack Musepack, I'll go with your vote.



OK. Thanks a lot.



Yeah, I would also batch process the anchor, now I only need to know if I'll use Blade or Sox for that. >_< (blade = encode, sox = lowpass)

If we use blade, I would also have to find out if there's any offset introduced when decoding it's encodes with Lame. <hint>



So, I think we can use Macabre instead, just to be safe.



Good. It's settled then.

@ErikS: I'm using (mostly) the same sample suite because it's a good suite (representing several different styles) and because I'm lazy.

I'm thinking about calling for new samples for the 64kbps extension test, but it's still too early to talk about that.

Ok, so what's (nearly) settled (unless someone comes with good arguments)

- Atrac3 will be tested
- Musepack will be tested
- WMA PRO will be tested instead ow WMA std
- LifeShatters will be replaced with macabre and beautyslept with bachpshichord

What's not settled:

- Will the anchor use lowpass or Blade? If it uses lowpass, one lowpass for all samples or one for each sample? If it uses Blade, plain 128kbps CBR?

Regards;

Roberto.

What kind of "voting" that is?
If this will be solved by voting, you should create a poll...

Anyway, I vote exactly opposite to ff123: sack wma9pro and MPC and keep Atrac.

6 codecs+anchor+original is hell of a job for those who want to do the full test with all samples.

Where can I find this Bachpsichord samle?

Sack WMA and MPC, favouring Atrac??? Heck, Atrac3 loses to other codecs (in features) in nearly every imaginable way (not multi platform (only Windows)), only CBR, no open decoder or encoder, not usable in media players like Winamp or Foobar, not usable in movies besides Real ones (unless you do hacks)...

I don't understand what is going on here. :B

And I agree 6 codecs is nearly too much, but, IMO, most of them aren't "sackeable". I see good reasons for including MPC, but, still, I think a poll would be the way to go.



@ ErikS: Ask Guruboolez

I'll try out blade tonight to see what it will sound like on the test suite and also to see if it has an offset

ff123

Too bad Lifeshatters is out ... I found the clip 'interesting'..

no legal way


should be possible with gabest's filter, i think...

I would like Atrac3 comparision!

Is the testing something a newbie like myself could help with? I actually tried to get in on the bitrate analysis phase, and had everything I needed to do it downloaded and ready to go. Then I got really busy re-encoding my collection and didn't have the time or HD space to follow through.

Now that I'm finishing up all my encoding, I'd like to take part in the ABX phase, unless a group has already been identified to do the testing.

I've only briefly skimmed this thread, so I could be missing something, but here's my opinion: I would personally like to see MPC included in this test, especially if this test is going to be geared more towards the users of this board than perhaps the general populace (and let's face it, it probably will be even if designed otherwise). There are a lot of MPC users here, and there has always been an open ended question about it's low bitrate performance. It'd be nice to see it put to the test once again.

I think the same argument above probably would work against atrac. While it is fairly widely used overall, I don't think it is particularly widely used on this board. Furthermore, it'd almost seem like a purely academic excercise to include it since I figure that it doesn't have a good chance of doing very well at all considering the more advanced and more highly tuned codecs it will be competing with. It could perhaps provide some useful results to point people to, but I believe most people who are serious about audio compression (which are the people who will appreciate these results the most) already realize that the alternatives are better on just about every front.

As for WMA vs WMA Pro, I'm not really sure. If we test WMA Pro, it has to be made very clear that there is a distinction between this and normal WMA, and that WMA Pro will not work on most (any?) portable devices. I think maybe it'd be best to actually test both, but that might push the codec count too high.

Personally, I'd favor dropping Atrac first out of those available.

Perhaps it'd be best to really define the target audience of the test before deciding which to include and which not to. Are we testing for the masses here? If so, then MPC doesn't make sense and WMA Pro probably doesn't either. Atrac would though. If we are testing for the more enthusiastic and technically minded users (like the people here on HA -- again, those who I think would appreciate these results the most also), then WMA Pro makes sense over normal WMA and MPC makes more sense than Atrac I think. If we're trying to strike a balance, then maybe it'd be best to go with my original suggestion: WMA/WMA Pro and MPC instead of Atrac.

Just my $0.02...

imho the audience will be all people who transcode audio cause it will be the first independent representative codec comparison @128 (as i know)

on any discussion on the net (professional or not) about codec quality this test could be quoted (which makes it very usefull)

so including the most used codecs will be necessary to keep the test as much usefull for as most people as possible (not only for some professional super-minds )

Ok, thx for clearing that up. I guess this was confusing me. Heh, actually, it still is: