I have to say, from experience, that even if the test conducter makes everything very transparent and well documented, there are still ways and means to influence test outcome.

I don't think it is safe to publish the ways to tamper a listening test while making it look perfectly valid, but trust me, there are several (both from the conducter side and listeners side). ff123 and I brainstormed some of them once.


Now, of course, I'm NOT saying Garf would tamper such test's results. I believe he is an honest guy and wouldn't do something so wicked, risking his reputation. But it is possible to tamper them.

OK. So you just said your claim is unverifiable under scientific conditions, therefore, it's pretty much useless. People here at HA aren't too fond to take claims for granted, specially if they defy common sense (32kbps being unidentifyable from bitrates 4x bigger)

I don't have your experience, but I'm curious as to how a test done with a completely transparent methodology could be tampered with by the test conductor. Further, how could it be unsafe to publish these thoughts you and ff123 had? Anyway, if you have good grounds for marking them "unsafe", then would you please consider sending them to me by a more secure channel (PM, e-mail...)? I still believe that it's possible to ensure fairness when there is complete transparency in how the test is setup and the results by the participants are treated etc...

Okay, I understand this point. But I do actually think that these factors are mostly irrelevant, unless they specifically impair one codec aspect in total (stereo imaging is what I have in mind here).

You can hear codec artifacts screetching through background noise.

What is going to matter most is listener attention, listener training and fatigue. All of these could be normalized in a well-set up test.



I claim the following:

1) Using good equipment (say, Audigy 2 NX + Senn HD580 + reasonably quiet environment) a mixed group of listeners (mixed age group and experience, but *including* a few trained listeners) will not be able to distinguish 48kbps HE-AACv2 against 128kbps MP3 with a statistically significant score.

2) Using average equipment (say, NForce4 Audio + PC speakers + reasonbly quiet environment) a mixed group of listeners (mixed age group, but *excluding* trained listeners), will not be able to distinguish 32kbps HE-AACv2 against 128kbps MP3 with a statistically significant score.

Do you have something in mind from the side of the participants or of the conductor?

Excellent! Now back to work!

And when you have finished your work on the Nero encoder, come back and prove your claims with a listening test. I'm willing to participate in the test.

I have an encoder here that will reduce WAV files to a bitrate FIVE times smaller and I say it's unidentifyable from the original (on average).

I even have one that will reduce the bitrate TWICE and they will be bit-for-bit-identical. Whoo!

I don't see how this defyes "common" sense. Technology advances.

Well, I was specifically speaking of 32kbps and 4x bigger.



It defies common sense in the aspect that "XXX is as good as MP3 at X times smaller bitrate" has been claimed by everybody+1 several times in the past, and it has been throughly debunked. Until proof comes out showing it is indeed true, it will be defying common sense.

(Somewhat to my disappointment), I found only 1 anchored test:

http://www.rjamorim.com/test/64test/plot12.png

This was 2 years ago, with a lot of trained listeners on good equipment.

Now, there is also this:

http://foobar2000.net/divers/tests/2005.09...HE/01/plots.png

which gives an impression of the advancement. As you can imagine, we (Nero) will not let the above result stand

That's why I sad it was throughly debunked. HE AAC, WMA, Real and MP3pro's marketing tricks were all shown to be false (at the time).

Right, so now your "common sense" is telling you any advances are impossible? Sorry, still making no sense. Someday that result will be obtained, and, in the conditions I stated, I hope that is today whenever-we-actually-release-this-thing-of-ours.

Edit: Reworded to please the nitpickers.

I am telling that I am skeptical until proofs contradicting that old data are presented. Simple as that.

I think that AAC-SBR/PS @32 CAN sound similar to average MP3@128 for most people in casual, not carefull, listening.
However, I do not think this is possible YET.

WOW ! TOS 8 violation by Gabriel!!!
Cease any LAME development immediately and provide proof as required!

Just kidding , sorry, couldn't resist...

Anyway, I'm with Roberto for what it's worth.

OT: Gabriel I luv u and I donated to prove it, now let's get 3.97 outta the beta phase!

Sorry, I but I avoided TOS 8.
"I think...can" formulation allows me to avoid TOS8. The golden rule is simply to avoid strict meaning and rather to specifically mention personnal ponderation words, like 'I think" or "in my opinion".

Don't worry, I saw ya tip-toeing on that thin line there Just thought this thread needed some lightening up, you can resume your LAME developing work now ...

BTW, I think I can ABX Winamp AAC-SBR/PS @ 32 kbps and LAME MP3 @ -V2 --vbr-new with my Creative MuVo Micro N200 1 GB and its standard earphones in a reasonably quiet environment easy , tho I'd prefer I couldn't as I'd like to be able to squeeze in mo music.

EDIT: Lodsa f**k ups.

EDIT 2 for the serious: I know my DAP doesn't support AAC/MP4 .

Do you really think that a blind test is the only scientific way to perform an study? Remember that I my claim is more related to people's habits than to codec quality. IMO a poll is more scientific than a blind test for this regard.

Regards,
Oki

Then go ahead, propose your methodology, conduct your test and produce your results. Stop beating around the bush, FFS! We already had enough wild, unbased speculations from you. It's time to present some hard data or STFU.

I'm not acquainted with the history of his beating around the bush, but the point here - a poll of a random sample of some population - may have some merit if the interest is in people's habits. The problem is that large-scale random-sample polls cost money, or a whole lot of time.

timcupery
department of sociology
university of north carolina, chapel hill (USA)

I understand. In that case, he should admit that he's unable to conduct such test and admit that his claims were based on pure speculation.

I find the whole discussion ironic, since so many people slammed Roberto's other tests as unscientific because the listening conditions were uncontrolled. And now they're being criticized (as a procedure for testing low bitrates) because the listening conditions are too controlled and not real-world enough!

I think Garf's 1st claim (at 48 kbps) is testable using the protocol Roberto has established to date. Garf's 2nd claim (at 32 kbps) is also testable using blind testing (I don't mean repeated ABX'ing), but would require some sort of questionaire to sort out trained from untrained listeners, quiet from noisy environment, speakers from headphones, etc., so you could pre-select the participants before analyzing the data. In fact, it's probably not a bad idea to just always include some sort of questionaire.

Garf's 2nd claim is essentially the same as oki's. Coming from Garf, whose listening experience I trust, the claim has some weight.

ff123

I would expect most people can tell 32kbps HE-AACv2 from 128kbps MP3 (Lame 3.97 -V5 --vbr-new for all i care), what's much more interesting would be a little poll in addition to the test:

Would you prefer 1 hour of MP3 music on your portable, or 4 hours of AAC?

That poll is completely misleading if you don't mention quality.

Very very interesting. We must discuss this further.

I meant ask it to the testers after they've tested. It gives an idea just how acceptable AAC is, even if it proves distinguishable.

Quite likely, but is it good for casual listening too? Your statement assumes that people are either transcoding to 32kbps aac before transferring their songs to their portable player or that they keep multiple copies. I have already quite often been in the situation that I'm filling up my ipod before driving with a selection of songs and find that even transferring them over firewire takes much too long. (you know, "let me just quickly put some songs on the thing" and have angry passengers wait in the hall for half an hour )

I keep my music on audio cd, archived in flac or ape format on dvd as backup and for transcoding once to the lossy flavour of the month and people think I'm a maniac. Whatever would they say when they would see me transcode all files to a lower-bitrate format whenever I fill up my portable player?

My point: I have my original cd's, a backup in lossless format as, well, backup, and the music I work with is either aac or mp3 at a bitrate around respectively 160 and 180 kbps. This seems to me (my ears at least, though I have not even tested he-aac) to be a minimal bitrate if your songs need to be played in several types of equipment. After all, who knows which stereo system I will have in 5 years?

On the questionnaire idea, I also think that this would be useful. (And feasible without prohibitive cost!) ABX tests are useful only for transparency - does the encoded file sound different than the original? However, some characteristics that are distinguishable from the original sound worse than others. So, following an ABX test with some question along the lines of "how bad does this artifact sound" could be useful in parsing out this difference.

Maybe it's a little OT, but what are the portable devices out today that support AACv2 with PS? Is it limited to phones, PDAs, or are there audio players?

ABC/HR Java allows comments with each rating, in addition to standard ABX testing. People should use this software when comparing lossy to lossy encodes precisely for this reason.

A feature being there doesn't mean it will automatically be used. I still see some good points in presenting everyone the same questions to get a better image of the differences between codecs and the way they degrade from original.

Well the ratings are not a feature, they are a requirement to doing a session. For those who have not done this before, the ratings are:

5 = Imperceptible
4 = Perceptible but not annoying
3 = Slightly annoying
2 = Annoying
1 = Very annoying

With gradients of one-tenth steps. Comments are available for the tester to describe the nature of the artifact, or anything else about the session in general. I guess I just don't understand what questions would be on a questionnaire that couldn't be achieved with the software that already exists. For example, if I rated something 4 or higher, I think that says I can tell the difference but not enough to get upset about.

Both tests are not comparable at all. First test is a collective one, and the second is an individual test, without anchor, and a different (very friendly) notation scale.
Compare this result to this one, and you could conclude that HE-AAC from Coding tech at 64 kbps is superior to LC-AAC at 175 kbps from Nero Digital and faac. Which is, obviously, wrong.

BTW, saying that for most (?) people HE-AAC at 32 kbps would be perceptually identical to MP3 at 128 kbps probably implies that these people don't hear any audible problem with both encodings. It's the famous "CD quality" claimed by several software editor. In other word, for these people, you can also claim that HE-AAC@32 = MP3@128 = PCM 44100Hz/16 = DSD 2820000000Hz/1bit = PCM 192000Hz/64bit etc...
Consequently, making such comparison doesn't really make sense. You can use any lossy format/preset for this equation. HE-AAC@32 = MPC 320 kbps is as true as HE-AAC@32 = MP3@128. But I'm tempted to think that for the same group of people, HE-AAC@32 would sound the same as LAME@90 kbps (which is far to sound bad), maybe less.

As a consequence, I wouldn't start a blind test in order to prove that HE-AAC with PS is four time efficient as MP3.

You misunderstood the point or misread my post. I wanted to show that 1) HE-AAC@64kbps was 'reasonably' close to MP3@128 kbps in the first test 2) that the HE-AAC encoders have improved tremendously since then (as much as 1.4 MOS points in your test).

I am not predicting what score HE-AAC will get from this, except that it will do quite a bit better than it did 2 years ago!

I stated *EXACTLY* what I am claiming a few posts up. And I think such a comparison makes a lot of sense, because it determines (for example) what format could be used on a portable device with limited storage.

I don't think the claim is the same as oki's. He was talking about noisy environments, the kind of areas where an artifact really has to "hit" you to be noticeable. I was talking about the equipment and conditions in have ready in my office, because that's a test I can actually perform!

BTW. "whose listening experience I can trust" --> If I am making statements about untrained listeners, I am working from my gut feeling and some personal experience with the quality of our latest encoders. My claims do stretch out maximally, as much that even some persons here said I would "lose" the test (though there's disagreement which one). But I believe that even if that would happen, the result would be darned close :-)

In any case, it would further our knowledge about the quality of todays state of the art, and I don't mind losing if we at least get some new data

I am sorry but these numbers are simply bogus. The paper you mentioned is outdated, and half of it is dedicated to LSP/LPC decoding, something which is not need for any file produced by any Vorbis encoder from the latest years. This is important for worst-case considerations, but in reality no vendor will have a problem with enforcing customers to not use years-old Vorbis encoders (in fact they are doing much more stringent things than that and customers still cope with it).

Xiph claims Tremor works on a 72Mhz ARM7TDMI with cache and a reasonably fast memory interface (not the most common config).

If I compare this to Real's claim of HE-AACv1 decoding, 60Mhz on an ARM9TDMI with zero-wait-state memory (similar remark as for Vorbis), then the requirements are almost identical.

These are the public, C based sources. I have no doubt much faster implementations are possible for both.

As I always understood it, Vorbis and AAC (LC?) are about on par WRT CPU usage, but Vorbis has some big memory requirement issues, specially at low bitrates. Is that correct?


Oki should get over the AAC zealotry. If I could do it, he should be able too

They're broadcasting 6pm to midnight MDT, M-F, and all day/night on the weekend in 320 kbps mp3. All other times are at 160 kbps mp3.

http://www.radiofreecolorado.net/

Not bogus but maybe outdated for current developements. Remember that current Vorbis DAP were developed using those numbers. My study is based on real numbers, real products and if you take a closer view, you will see that real battery life playing OGG on several OGG DAP matches my predicitions. I hope that with the current improvements, the requirements for OGG would be reduced.
Good!! that is a great improvement. Do you know any DAP with that codec?

iPod Shuffle has the same battery life when playing MP3 and AAC. That means one of these two options:

1st. The Apple MP3 decoder requirements are less but similar to the higly optimized Apple AAC decoder.

2nd. Apple MP3 decoder implementation is very bad and there are other players with a more efficient implementations.

The fact is that life battery is the same whem playing MP3 and AAC.

I am not an AAC zealot. I am more interested in signal processing in general. AAC has the tools for solving all the problems that MP3 has. It has a lot of potential because it is the new standard of the industry. There are other very interesting codecs but are more academic and the improvements can not be used for any real implementation. Playing with them is just for fun.

MP3 from my point of view is old and has nothing interesting. Working on improving the PAM is not as funny than working in a PAM for AAC since ti has more tools to play with. I like signal processing.

Regards
Oki

http://www.hydrogenaudio.org/forums/index....ndpost&p=330809

On another note, lots of Vorbis players from Korea use Tremor. Some examples that come to mind are IOPS and MobiBlu that use Telechips's chips. I don't know about recent developments in the Korean DAP market though.

What?? Do you have anything to substantiate that?

How would current Vorbis DAP's be based on those numbers? Do you have any idea how old RC2 is and how long it is since the LPC filter ("floor0") was removed from the main encoder? It was around the same time they were stabilizing their decoder spec.



Show me. I don't believe this at all.



Almost all of them use Tremor.

I can confirm that compiling the Tremor source with eMSVC 4.2 (arguably far from the best ARM compiler), without any ASM optimizations, decodes 128kbps realtime on a 60Mhz XScale. This is including the time lost reading the file from CF storage.

Garf, please, read the whole post where you took those lines, all the data is there. Anyway if you take 3 lines out of its context, the entire post, then they can be taken as misleading. I was writing about 2 implementations of OGG Vorbis: Rio Karma and iRiver. Please read post #22 and post #33 related to this question.

zombiewerewolf gave a link to an iRiver Battery test.

I just took the data from real a real DAP. I am pretty sure about my calculations and nothing else can explain the big difference in battery life when playing OGG. Why were they using 2 ARM processors? Why battery life difference is so big from MP3 to OGG? Chances are that Rio Karma was developed using the same conclusion as the mentioned paper.

Again, I took 2 DAP with OGG vorbis support. My information is not misleading, it is based on real products and not on theoretical performance. I took the "x18" factor from that document as an hypothesis and found that the numbers were consistent with other real implementation, so the hypothesis was true and both were using the same "x18" factor.

If you have valuable data from real life it is OK, if you share the information with the rest of the people, even better. Samsung has a very efficient implementation of OGG in their DAP, But I could not find enough information. I am even interested in buying a YEPP player because of that.

By the way, I never said that this is true for ALL implementations. I never said that and I always said the opposite. I will quote miself:

I am sorry if any OGG fan or you have taken it wrong.
Regards,
Oki

Aw please. Just an accidental choice of a more suboptimal reading algorithm when reading/parsing OGG bitstreams would already do this.

Been there, done that. (for MP4!)



Because there are cheap, massmarket, low power consumption chipsets optimized for portable players available, which consist of a dual ARM7TDMI core? (idea being that one chip decodes, other chip handles GUI)

Apple uses exactly the same solution for the iPod. Is it because AAC is so slow?



The chance is exactly 0, because the paper was written before the decoding spec was finalized (as I already said). The situation was rectified before anyone could seriously think about making an OGG player.



In real life, the complexity of HE-AAC decoding is about the same as OGG decoding.

In the current situation we will always find a DAP with the same battery life when playing AAC or MP3 (iPod Shuffle). In the same way, we will always find a DAP with the same battery life when playing OGG or MP3 (YEPP). Does it means that MP3, OGG and AAC decoding has the same requerements?

The answer is NO, but there are other modules inside a DAP that makes negligible the difference in decoding power only if the difference is not that big.

Regards,
Oki

What calculations? The ones you linked to are entirely based on the bogus 180Mhz figure. You admit calculating the power consumption is tricky because decoding is not the only thing going on, and then fill in the missing data with wrong data!

BTW. Another reason for the difference may be that MP3/WMA are handled by a special decode circuit (present in many chips or available standalone), while OGG is being handled by the ARM (as an afterthought because the device wasn't originally designed for that).

OK, are you suggesting that a suboptimal reading algorithm made Karma and iRiver OGG decoding 18 times more complex??



iPod was never less battery efficient for ACC than for MP3. Why should I think that they needed more power?

Maybe, but they did not make a battery efficient product, Hey! but only when playing OGG.

Regards,
Oki

YOU are the one claiming the 18x number.




Don't ask me, you are the one claiming that the other devices needed 2 CPUs just to be able to decode Vorbis: "Why were they using 2 ARM processors?"

Because they can get em cheaply, that's why! It has nothing to do with this alledged 18x speed disadvantage.

What exactly is the "data" here?

a) You say that battery life on device x is y% longer with MP3 vs Vorbis (agree!)
b) You say there is an unknown factor because decoding is not the only thing eating CPU (agree!)
c) Vorbis is 18x more intensive to decode than MP3 (quote paper), hence conclusions about unknown factor, "verified" against another device with *another* unknown factor.

It's the last step I vehemently disagree with. This isn't data, it's wishful thinking!

I made decoding speed test with foobar2000 (0.9b9) which I beleave gives some clue about decoding speed of the formats (with PC). With ~200 kbps VBR, AAC-LC was fastest, second was OGG and last was MP3. There wasn't very big differencies and of course the results are surely different with mobile devices, but my guess is that there isn't 18x differencies between them.