Help - Search - Members - Calendar
Full Version: Multiformat@128kbps listening test - FINISHED
Hydrogenaudio Forums > Hydrogenaudio Forum > Listening Tests
Pages: 1, 2, 3, 4, 5, 6
SebastianG
QUOTE(maikmerten @ May 29 2004, 12:34 AM)
QUOTE(SebastianG @ May 28 2004, 11:05 PM)
It is a widely believed fact that we are unable to perceive phase differences of high frequencies, so IS is an appropriate tool, even for near transparency encodings.

The problem with MP3 IS is that itīs not possible to restrict IS usage to certain frequencies - you can only switch stereo modes on a block level, not on a frequency one.


This is a quote from the mp3 specification:
QUOTE
Intensity Stereo
This mode switch (found in the header: mode_extension) allows switching from 'normal stereo' to intensity stereo. The lower bound of the scalefactor bands decoded in intensity stereo is derived from the "zero_part" of the right channel. Above this bound decoding of intensity stereo is applied using the scalefactors of the right channel as intensity stereo positions. An intensity stereo position of 7 in one scalefactor band indicates that this scalefactor band is NOT decoded as intensity stereo.


I guess this means the encoder can choose some kind of split frequency. Below this frequency L/R or M/S coding is applied and above IS coding is used.

Agree ?

bye,
Sebastian
robert
QUOTE(SebastianG @ May 29 2004, 12:31 PM)
I guess this means the encoder can choose some kind of split frequency. Below this frequency L/R or M/S coding is applied and above IS coding is used.

Agree ?

yes.

M/S coding is some special case of doing some main axis transformation of the stereo plane and transmitting the rotation angle, the sum and the difference signal. for mid/side coding the rotation angle is fixed and is not transmitted.

IS coding is some simplification where you leaf out the difference signal.
SirGrey
QUOTE
ff123: mp3enc31 is recognizable by low frequency glitches. Ironically, bAdDuDeX (an mp3 connoiseur from long ago), who could hear a 16 kHz lowpass in applaud.wav, loved mp3enc31 despite the glitching and despite its relatively low 14.5 kHz lowpass because it was free from high-frequency ringing.

BTW, it was officially recommended to change it's default lowpass to -bw 15995 (number can be wrong).
And it's ability not to produce ringing and very small pre-echo always was a point all free developers gonna to achieve, as I remember. biggrin.gif
Interesting, may be now, when mp3 seems to be already mature standart, we could find somebody from Fhg and ask how they avoid ringing in their encoder ?
If it is not already known, of course...
SirGrey
I've had a strong feeling that I already saw this discussion about mp3enc.
I've find it at the end: Test old Fhg encoder or not
QuantumKnot
I can also confirm that the current Vorbis encoders use a mix of lossless stereo (full mag, full ang preserved) and point stereo (zero ang) below q 6. Point stereo kicks in for components above a certain frequency which is dependent on the quality. For lower quality values, more point stereo is used, hence the recognisable 'stereo collapse'. It does not appear to be the optimal way of doing things but considering the quality we get from current Vorbis, it doesn't do a bad job either. Monty has plans of implementing a better stereo model.
Gabriel
QUOTE
Because of its so so legal status none of these programs can incorporate Lame, even if many popular applications work with it.

Lame project is only providing a technology implementation. It is up to the the company wanting to use it to acquire a patents license regarding the mp3 patents.
Several companies choosed this solution and are using Lame in their products.
Ivan Dimkovic
QUOTE(SebastianG @ May 28 2004, 10:50 PM)
Anyway, I'm surprised that LAME peforms so well WITHOUT Intensity Stereo in the 128-ish bitrate area - Same for FAAC.  (no IS AFAIK)

There is absolutely no need to use IS @128 kb/s for MP3 or AAC.

And I also disagree with the claims that IS could bring good quality - there are lots of cases with stereo configuration impossible to code properly with IS, because IS saves only ILD information (level difference) and not ITD (time difference) and inter-channel cross corellation.

Equalized and mixed "left" (IS) channel could completely distort the phase information, and you end up with something which is quite different from the original when the coloration of the sound comes into the question.

Applaud is one of the examples that is impossible to code properly with IS.

Smart psychoacoustic would be able to disable IS for such frames, but @128 kb/s there woud be no need for lossy bit savings, same goes for PNS (in AAC) more or less- we did a lot of tests with PNS @128 kb/s and in most cases is pretty much useless, or degrades the quality.
maikmerten
QUOTE(robert @ May 29 2004, 10:51 AM)
QUOTE(SebastianG @ May 29 2004, 12:31 PM)
I guess this means the encoder can choose some kind of split frequency. Below this frequency L/R or M/S coding is applied and above IS coding is used.

Agree ?

yes.

M/S coding is some special case of doing some main axis transformation of the stereo plane and transmitting the rotation angle, the sum and the difference signal. for mid/side coding the rotation angle is fixed and is not transmitted.

IS coding is some simplification where you leaf out the difference signal.

Thanks alot for the explanations. I want to apologize for making obviously wrong statements about MP3 IS.
SebastianG
QUOTE(Ivan Dimkovic @ May 30 2004, 10:39 AM)
And I also disagree with the claims that IS could bring good quality - there are lots of cases with stereo configuration impossible to code properly with IS, because IS saves only ILD information (level difference)  and not ITD (time difference) and inter-channel cross corellation. 
[...]
Smart psychoacoustic would be able to disable IS for such frames, but @128 kb/s there woud be no need for lossy bit savings,  same goes for PNS (in AAC) more or less- we did a lot of tests with PNS @128 kb/s and in most cases is pretty much useless, or degrades the quality.

Thanks for your reply.

Let's compare ogg to aac. Monty said once, lossless coupling would be like wasting space for 128kbps and 160kbps modes. You guys keep telling me IS is inappropriate for that bitrates. Sure, this mapping is irreversible and only preserves the channel's energy levels - not all their phase relations. But AFAIK phase relations are not that important to us at above 10 kHz because the wavelength is already very short. So if an advanced encoder would make use of this psychoacoustic effect properly by using IS this could save some space and allows to use smaller scalefactors to improve the SNR.

AFAIK IS can be switched on/off for each scalefactor band (AAC). Another cool thing: IS can be done in-phase and out-of-phase. How about the following sheme for scalefactor bands above 10 kHz:

- treat MDCT samples for a scalefactor band as multidimensional vector
- compute the cosine of the angle between L and R by cos_a := \frac{<L,R>}{||L|| ||R||}
- use in-phase IS for cos_a > 0.5
- use out-of-phase IS for cos_a < -0.5

These thresholds (in this case 0.5) could be chosen depending on the quality-preset and frequency area.

Well, I don't know, if Intensity Stereo is or is not appropriate for 128 kbps. But I do know that Vorbis makes use of it and "won" the listening test.

edit: corrected cos_a correlation formula

bye,
Sebastian
enry2k
Suprisely! How it is possible Lame mp3 better than Itunes aac?
are previous tests wrong?
Lyx
QUOTE(enry2k @ May 30 2004, 10:19 PM)
Suprisely! How it is possible Lame mp3 better than Itunes aac?
are previous tests wrong?

the encoder-version + settings used for lame mp3 during this test for sure weren't the same as in previous large-scale tests.

Lame is not = Lame. In recent versions, lame seems to have made good progress in improving at mid-bitrates. You can see this in the lame 3.90.3 vs. lame 3.96 thread.

- Lyx
rjamorim
QUOTE(Lyx @ May 30 2004, 07:56 PM)
QUOTE(enry2k @ May 30 2004, 10:19 PM)
Suprisely! How it is possible Lame mp3 better than Itunes aac?
are previous tests wrong?

the encoder-version + settings used for lame mp3 during this test for sure weren't the same as in previous large-scale tests.

It's also worth mentioning that Lame is not better in the test. It's officially tied, with a tendency to be a little worse.
neomoe
QUOTE
ok, so it looks like the vbr contenders did very well and itunes's cbr held its on. how safe would it be to assume that using vbr with AAC (for instance the most recent FAAC with FB2K) would be a contender?


okay, it has been proven that iTunes AAC is better then FAAC for instance.
http://www.rjamorim.com/test/aac128test/results.html

but how sure could one say how good iTunes AAC would be if it had VBR implemented?

and, one silly question:

how sure is it how good/bad one encoder would perform at higher bitrates,e.g. 160kbps?
i mean is it right to say that vorbis for instance can reach transparent level at a lower bitrate then MPC or AAC?
Liquid_Predator
QUOTE
okay, it has been proven that iTunes AAC is better then FAAC for instance.
http://www.rjamorim.com/test/aac128test/results.html


This is an rather old test, here are newer test results: http://www.rjamorim.com/test/aac128v2/results.html but iTunes is still the winner.

QUOTE
how sure is it how good/bad one encoder would perform at higher bitrates,e.g. 160kbps?
i mean is it right to say that vorbis for instance can reach transparent level at a lower bitrate then MPC or AAC?


You canīt extrapolate the results! WMA is generally better than MP3 at 64kb/s, but at 128kb/s MP3 is better.
neomoe
QUOTE
You canīt extrapolate the results! WMA is generally better than MP3 at 64kb/s, but at 128kb/s MP3 is better.


okay, thank you, but
QUOTE
i mean is it right to say that vorbis for instance can reach transparent level at a lower bitrate then MPC or AAC?


i assume we would need a listening test at this high bitrate, right?
Liquid_Predator
QUOTE(neomoe @ May 31 2004, 09:53 AM)
i assume we would need a listening test at this high bitrate, right?

Indeed
neomoe
even if i accounted harashin's private listening test i wouldn't be able to do so?
see, we know auTuV is very good at 128kbps and at ~200kbps (at least for harashin).
now we should be able to say how Vorbis would perform at around 160kbps shouldn't we?

and what about my other question:
QUOTE
but how sure could one say how good iTunes AAC would be if it had VBR implemented?
SirGrey
My post will not be very informative tongue.gif
QUOTE
but how sure could one say how good iTunes AAC would be if it had VBR implemented?

No one knows.
As example - Fhg mp3 encoders on low bitrates tends to be better with CBR, than VBR (search forum, if you wish to have more info) but no one would tell (I hope smile.gif ) VBR is loosely implemented there...
VBR have it's own problems, as discussed in this thread...
EDIT: grammar
neomoe
oh gosh!

trying is superior to studying - isn't it?
(well, I tried to translate a german saying)

okay.now for me it IS like this:

aoTuv is superior to iTunes AAC even at 160kbps and above and second personal truth is iTunes AAC would be better with VBR implemented presumed that it is decent implemented. harrrharrr! biggrin.gif
Jojo
I wonder how ATRAC3plus performs...it's a pity that it wasn't included in the test sad.gif
guruboolez
It's currently not possible to oppose atrac3+ to other encoders at 128 kbps. For a simple reason: there's no 130 kbps mode with current and public atrac3+ encoder. Only low bitrate (48 & 64 kbps) and high bitrate (256 kbps) setting.
Jojo
QUOTE(guruboolez @ Jun 6 2004, 09:41 AM)
It's currently not possible to oppose atrac3+ to other encoders at 128 kbps. For a simple reason: there's no 130 kbps mode with current and public atrac3+ encoder. Only low bitrate (48 & 64 kbps) and high bitrate (256 kbps) setting.

interesting...so I wonder what bitrate is used in Sony's music store...I read they use ATRAC3plus...don't tell me they use 64kbps unsure.gif
Pio2001
Hello, I'm back from holydays...

I just wanted to point out an odd thing that happened to me during this test : contrary to the common way of things, I could only ABX the 7th sample (gone) with speakers, and not with headphones ! (Dynaudio Gemini speakers vs Sennheiser HD600 headphones).


A picture of me ABXing "gone" : listeningtest.jpg wink.gif
To avoid any background noise, the picture is video-projected on a screen in front of me, and the computer is in the next room. 5 meters mouse, keyboard, SPDIF, and DVI cables.

Mhhh, actually, I must admit that my speaker setting is often refered by my father as "the biggest headphones I've even seen".
Lyx
QUOTE(Pio2001 @ Jun 7 2004, 07:58 PM)
A picture of me ABXing "gone" : listeningtest.jpg wink.gif
To avoid any background noise, the picture is video-projected on a screen in front of me, and the computer is in the next room. 5 meters mouse, keyboard, SPDIF, and DVI cables.

Mhhh, actually, I must admit that my speaker setting is often refered by my father as "the biggest headphones I've even seen".

blink.gif blink.gif blink.gif
SirGrey
QUOTE
A picture of me ABXing "gone" : listeningtest.jpg 

Nice settings laugh.gif
rjamorim
QUOTE(Pio2001 @ Jun 7 2004, 04:58 PM)
contrary to the common way of things, I could only ABX the 7th sample (gone) with speakers, and not with headphones ! (Dynaudio Gemini speakers vs Sennheiser HD600 headphones).

Impressive!

Can you describe the artifact you can only detect with your speakers, and not your headphones?
Pio2001
From the "user comment" section, matching the samples ID with the filenames, then the filenames with the codecs, I got

Lame : 5/5

AAC : 4/5 : Ringing on the first guitar note.
ABX 13/16 from 8.3 to 22 s

Musepack : 5/5

Atrac : 4/5 More treble from 17s
ABX 15/16 from 17.17 to 24.21 s

Vorbis : 5/5

WMA : 3.5/5 : More treble when the guitar comes in
ABX 13/16, from 8 to 26.6 s


When I say "more treble", it is just a subjective impression. It does not necessarily mean the the treble level is higher, but rather the the treble sounds brighter.

Edit : these speaker have not a linear response. The tweeter is set 1.5 db louder than the woofer. The crossover frequency is 2 kHz. I usually cancel this with Foobar convolver, but with ABCHR I couldn't.

Edit2 : the fact that they are rather to the sides of the listener than in front of him makes treble even harsher (this is the case with audio sources when they are to the side of the listener). But that's the way I often listen to music. I did not move the speakers especially for the test.
phong
I thought it was quite impressive that you look so much like your avatar.
Cutter
Hello!

I would like to know if the poor results of lame aren't due to its popularuty. The more you listen to a format, the easier it is for you to recognize its artefacts, right? Maybe the people who participated where more able to recognize MP3 than other formats. What do you think?
rjamorim
QUOTE(Cutter @ Oct 16 2004, 05:46 PM)
Hello!

I would like to know if the poor results of lame aren't due to its popularuty. The more you listen to a format, the easier it is for you to recognize its artefacts, right? Maybe the people who participated where more able to recognize MP3 than other formats. What do you think?
*



Well, I wouldn't consider Lame's resulst poor. It ended up tied with AAC - that is supposed to sound much better!

Now, for your concern: I think the artifacts that happen on lossy music are pretty much the same across all formats. So, if you learn to distinguish pre-echo, smearing or stereo collapse on MP3, you will probably detect these same artifacts in AAC, Vorbis, MPC... if they are there.

That's just a supposition though, maybe MP3's popularity did affect its results in some way...
guruboolez
QUOTE(Cutter @ Oct 16 2004, 09:46 PM)
The more you listen to a format, the easier it is for you to recognize its artefacts, right?
*



I don't think so. For exemple, tons of people are listening to vorbis for years, and still can't detect anything wrong in stereo image or timbre coarseness...
To recognize with ease artifacts, you probably need to track them. It's an active attitude, opposed to the daily listening, which is passive.

On the other side, artifacts don't really differ from one encoder to another. mp3, aac, mpc, wma, atrac3... are really close each others. SBR (mp3pro, he-aac) introduce specific problems in addition to the previous one; vorbis is also slightly different (see above); hybrid encoders produce noise. But most artifacts (pre-echo, warbling, chirping, metallic sound...) are common to all transform encoders.
Cutter
QUOTE(guruboolez @ Oct 17 2004, 01:25 AM)
QUOTE(Cutter @ Oct 16 2004, 09:46 PM)
The more you listen to a format, the easier it is for you to recognize its artefacts, right?
*



I don't think so. For exemple, tons of people are listening to vorbis for years, and still can't detect anything wrong in stereo image or timbre coarseness...
To recognize with ease artifacts, you probably need to track them. It's an active attitude, opposed to the daily listening, which is passive.

On the other side, artifacts don't really differ from one encoder to another. mp3, aac, mpc, wma, atrac3... are really close each others. SBR (mp3pro, he-aac) introduce specific problems in addition to the previous one; vorbis is also slightly different (see above); hybrid encoders produce noise. But most artifacts (pre-echo, warbling, chirping, metallic sound...) are common to all transform encoders.
*


That's what most of the people who participated to this test did, I guess. We're not talking about average music listeners here, but people who have "trained ears". tongue.gif
guruboolez
QUOTE(Cutter @ Oct 17 2004, 01:57 AM)
We're not talking about average music listeners here, but people who have "trained ears". tongue.gif
*



Sorry, but when you said "The more you listen to a format, the easier it is for you to recognize its artefacts, right?" I thought you made a general assumption.

To answer to this: many results were sent by people which are not trained. Take a look to the overall notation: wma@128 is "near transparent" according to the test. It can't be true for someone having a small experience in artifacts hunting.
Cutter
Ok. Thank you both for your answers.
moi
QUOTE(harashin @ May 24 2004, 04:59 AM)
QUOTE(guruboolez @ May 24 2004, 09:54 PM)
Roberto> what software did you used to obtain wma9 files? Is it VBR-2 pass 128 kbps? What decoder? I've tried to reproduce the same wavform with different settings, and I wasn't able to do it.

I already asked him about this.
http://www.hydrogenaudio.org/forums/index....ndpost&p=210584

EDIT:It's certainly Bitrate VBR 128kbps, 44kHz, stereo VBR 1pass.


I don't get that. From what I have seen, for 1 pass WMA VBR you cannot specify a bit rate at all, only the "quality settings"such as 50, 75, 90, etc.

With two pass WMA VBR you specify an average bit rate.

You state it was WMA one pass 128 kbps VBR. How could that be?
Latexxx
Windows media encoder allows you to do 1-pass bitrate vbr. It is somekind of ABR.
moi
QUOTE(Latexxx @ Oct 24 2004, 12:42 AM)
Windows media encoder allows you to do 1-pass bitrate vbr. It is somekind of ABR.
*


I guess I've only tried WMA VBR using DBPoweramp. For 1 pass VBR, there are no bit rate settings, just the quality settings like 50, 75, 90, etc. With the two pass VBR, you set a target bit rate. (I guess they figure that with two passes they can come closer to a target bit rate, but not with one pass.)

Surprised it's different in WME. It doesn't have the "quality settings"?
Latexxx
It has them also.
rjamorim
I used two-pass VBR (Bitrate VBR)
1kyle
In some cases tests like this are not very subjective -- for example an Opera lover will probably cringe at listening to tracks of "House Music" played with ANY CODEC and probably vice versa. It's almost impossible to find music that everyone likes which rather invalidates some of the test findings.

I've tried the new HI-MD minidisc units from Sony particularly the NH-1 and I'm pretty fussy with my music. The HI-SP (Atrac3 +) format seems to me certainly for music on the move or when wearing some decent cans as good as CD (also CD's have pretty varying quality as well).

For Classical Music which on the whole has a higher dynamic range than most rock type music then MP3's can sound pretty hopeless. Acoustic instruments also tend to sound somewhat "quirky" on MP3's as well whereas the more "electronic sound" of dance music tends to hide some of the more obvious problems with MP3's especially at the lower bit rates.

My main problem with ATRAC3 + is some of the really STUPID DRM problems which make copying and distributing YOUR OWN MUSIC a real pain.

-K



Gabriel
You are right, placebo effect is way more suggestive, and often works quite well.
Kblood
QUOTE(Gabriel @ Jan 28 2005, 10:54 AM)
You are right, placebo effect is way more suggestive, and often works quite well.
*


biggrin.gif
Pio2001
QUOTE(1kyle @ Jan 28 2005, 10:50 AM)
Acoustic instruments also tend to sound somewhat "quirky" on MP3's as well whereas the more "electronic sound" of dance music tends to hide some of the more obvious problems with MP3's  especially at the lower bit rates.
*



My experience is the exact opposite (at high bitrates at least). Guruboolez' harpsichord and orchestral samples sound transparent to me, while the electronic boxes of Amnesia, Fsol, Autechre, Spahm, Astral, Transwave etc. sound ugly to me once encoded.
jmitch
Anyone who is basing their ideas off this listening test is taking a bit of a risk. This test is very good and I commend rjamorim for taking his time to conduct it, however I don't believe there was nearly enough testers to validate any accurate data, and therefore come to any valid conclusions. I think this test should be redone and spread much more widely over the internet audio boards, not just this one. Then we could formulate some accurate conclusions. In my opinion, there is just not enough data to do that.
nyarlathotep
QUOTE(jmitch @ Feb 15 2005, 04:16 PM)
I think this test should be redone and spread much more widely over the internet audio boards, not just this one.

Yes I think that's right to say that not enough people did participate.

Still, IIRC there were anouncements made on other boards about this test. I personnaly made one there (on a French popular board, but not as specialized as HA is about audiocoding):
http://forum.hardware.fr/forum2.php?config...sh=0&subcat=131

To tell the truth of what I think: not so many people really want to spend some time testing different samples. Not even mentionning those who don't know what an ABX test is and claim that everything is just like "night and day" or so.
rjamorim
Formal listening tests conduced by the ITU and EBU sometimes use as few as 9-10 listeners. Trained listeners, of course, but still, it's quite few compared to the amount of people that participated in some of the samples of this test...
Busemann
QUOTE(rjamorim @ Feb 15 2005, 06:29 AM)
Formal listening tests conduced by the ITU and EBU sometimes use as few as 9-10 listeners. Trained listeners, of course, but still, it's quite few compared to the amount of people that participated in some of the samples of this test...
*



They also use reference systems, though. The downside to these internet tests is the wide array of equipment, which means the transparency threshold is often quite low. But then again, it reflects the real world nicely.
ff123
QUOTE(jmitch @ Feb 15 2005, 06:16 AM)
Anyone who is basing their ideas off this listening test is taking a bit of a risk. This test is very good and I commend rjamorim for taking his time to conduct it, however I don't believe there was nearly enough testers to validate any accurate data, and therefore come to any valid conclusions. I think this test should be redone and spread much more widely over the internet audio boards, not just this one. Then we could formulate some accurate conclusions. In my opinion, there is just not enough data to do that.
*



I think the conclusions reached were quite valid and accurate -- for the goup of people who participated and for the samples listened to. That was the whole point of doing a statistical analysis.

If one wants to generalize to a larger group of people or a different set of samples, yes there is a bit of a risk, but the results are probably not far off the mark. A different sample set would probably get you the most different results. And of course trying to apply group results to a particular individual is quite a bit more risky. I would say that the variations are bigger from individual to individual than from one group to another.

ff123
jmitch
Yes, there are many factors and variables that ruin the validity of the test. One being, which you named, the audio equipment being used to do the testing. Most users have shit audio equipment, therefore their results are pretty poor and innacurate. Secondly, many people, like you said, don't even know what the hell ABXing is, so you can tell by that that they don't know much about audio. Their ears, and/or listening skills probably suck. This would dramatically alter the results of the test.

Anyways, the test is better than no test. It gives us a reasonable idea, but not accurate enough, in my opinion, to really make any conclusive judgements.

I would be interested in gathering a group of good listeners that have quality equipment. I think we should have enough here. I myself have Etymotic Research ER-4s, which are basically the best you can get as far as equipment goes. smile.gif
rjamorim
QUOTE(jmitch @ Feb 16 2005, 11:31 AM)
I would be interested in gathering a group of good listeners that have quality equipment.
*



And, by doing that, you would conduce a test that would only have meaning to people with good listening and quality equipment smile.gif

By accepting everyone and all equipment on my test, I got much closer to the average user than if I only targeted it at golden ears with headphones that cost more than 100 dollars.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2008 Invision Power Services, Inc.