Very high bitrate listening test (~320-400kbps), Pre-test discussion & how-to Options

[memomai]

would it be possible to do such a test?

I'm very interested in how different codecs behave in very high bitrate encoding. I'd like to compare them in encoding speed, problem samples, artefacts and transcoding efficiency (which will be a problem for all encoders).

For the test I'll need your help. It's my first one. Please give advice to me how to do this test in the best way. Of course you can also add your interests, opinions and other tipps. So at the end I or we will do the test then.

I'd like to test these ones:

- MP3 Lame 3.98b3 @ 320 kbps CBR (maybe for the low anchor?)
- MP2 twoLame @ 384 kbps, newest release
- NeroAAC LC-AAC @ 384/400 (-Q0.90, -Q1.0), newest release; 400 because I'd like to see if there ARE differences
- MPC @ ~350 kbps (--quality 10), newest release
- Ogg aoTuV @ 384 kbps (-q9.2, maybe?)
- WavPack lossy @ 384 kbps (-b384), newest release
- WavPack lossy @ 448 kbps (-b448), newest release (maybe high anchor?)
NOW ADDED:
- WMA 9.2 Standard @ 384 kbps
- OptimFROG Dual Stream lossy @ ~384

The ones which should be tested can be changed of course or someone being added.

Thanks for replies

This post has been edited by memomai: May 30 2007, 00:14

[pest]

I'm very interested in how different codecs behave in very high bitrate encoding. I'd like to compare them in encoding speed, problem samples, artefacts and transcoding efficiency (which will be a problem for all encoders).

If many people can't distinguish a 128kbps encoding from the original there's a really low to no reason to do such test for spotting artifacts in natural music.

[DigitalDictator]

Lame @ 320 kbps as a low anchor? How does that add up when that is the most transparent setting Lame can offer? An anchor should, afaik, be a reference point and clearly (audibly) lower in quality than the rest. Correct me if I'm wrong.

edit: grammar

This post has been edited by DigitalDictator: May 29 2007, 15:14

[shadowking]

From a perceptual only point of view such measure isn't needed as most modern coders do this at 160~250k.

In transcoding ogg, aac and mpc should do well at > 320k, but I would favour wavpack. My feeling is that such a test is well beyond most people, but it can be done.

A normal abx is test also not really meaningful as you will have to test only known problem samples which could also be hard to isolate. Given different codec behaviour, the results might not be meaningful.

If transparency is a life and death concern then I suggest lossless encoding or: wavpack HQ at 512k or even 550k as the 'end-of-all-armaggedon' setting. You will save 50% on loud music and even 10~25% on classical. This is transparency for listening, edting and transcoding.

[mickywicky]

I'm very interested in how different codecs behave in very high bitrate encoding. I'd like to compare them in encoding speed, problem samples, artefacts and transcoding efficiency (which will be a problem for all encoders).

If many people can't distinguish a 128kbps encoding from the original there's a really low to no reason to do such test for spotting artifacts in natural music.

I'll second that. Above 200 Kbps most if not all encoders should deliver transparent music except for very problematic samples.
I'm pretty good at ABX'ing "normal" samples that I know well, but lameMP3 goes fully transparent to me around 220-240 Kbps.

What you could compare is the encoding speed for identical bitrates, or the bitrate at which you cannot find any more problem samples. Not quite your average listening test then.

MW

[shadowking]

Transparent can also be extended to transcoding , editing, postprocessing, DSP etc. This is much different than a normal listening test and is also a shady area for lossy encoders. For such conditions I would favour the hybrid encoders based on some testing. I think at such high bitrates mpc, ogg, aac could also be competitive.

[Squeller]

Well, 400 kbps+ is what I often achieve when encoding to wavpack or tak lossless...

[xmixahlx]

Well, 400 kbps+ is what I often achieve when encoding to wavpack or tak lossless...

emphasis on that "+" there...

musepack --insane FTW!

i agree with the "200+ kbps on any current format should be sufficient for real stuff" opinions here.


later

[bubka]

you are not going to get any statistical differences in your results

This post has been edited by bubka: May 29 2007, 17:59

[LANjackal]

I'd suggest you scrap this test idea...

1 - You're missing WMA 9.2 Standard, which can encode at the bitrate range you're interested in

2 - At those high bitrates, discerning differences is nigh impossible for everyone, no matter how good their hearing. But then again, perhaps the best way to get this point across is to have you do the test and see for yourself

[memomai]

Transparent can also be extended to transcoding , editing, postprocessing, DSP etc. This is much different than a normal listening test and is also a shady area for lossy encoders. For such conditions I would favour the hybrid encoders based on some testing. I think at such high bitrates mpc, ogg, aac could also be competitive.

I only know WavPack is a hybrid encoder. Are there others?

I'd suggest you scrap this test idea...

1 - You're missing WMA 9.2 Standard, which can encode at the bitrate range you're interested in

2 - At those high bitrates, discerning differences is nigh impossible for everyone, no matter how good their hearing. But then again, perhaps the best way to get this point across is to have you do the test and see for yourself wink.gif

OK, WMA 9.2 could also be added (sorry I forgot this format because I never used it).
The main topic won't be transparency, but:

I'd like to compare them in encoding speed, problem samples, artefacts and transcoding efficiency (which will be a problem for all encoders).

Lame @ 320 kbps as a low anchor? How does that add up when that is the most transparent setting Lame can offer? An anchor should, afaik, be a reference point and clearly (audibly) lower in quality than the rest. Correct me if I'm wrong.

edit: grammar

MP2 is supposed to perform better than MP3 at 256 kbit/s and above. I'd like to see how it behaves and also compare it to MPC.

This test could also say then if MPC is in quality the format for audiophiles ^^. Some say yes, it is the best format for quality transparency and most artefact safe in lossy encoders, some say no it isn't the best one for that area. So MPC could be considered as absolutely useless nowadays then and can be said to be dead, OR it can stay as the standard format for highest given quality.

What you could compare is the encoding speed for identical bitrates, or the bitrate at which you cannot find any more problem samples. Not quite your average listening test then.

encoding speed is already within. Finding the bitrate where it isn't anymore a problem sample is a interesting point, I'll keep that in mind!

EDIT: spelling

This post has been edited by memomai: May 29 2007, 19:45

[DigitalDictator]

I think you missed my point regarding the low anchor. You're suggesting a low anchor that is (probably) indistinguishable from the rest.

[halb27]

I welcome the idea cause I'd like to see how for instance mp2 at extremely high bitrate compares to mpc or wavPack lossy or Optimfrog Dualstream lossy (quality controlled).

But I also see practical problems how to do the test. I don't have a good idea how to go ahead.

[odyssey]

Stupid idea. Why would anyone use ~400 kbps for lossy compression when you can get lossless compression at ~600 kbps? And especially audiophiles would never use lossy anyway...

[bryant]

Stupid idea. Why would anyone use ~400 kbps for lossy compression when you can get lossless compression at ~600 kbps? And especially audiophiles would never use lossy anyway...

I agree that when you can get lossless at around 500-600 kbps this doesn't make any sense, and this is common with classical music and some jazz.

However, a lot of popular music compresses lossless at around 1100 kbps. In those cases even 500 kbps would be a big savings.

[halb27]

With classical and other music with a lot of rather quiet scenes or few instruments/voices in it lossless compression can get into the 600 kbps range.
But in these cases you usually get an excellent quality with the lossy variants in the 300 kbps range - half the file size of lossless.

Pop/Rock music is compressed losslessly with ~ 900 kbps, not ~ 600 kbps. ~ 400 kbps is a lot less than that.

[Porcupine]

odyssey, lossless compression requires 1000 kbps, not 600 kbps. It depends what kind of music you listen to. Most music requires 950 kbps. I have some real songs that require 1200 kbps lossless. Some types of music will compress at 600 kbps, but this can't be stated to be the general case.

Most people who say that high bitrate tests are foolish, and that one should just go lossless if they demand perfect quality, are not correct in my opinion. I recently investigated this thoroughly (and am still investigating, but not as much) and learned many interesting things.

High bitrate tests are still very fruitful for any lossy codec. The key thing is to understand that for the majority of the time, stuff is indistinguishable from the original at bitrates such as 200 kbps. But for certain types of signals, differences from the original can still be heard even at very high bitrate.

For lossless-based lossy encoders such as WavPack Hybrid...the weakness is a very tonal signal without a lot of "busyness" to the sound. Encoding a 100% amplitude 17 kHz stereo sine wave causes a disaster for WavPack that is not transparent until 1000 kbps is reached, and is not even acceptable until 800 kbps. In reality you should never get such a signal though, but this is a good way to do a real listening test and figure out the strengths and weaknesses of the codec.

For transform-based formats such as MP3...the biggest weakness is a noiselike transient signal. An additional helpful requirement is that there be mostly silence before the transients. The 'Eig' problem sample that halb27 kindly sent me earlier is the best example I know of. The machinegun sounds in 'Eig' are very close to being delta function pulses. The delta function pulse is the transform-based encoder's version of a disaster. In reality you'd rarely or never encounter such a signal, but it should be noted that 'Eig' seems to come from a real song, which is all the more dangerous. (I'm certain that a real song can cause a disaster for WavPack too but I don't happen to have any which fits the requirement, though I'm sure many exist.)

Eig is still far from transparent at 320 kbps CBR with most versions of LAME. With some versions of LAME (3.92 is the best I know of, but I never tried 3.97 and theoretically it should be good too) it is close though. The reason Eig is close to transparent in those versions is that they leave the bit reservoir on even at 320 kbps CBR, so in actuality Eig is being encoded at 480 kbps in those versions of LAME.

If you test encode Eig as a freeformat 480 kbps MP3 file with other versions of LAME, the quality improves amazingly also, to near-transparency. Probably it is still not transparent but I did not test further as I could not get anything higher than 480 kbps to decode properly. Probably to be truly transparent, I am guessing Eig might need around 800 kbps MP3...similar to the disaster case for WavPack (which by the way encodes Eig transparently at 200 kbps to me).

This post has been edited by Porcupine: May 29 2007, 23:40

[memomai]

I think you missed my point regarding the low anchor. You're suggesting a low anchor that is (probably) indistinguishable from the rest.

Oh, I see now, sorry. But that would also be interesting to see how the maximum setting of MP3 compares to the others.

I found another possible solution for low anchor, maybe the ADPCM codec which is already implemented in MS Windows (gives bitrate of 350 kbps)?

Stupid idea. Why would anyone use ~400 kbps for lossy compression when you can get lossless compression at ~600 kbps? And especially audiophiles would never use lossy anyway...

Wrong. First, like halb27 and porcupine already mentioned, a lossless backup takes in average about 900 kbps, sometimes more sometimes less. Second, there are people (including me ^^) who prefer a lossy choice where you reach archiving quality, meaning from the source converted files where artefacts / problem samples can't be detected anymore. And by reaching that quality status with a filesize that's just the half or even less than a lossless backup is for many people still a more attractive option. This sort of audiophiles exists too, and that's not just a small group, believe me.

This post has been edited by memomai: May 30 2007, 00:10

[HisInfernalMajes...]

Do y'all think it would be a good idea to try LAME at 320+ kbps using the freeformat command?

[Curtor]

Oh, I see now, sorry. But that would also be interesting to see how the maximum setting of MP3 compares to the others. I found another possible solution for low anchor, maybe the ADPCM codec which is already implemented in MS Windows (gives bitrate of 350 kbps)?

As everyone on the board has already pointed out and you seem to not be listening to: The codecs are transparent by 224kb/s at the most. All the other more meaningful tests have shown that transparency typically happens well below that. You could not hold your listening test because it's way past the point where people can tell a difference. There is no point to picking a low anchor because everything will sound the same as the original wave... including your anchor! A 320-400kb/s listening test has no practical purpose in the world as it would not be able to establish anything, therefore there's nothing to be learned from it.

[Porcupine]

No, that is totally wrong.

As I wrote in my previous post, there are still rare samples that are quite untransparent at 320 kbps and even higher, with all these encoders. It is just that they are rare, you must search for a specific type of signal (which I already said what they are).

The problem with trying to compare the different encoders is that there are 2 groups...the transform-based encoders (mp3, ogg, etc) and the lossless-based lossy encoders (WavPack hybrid, Optimfrog)...and the worst-case problem samples that require 320+ kbps are totally different between these groups. Therefore, how do you compare them with each other in a practical listening test?

The only thing I know of to do is test the worst samples with mp3/etc vs the worst (and different) samples with WavPack/etc, and consider which one you are more likely to encounter in the kind of music you listen to, and consider which one sounds "worse" to you (but it will naturally be highly subjective).

[Light-Fire]

It would be obviously a meaningless test. The best it is to concentrate test and development efforts to lower bit rates. Trying to make them transparent. Testing high bit rate encoders for transparency (once they already achieved it in lower rates) is illogical.

This post has been edited by Light-Fire: May 30 2007, 01:00

[memomai]

As everyone on the board has already pointed out and you seem to not be listening to: The codecs are transparent by 224kb/s at the most. All the other more meaningful tests have shown that transparency typically happens well below that. You could not hold your listening test because it's way past the point where people can tell a difference. There is no point to picking a low anchor because everything will sound the same as the original wave... including your anchor! A 320-400kb/s listening test has no practical purpose in the world as it would not be able to establish anything, therefore there's nothing to be learned from it.

I think you don't get the point here. Question of transparency is not the aim of the test. Please read above which points I want to focus on.

As I wrote in my previous post, there are still rare samples that are quite untransparent at 320 kbps and even higher, with all these encoders. It is just that they are rare, you must search for a specific type of signal (which I already said what they are).

It wouldbe very helpful if you could give me a link for these problem samples.

This post has been edited by memomai: May 30 2007, 01:03

[2Bdecided]

If you were going to do a straight listening test at 320kbps+, it would be a complete and utter waste of time. But that's not what you're talking about.

If you're going to do a transcoding test from high bitrates to "low" bitrates, that's fine. It could work, as Guruboolez has demonstrated in the past (though he is a very good listener).

If you're interested in the samples that break various encoders, then I think a listening test (in the HA tradition) makes less sense at this stage. It's more a case of collecting them, counting them, and (as mentioned) trying to figure out what makes them special, and how often this will occur in your or my music collection.

After you've done this, it might make some sense to do a listening test on these problem samples.

I think the most interesting part is collecting problem samples. This happens on HA already, in a disorganised way. There is no single "problem samples" list - you have to search them out individually. The uploads forum is a fair place to start looking.

Cheers,
David.

[kennedyb4]

It seems to me that there are more pressing issues at the moment.

I would like to see if nero AAC has outstripped Itunes' encoder yet at some flash device useable bitrate like 128 or 96.Or perhaps Ogg still rules at these bitrates?