Hello!

After assisting like many of you on the controversy of guruboolez' tests concerning the Nero AAC encoders and having seen many suggestions towards preparing a listening test around 48kbps - in particular with the different HE encoders, I decided to take matters into my own hands.
I currently have 8 samples testing for tonality, transients and stereo separation. These are generally classical music, but also include electronic music, 80s pop, and I might add a jazz recording for ambience reproduction.
As for encoders, I'm stumped.

Should I also include an old version of nero (old being relative) as to see the progress in a general evaluation ? At this bitrate, many encoders is not really a problem, is it? For now, I was thinking :

-Nero AAC Tape / PNS Enabled / HE / High (Should I enable hint track?)
-Ogg Vorbis, latest AoTuV, q -2 to -4 (whatever's approximately the same bitrates ~42-53)
-Lame ABR 48 (any tunings suggested?)
-mp3pro, probably, at any close bitrate I can achieve (adobe audition's encoder)
-Itunes AAC (HE if available)
-CT encoder (winamp 5.12) 48kbps, HE.

If I should use an older nero as well, where can I find it?

Also, could anyone help me prepare the ABC/HR logs? (eg, Sebastian Mares)
Thanks in advance,
Tristan.

Is it a collective listening test or a personal exercise?

Collective

Isn't it a bit premature to start a new pretest discussion thread, two weeks before the end of the current 128 test?

If it is a collective test, may I ask for delaying it at least a week or so, because we could finalize new LC-AAC encoder being actually used with the SBR core, it would be also put in the next Nero Release along with the LC improvements currently being tested @128 kbps.

But in general I also think it is a bad idea to conduct few tests at the same time.

I wasn't intending to start the test until the beginning of january, but, sure, I can wait.

I'll copy the contents of this thread into a notepad and set it aside.

As a digressional note, for those of you on linux.. emacs or vi?

Well, I can't see the problem of starting the discussion now. It's not like the discussion has to be a huge amount of work for anyone, unless it gets out of hand like last time...

I think the whole thing is too rushed. Personally, I would check what people want more and then start the "advanced" discussion about encoders, samples, settings...

• 128 kbps extension test featuring ATRAC3, MusePack and WMA Standard (later might be problematic because of the settings)
• Low bitrate (32/48/64 kbps) multiformat test
• Low bitrate (32/48/64 kbps) AAC test

I would also wait until the end of January and check if Apple has an HE-AACv2 encoder ready.

We all know how that would end... Just be an enlightened dictator Shade, and use your wisdom.

The result of the "Fifth competitor..." poll was pretty clear, so I see nothing wrong in running a poll again.

Edit 1: Also, so far, I see you have six encoders in mind. If you include an older version of Nero, that makes seven - without any anchor. That is too much, so I would sort out some of the encoders leaving only five and a high anchor - maybe the 128 kbps winner?

And sure, I can help you set up the ABC/HR configuration files. Do you have ICQ or another IM client, Shade[ST]?

Edit 2: Is there any major difference between the mp3PRO encoder included in Nero and the one from Audition?

Regarding the encoders again - personally, I would exclude LAME (not tuned for such bitrates - even developers recommend FhG for bitrates lower than 128 kbps because FhG also uses IS) and mp3PRO (no development since last test AFAIK and it is being replaced by HE-AAC anyways). On the other hand, that leaves you with only Ogg Vorbis as format other than AAC. Anyone knows how WMA Standard 9.1 is supposed to perform at this bitrate?

Yes, the poll was great! I was more thinking about the discussion that went on for eternity. (Which I hardly need tell you is a heck of a long time, to quote Rowan Atkinson aka The Devil aka Toby.)

With all respect, Shade[ST], this is a really bad time to start this. The 128kbps test is running, and we should concentrate on that and not confuse people with another listening test talk. And while I understand your enthusiasm, I think, now that we have Sebastian running the test, you should show some respect towards him. I'm sure he wants to run consequent tests after this one, so it's a bit inept to "mess with his business" now.

I was thinking maybe about making LAME the low anchor.. lame @ 32? And maybe set ogg vorbis around a bitrate of 100, or something of the like, to make it perceptible to some, but not annoying -- Personally, I was quickly irritated in the 128 test by not being able to discern anything.

I don't mind adding WMA std to this test, either. The samples are pretty much chosen, I think - I've spoken to guru about this, and I'll try to have sebastian mares tell me if my samples are ok or not.

I don't think many people use itunes to encode at such a low bitrate, but then again, who knows? some cell phones come with itunes now. I'm arguing with myself as to keeping it or not.

I sort of insist on this being a 48kbps test only : I don't like the number 32, it's ugly (dictator, says naylor83?), and IMO 64 kbps will still be too hard to make out.

This needs to be a quick-and-dirty test, making it easy to pick up a few samples in 15 minutes of spare time and send in results. I'll also set up a gmail for recieving test results. I could give out the password later to anyone wishing to organise another listening test, or to a HA admin or whatnot, if results are wished to be kept.

For now, that's all I can think of. If anyone wants to hear / comment on samples I chose, you can PM me or leave a message on this thread (preference for PM), and contact me on MSN messenger (or, by email.)

Comments are welcome,
Peace,
Tristan.

I'd personally drop mp3pro because its marginal presence is being overshadowed by he aac.

FhG instead of lame would be a rational choice because most net radios use FhG AFAIK.

Some years ago Microsoft sais that wma std @ 48 is almost cd quality. What a pity if we can't even prove that there are better codecs at 48 kbps than wma thus wma should be definately included.

I can't really see any point in including Apple's aac unless they manage add support for sbr and ps before the test starts. So my list would basicly be the following.

1. mp3 (fraunhofer?)
2. ogg vorbis (aotuv?)
3. wma standard (2-pass cbr or whatever their superfancy preset in wmenc is)
4. Nero he aac v.2
5. CT aac+ v.2
6. high anchor (mp3 @ 128?)

No point in using a low anchor such as gsm efr because the artifacts will be clearly present and a high anchor will give enough context.

I personally believe you should give listeners at least one month of rest until you start a new test.

Any reason in particular? The same people don't have to be participating, as long as a reasonable amount of members do..

don't forget about 3GP reference encoder:
http://forum.dbpoweramp.com/showthread.php?t=7107

Yeah. And a 48 kbps test would be much easier, if you ask me...

There are two things to take care when using it:

- It works only with 48 kHz input
- At 48 kbps, it will produce HE-AAC v1 Streams

Speaking of which, I will organize a small pre-test soon with HE-AAC vs. HE-AAC v2 at 48 kbps, with two HE-AAC and two HE-AAC v2 implementations - so we can see which mode is better for each of the implementators (also would be good for reference for others encoding their material at 48 kbps).

Or, maybe we should include both HE-AAC and HE-AAC v2 in the 48 kbps test, but it would already be too much of the competitors?

My opinions is that it could be a good idea to first do an AAC low bitrate test before a low bitrate multiformat test.

This way, we would have less competitors per test, and perhaps more results (in the meaning of more information).

To me competitors would be (at least):
*Apple
*Nero
*CT commercial
*3gpp reference code (ie CT non commercial)

Is any product using some FhG encoder?

I think that bitrate of such test should be 32kbps, as this would allow to use HE-AAC v2 with the 3gpp code.

This way we would know which AAC encoder is better at low bitrates, but also how does the 3gpp reference code fares against commercial encoders.
This would also allow to reduce the number of contenders in a 48kbps multi-format test, as we would already have some info regarding AAC.

But @Gabriel, do you think that the 32kbps resutls would extrapolate well to 48 kbps to be useful for the selection of the encoder? I have a fear that some codecs >might< show completely different behavior in terms of ranking - not sure about that, but just maybe.

Also, there is another problem @48 kbps - while @32 kbps it is pretty clear that HE-AAC v2 will outperform HE-AAC v1 in all implementations, at 48 kbps this is starting to be a bit questionable as it has been shown on few internal tests of HE-AAC v1 vs HE-AAC v2 at 48 kbits/s.

I will definitely organise a separate small listening test of HE-AAC v1 and v2 codecs @48 kbps (and Ogg maybe) - by using of the 10-or-so average worst samples (combined average SDG) from the previous 128 and 64 kbps tests.

My intial plan was to use:

* 5 samples from the previous multiformat 64 kbps test, with worst average SDG
* 5 samples from the previous 128 kbps test, with worst average SDG

This will help chose the proper AAC mode at 48 kbps for the future 48 kbps multiformat listening, because there is an open issue whether Parametric Stereo (PS) or HE-AAC v2 is improving quality enough or the stereo-field regressions are too bad to justify the quality increase on some clips. Testing these two modes side-by-side directly would help us make the choice for the future.

Also I think this will have very big value for the people considering HE-AAC/HE-AAC v2 for their 48 kbps encodings/streams - as it would show whether Parametric Stereo is good for that bitrate or not. Currently there is not yet enough test data to make advisory - this test would definitely help

I planned to do this test soon (in 10 or so days maybe)- I will post it on HA, too - as well as methodology and the selection criteria for the samples which will be completely transparent and unbiased toward any codec/solution.

For sure there will be:

- CT HE-AAC v1 (aacPlus)
- CT HE-AAC v2 (aacPlus v2)
- Nero Digital - HE-AAC v1 [new codec]
- Nero Digital - HE-AAC v2 [new codec]

- And probably LAME @128 kbps as high-anchor
- LAME @48 kbps (edit: or, even better, AAC @48 kbps), as low anchor


But I am not sure whether to reveal the codec names in the test resutls (I would anyway do it for the orgnizers of the future tests), as I am sure that there will be doubts about the credibility here (as always - and by not promoting any codec I am trying to stay as neutral as possible - this test is about the v1 and v2 HE-AAC technology, not about the particular implementations.

You are kidding right?

As long as results are reproducible (if we now, or in the soon future, can use an encoder to create the files that were tested)and the selection of samples is transparent and open for discussion, I can't see any reason why it would hurt the credibility of the test if the names where revealed.

The reason I'm bringing this up is two-fold:

1. The results will be less interesting and people might feel less inclined to participate. (assuming your test is open for us)

2. Technology and implementation are not always very separable. For example, you can't conclude that MP3 is a better technology than Vorbis or AAC by comparing LAME with a very bad AAC or Vorbis encoder. I don't know if this comparison holds also when it comes to SBR and PS, but it would be something to consider I think.

I wish - but reading the usual lisetning test rants I got that feeling

Anyway - I am discussing this matter with another well known name in audio compression, as he initally proposed idea of teaming up for such test - if he agrees, we might do this together and make the test even better.



Well... I agree on all that, but nevertheless we might have some wise guys claiming that someone here "biased" the test towards something - which I clearly woudln't ever do.

Anyway I also think a very high level of transparencu could be achieved if:

a) Encoders are available to everyone
b) Encoded bitstream verification is done by few people

c) Optionally, that the selection of the samples comes after the encoders were released (to avoid doubts that encoders were "produced" for the sample sent), or to provide clear and transparent way of selecting samples.



Good point. Taken



Actually we would compare two state-of-the-art HE-AAC v1 and v2 solutions - so the test cannot be wrong with that regard.

Any word of yet as to how any of these tests are going to be set up?

I think Gabriel has a good idea at comparing various AAC contenders before a multiformat test; Also, organizing many tests might make people feel more enthusiastic about the community, and thus conduce a retour en force of hydrogenaudio.

à la grâce de Dieu,
Tristan.

there are quick fix to allow 44.1 kHz input !

As for my idea of 48 kbps AAC-only pre-test, I am still waiting for the reply of the person I mentioned (sorry to keep name private, I will of course reveal it as soon as/if he accepts)

My proposal (to be discussed further) would be the following:

#1 - We will have open discussion on HA of the sample set, that should last couple of days - my current idea and proposal would be to use 10-15 samples that scored worst in previous Roberto's 64 and 128 multiformat listening tests. "Worst" means lowest average SDG for all codecs used to code that sample, excluding low-anchors

#2 - Myself and Person X will encode files in CT's, Nero's and anchor formats, previously agreeing on the coding modes, and making public verification of the bitrates.

#3 - We will exchange the files and verify they are the same and within the bit-rate criteria

#4 - We will preapre and sign the ABC-HR test sets, upload the file and store the codec list.

This is just my rough idea about the AAC pre-test - I do have very strong belief that testing v1 and v2 HE-AAC modes @48 kbps would have a huge value before the multiformat 48 kbps test.



Agreed completely

I also think that it would be good that HA as the whole takes more active role in this - people behind the test organization would surely get the credit, and HA could host the results as unique one-and-only listening test database.

It wil be good to see not only music samples but also movie tracks (for video encoding), (wo)man speech for radio. This way test will be more usefull for different aplications.

Maybe it will be also interesting to see how perform codec with and withoust PS.
48 kbit/s (VBR aprrox 51-52 kbit/s) is highest bitrate when PS is still usefull.

Before you start any test, I suggest you take the sane advice given already in this thread.

Also, read all the previous pre-test discussion threads very carefully and ask if you don't understand something.

The first question any tester should ask is: why am I doing this test? What do I want to find out.

Define your test question first.

As said by Gabriel, it would make sense to make an AAC test first, maybe on personal basis, or only few interested participaters, to select best AAC.
Then a multiformat low bitrate 48k test, including lame abr48, and maybe Fraunhofer, if that is known to be best mp3@48, we will see, that is the reason for tests. Excluding mp3 from those tests is not appropriate, as it is most widespread and compatible encoder.


[*]128 kbps extension test featuring ATRAC3, MusePack and WMA Standard (later might be problematic because of the settings)

I have said already something to this,
best solution imo: set down the averaged all-music bitrate from 128k down to 9x or 10x k., so that wma suits regarding bitrate. And, as bitrate is lower than 128k, testing is little bit easier.

Gabriel and I discussed these ideas - and we will organize this 48 kbps AAC pre-test somewhere near the end of Jaunary.

I will soon post the general ideas for the test, so we could discuss them before anything is done

Basically, the test should test current state-of-the-art AAC encoders at 48 kbps:

* HE-AAC v1: Nero Digital HE-AAC v1, Coding Techologies aacPlus, 3GPP Reference Encoder

* HE-AAC v2: Nero Digital HE-AAC v2, Coding Technologies Enhanced aacPlus

* Low-Anchor: LC-AAC at 48 kbps

* High Anchor: Maybe MP3 @128 kbps?

So the main idea of the test is not just to chose best candidates for the 48 kbps multiformat, but also to verify the usefullness of Parametric Stereo at this bet rate, as well as to check the quality of the 3GPP reference encoder.

Before testing aoTuV beta 4.51 would be interesting compare beta 4.51 with beta 4, looking which has better quality (beta 4.51 should have better quallity with q < 3, but no one have confirmed it).

i cant hear any difference between them, the latest version should probably be used unless problems are found.

suggested encoding settings:
vorbis aotuv b4.51: -q -.5 --resample 37800 --advanced-encode-option lowpass_frequency=17
lame v3.93.1: -q 1 -V 9 --lowpass 9 --resample 22 --nspsytune (about 48kbps, and sounds better than lame3.97 abr 48 IMO)

what about including plusV in this test? (i know, i know, its old and dead, but it is meant to be used with low bitrates) the only improvement i have hacked into the plusV reference software is control over the plusV bands, the ability to encode down to 16khz input, and the ability to have full control over how the mp3 portion is encoded . Was plusV ever included in any listening tests? I for one am curious how people would rate it.

edit: corrected lame command line
edit2: 'better' vorbis settings

Ahhh... the thread has been resurrected

So. Is this test for real? What if we put a link on the Current Events wiki page informing about another up-coming test? Although it is currently still in pre-test discussion stage... so the link should point to this pre-test discussion instead

If the test will go through, of course.