IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Ivan Dimkovic
post Jan 24 2006, 12:13
Post #1


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Dear All,

I would like to announce that the 48 kbps test of upcoming Nero AAC encoder has just started. The purpose of this test is to determine which bit allocation mode gives highest quality, and how big the quality difference is between ABR, CBR and VBR.

With this test, I wish to include true audiophile and scientific community into making the HE-AAC /v2 solutions even better in the future. Many thanks in advance for your help!

Test runs for 10 days, Until february 3rd 2006. It should be much easier and more interesting to the wide audience than the last 128 kbps test, since I hope many more people would be able to actually judge the quality smile.gif

So, What's in?

There are two tests actually - Test 1 and Test 2.

- Test 1 contains HE-AAC v1 tracks (AAC+SBR)
- Test 2 contains HE-AAC v2 tracks (AAC+SBR+Parametric Stereo)

The reason why they are split is that rating HE-AAC vs HE-AAC v2 is not the point of this test (it is the point of the next public 48 kbps multivendor AAC test coming in February)

Which codecs are tested?

For both tests, following codecs are being tested:

- HE-AAC (or v2) CBR - Constant Bit Rate, optimized for online streaming
- HE-AAC (or v2) ABR - Average Bit Rate, little bit more pre-buffer delay, more room for "hard to encode" frames
- HE-AAC (or v2) VBR v1 - Quality iterated VBR 1
- HE-AAC (or v2) VBR v2 - Quality iterated VBR 2
- MP3 at 128 kbps (LAME -b128) *** High Anchor ***
- MP3 at 48 kbps (LAME -b48) *** Low Anchor ***

Difference between quality iterated VBR modes is in the bit allocation strategy. All encoding modes (except high anchor) should produce clips of average bit rate close to the 48 kbps.

How To Test

1. Download test1.rar and test2.rar (links in the next message)

2. Extract these two RAR files in separate directories (for example, C:\TEST1 and C:\TEST2)

3. Download samples for the test, and extract them in the directory of the test (so, test1_atrain.rar should go to C:\TEST1 and test2_atrain.rar should go to C:\TEST2)

4. When you download all the samples, in each test directory (C:\TEST1 or C:\TEST2 in example) execute the decode_all.bat files. This will decode all the files and prepare the test. Note that you have to do it for both tests separately.

5. Alternatively, you can download and decode each sample, and execute individual bat files to decode just that sample (e.g. for test1_atrain.rar, unpacked in C:\TEST1, you have to execute C:\TEST1\atrain.bat)

6. Now everything is ready for the test, and the file structure should look like:

CODE
C:\TEST1\
C:\TEST1\ATRAIN\
                           atrain.wav
                           atrain_codeca.wav
                           atrain_codecb.wav
                           atrain_codecc.wav
                           atrain_codecd.wav
                           atrain_codece.wav
                           atrain_codecf.wav



7. Open ABC/HR by executing abchr.jar - and, to test each sample, open up the individual .ecf files. Each test set (test1 or test2) has an .ecf file for each sample.

8. From now on, typical ABC/HR methodology applies, you need to judge the quality of individual files - save the results after you're done, and send the files to me, either by PP or directly to: idimkovic (at) nero.com (replace at with @ to get an e-mail address)

Samples coming in the next post...

Happy Testing smile.gif

This post has been edited by Ivan Dimkovic: Jan 24 2006, 12:47
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 24 2006, 12:20
Post #2


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Sample selection

Total 14 samples are tested, exposing all relevant issues in audio coding domain (pre-echoes, high tonal content, human speech, strong stereo separation, complex harmonic structure, etc...) - Two of these samples were transcoded from the lossy source (Yello and Sympathy) in order to reflect typical 48 kbps codec usage even more.

So, here are the links:

ftp://ftp.morbo.org/pub/Nero_48_Test/

TEST 1

ftp://ftp.morbo.org/pub/Nero_48_Test/test1.rar -- Test 1 SETUP
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_atrain.rar -- Test 1, Sample 1
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_biniou.rar -- Test 1, Sample 2
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_clapton_44k.rar -- Test 1, Sample 3
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_es02.rar -- Test 1, Sample 4
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_french_ad.rar -- Test 1, Sample 5
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_harp40_1.rar -- Test 1, Sample 6
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_mahler.rar -- Test 1, Sample 7
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_sandman.rar -- Test 1, Sample 8
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_si02.rar -- Test 1, Sample 9
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_sm01.rar -- Test 1, Sample 10
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_sympathy.rar -- Test 1, Sample 11
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_te09_44k.rar -- Test 1, Sample 12
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_yello.rar -- Test 1, Sample 13
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_youcantdothat.rar -- Test 1, Sample 14

TEST 2

ftp://ftp.morbo.org/pub/Nero_48_Test/test2.rar -- Test 2 SETUP
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_atrain.rar -- Test 2, Sample 1
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_biniou.rar -- Test 2, Sample 2
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_clapton_44k.rar -- Test 2, Sample 3
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_es02.rar -- Test 2, Sample 4
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_french_ad.rar -- Test 2, Sample 5
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_harp40_1.rar -- Test 2, Sample 6
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_mahler.rar -- Test 2, Sample 7
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_sandman.rar -- Test 2, Sample 8
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_si02.rar -- Test 2, Sample 9
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_sm01.rar -- Test 2, Sample 10
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_sympathy.rar -- Test 2, Sample 11
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_te09_44k.rar -- Test 2, Sample 12
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_yello.rar -- Test 2, Sample 13
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_youcantdothat.rar -- Test 2, Sample 14

Edit - typo.

This post has been edited by Ivan Dimkovic: Jan 24 2006, 15:20
Go to the top of the page
+Quote Post
Shade[ST]
post Jan 24 2006, 14:24
Post #3





Group: Members
Posts: 1189
Joined: 19-May 05
From: Montreal, Canada
Member No.: 22144



Finally it started! I'm going to take part in this ASAP!
Go to the top of the page
+Quote Post
IgorC
post Jan 24 2006, 15:18
Post #4





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



ftp://ftp.morbo.org/pub/Nero_48_Test/test2_atrain.rar.rar link is broken
Doble .rar.rar

This post has been edited by IgorC: Jan 24 2006, 15:19
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 24 2006, 15:21
Post #5


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Thanks IgorC for noticing it - It is corrected now.
Go to the top of the page
+Quote Post
sehested
post Jan 24 2006, 15:39
Post #6





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



QUOTE (Ivan Dimkovic @ Jan 24 2006, 03:13 AM)
8. From now on, typical ABC/HR methodology applies, you need to judge the quality of individual files - save the results after you're done, and send the files to me, either by PP or directly to:  idimkovic (at) nero.com (replace at with @ to get an e-mail address)
*

What are the requirements for producing valid results?

For example:
- Will ranking any reference make the result invalid?
- Will it be acceptable to mistake between the high anchor and reference?
- Will results with 5.0 ratings be invalid?
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 24 2006, 16:04
Post #7


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



QUOTE
- Will ranking any reference make the result invalid?


I would say - yes. Ranking reference in such a low bit rate test is clearly a sign of either a mistake or lack of listening training.

QUOTE
- Will it be acceptable to mistake between the high anchor and reference?


I will have to think about that - and discuss with some other people, too.

QUOTE
- Will results with 5.0 ratings be invalid?


5.0 results will be valid.
Go to the top of the page
+Quote Post
Firon
post Jan 26 2006, 00:12
Post #8





Group: Members
Posts: 830
Joined: 3-November 05
Member No.: 25526



Do we have to perform all the tests for the results to count or can we do say, only test2? (I'm of course planning to do both)
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 26 2006, 09:26
Post #9


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



You can do both the tests, or a single test, or only few samples of any set (of course, I'd prefer full results, but we also accept single sample results, too wink.gif
Go to the top of the page
+Quote Post
vinouz
post Jan 26 2006, 19:24
Post #10





Group: Members
Posts: 4
Joined: 20-January 06
Member No.: 27240



suggestion : a little shell script instead of the batch file, for linux users...
Go to the top of the page
+Quote Post
IgorC
post Jan 29 2006, 03:42
Post #11





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



Maybe it is my ignorance but it seems to me there is no big difference between Nero VBR1/2 , ABR, CBR on some (not on all?) samples like Sadman and Beatles (You can do it)
I also tried to ABX between Nero's VBR1&2 /CBR/ABRsamples . It does not help.

CODE
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname: Beatles

1L = C:\48\test2_youcantdothat\005 youcantdothat_codecD.wav
2L = C:\48\test2_youcantdothat\001 youcantdothat_codecE.wav
3L = C:\48\test2_youcantdothat\006 youcantdothat_codecF.wav
4R = C:\48\test2_youcantdothat\002 youcantdothat_codecA.wav
5R = C:\48\test2_youcantdothat\004 youcantdothat_codecC.wav
6R = C:\48\test2_youcantdothat\003 youcantdothat_codecB.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: C:\48\test2_youcantdothat\005 youcantdothat_codecD.wav
1L Rating: 2.3
1L Comment:
---------------------------------------
3L File: C:\48\test2_youcantdothat\006 youcantdothat_codecF.wav
3L Rating: 1.0
3L Comment:
---------------------------------------
4R File: C:\48\test2_youcantdothat\002 youcantdothat_codecA.wav
4R Rating: 2.3
4R Comment:
---------------------------------------
5R File: C:\48\test2_youcantdothat\004 youcantdothat_codecC.wav
5R Rating: 2.3
5R Comment:
---------------------------------------
6R File: C:\48\test2_youcantdothat\003 youcantdothat_codecB.wav
6R Rating: 2.3
6R Comment:
---------------------------------------
ABX Results:
Original vs C:\48\test2_youcantdothat\005 youcantdothat_codecD.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_youcantdothat\001 youcantdothat_codecE.wav
   5 out of 8, pval = 0.363
Original vs C:\48\test2_youcantdothat\006 youcantdothat_codecF.wav
   8 out of 8, pval = 0.004
Original vs C:\48\test2_youcantdothat\002 youcantdothat_codecA.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_youcantdothat\004 youcantdothat_codecC.wav
   8 out of 8, pval = 0.004
Original vs C:\48\test2_youcantdothat\003 youcantdothat_codecB.wav
   5 out of 5, pval = 0.031


CODE
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname: Sandman

1R = C:\48\test2_sandman\sandman\004 sandman_codecD.wav
2L = C:\48\test2_sandman\sandman\002 sandman_codecB.wav
3R = C:\48\test2_sandman\sandman\006 sandman_codecF.wav
4R = C:\48\test2_sandman\sandman\003 sandman_codecC.wav
5L = C:\48\test2_sandman\sandman\001 sandman_codecA.wav
6L = C:\48\test2_sandman\sandman\005 sandman_codecE.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: C:\48\test2_sandman\sandman\004 sandman_codecD.wav
1R Rating: 2.0
1R Comment:
---------------------------------------
2L File: C:\48\test2_sandman\sandman\002 sandman_codecB.wav
2L Rating: 2.0
2L Comment:
---------------------------------------
3R File: C:\48\test2_sandman\sandman\006 sandman_codecF.wav
3R Rating: 1.0
3R Comment:
---------------------------------------
4R File: C:\48\test2_sandman\sandman\003 sandman_codecC.wav
4R Rating: 2.0
4R Comment:
---------------------------------------
5L File: C:\48\test2_sandman\sandman\001 sandman_codecA.wav
5L Rating: 2.0
5L Comment:
---------------------------------------
6L File: C:\48\test2_sandman\sandman\005 sandman_codecE.wav
6L Rating: 4.7
6L Comment:
---------------------------------------
ABX Results:
Original vs C:\48\test2_sandman\sandman\004 sandman_codecD.wav
   8 out of 8, pval = 0.004
Original vs C:\48\test2_sandman\sandman\002 sandman_codecB.wav
   8 out of 8, pval = 0.004
Original vs C:\48\test2_sandman\sandman\006 sandman_codecF.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_sandman\sandman\003 sandman_codecC.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_sandman\sandman\001 sandman_codecA.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_sandman\sandman\005 sandman_codecE.wav
   7 out of 8, pval = 0.035


This post has been edited by IgorC: Jan 29 2006, 03:49
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 30 2006, 20:20
Post #12


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



QUOTE
suggestion : a little shell script instead of the batch file, for linux users...


Huh... must be because I am dumb windows user smile.gif If someone is willing to share some shell scripts doing this job with me - I could update the archives of course wink.gif

QUOTE
IgorC
Maybe it is my ignorance but it seems to me there is no big difference between Nero VBR1/2 , ABR, CBR on some (not on all?) samples like Sadman and Beatles (You can do it)
I also tried to ABX between Nero's VBR1&2 /CBR/ABRsamples . It does not help.


Huh... "discussing of the results ist verboten" smile.gif

Anyway - this is the sole purpose of the test - to be honest, I am not at all amused in how would VBR perform on 48 kbps compared to ABR or even CBR - since there is not so much space to actually "adapt' the bit rate, and SBR anyway takes half of the spectrum (and SBR is not scalable) - but let's see after the test ends.

Also, too bad there is no time to test 2-pass CBR, too - but maybe in one of the future tests later this year smile.gif

I would like to ask for more people to do the test, please smile.gif I only got 2 incomplete results so far - and if this trend continues, there won't be too much to discuss and analyse... thanks in advance!
Go to the top of the page
+Quote Post
archish
post Jan 31 2006, 12:58
Post #13





Group: Members
Posts: 23
Joined: 26-July 03
From: India
Member No.: 8008



which which version of nero encoder is HE-AAC v2 supplied?
thanks
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 31 2006, 13:17
Post #14


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Basically all recent Nero7 AAC encoders have HE-AAC v2.

What we are testing here is completely new AAC encoder - which will be releaesed in short time - so its performance is quite different (I believe) than the current public N7 AAC encoder.
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Feb 1 2006, 12:56
Post #15


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



2 more days left for the test.

Due to lack of results (3 for test 1, 1 for test 2, many incomplete) I might consider postponing finishing of this test until Sunday evening (5th of February), hopefully more people will have time to do it during weekend.

Please if you have time, do the test - I would expect 2 or 3 certain results for test 1 and test 2, but that is still not enough for reliable analysis.

Thanks a lot in advance!
Go to the top of the page
+Quote Post
Alex B
post Feb 1 2006, 13:42
Post #16





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



I am not promising anything, but I could give it a try next weekend. I have been very busy during the last few weeks and had no time for hobbies like this.


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
Gabriel
post Feb 1 2006, 14:14
Post #17


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



QUOTE
Due to lack of results (3 for test 1, 1 for test 2, many incomplete)

Does it means that right now you do not even have full results from you, Menno, Garf and Juha? wink.gif
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Feb 1 2006, 14:32
Post #18


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



smile.gif

Actually the results I mentioned are excludig @nero.com results.

I will refrain from testing the clips I authored because I can tell which codec is which and I feel I would be biased by that fact.
Go to the top of the page
+Quote Post
Firon
post Feb 2 2006, 01:51
Post #19





Group: Members
Posts: 830
Joined: 3-November 05
Member No.: 25526



I'm still working on it, I've been busy unfortunately... I've finished test1, done 2 samples in test 2, so I've got a ways to go. I'm hoping to finish tomorrow.

I have a question though, what's the point of the low anchor in listening tests? So you can establish a scale or...? Pardon the n00b question. tongue.gif
And when the test is finished, do you get to see individual results? (so you can see if you screwed up and so on)

This post has been edited by Firon: Feb 3 2006, 03:32
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Feb 3 2006, 10:38
Post #20


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Just a quick note - test is definitely extended until Sunday, 5th of February!

(I got more results, but still I need more for reliable analysis) - your feedback will be greatly appreciated!

QUOTE
I have a question though, what's the point of the low anchor in listening tests? So you can establish a scale or...? Pardon the n00b question. tongue.gif


Yep - it is to establish a reliable scale, since IMHO most of the codecs could be very tied.

QUOTE
And when the test is finished, do you get to see individual results? (so you can see if you screwed up and so on)


Yes, I will upload all individual results - and post the test keys, so everyone could check their own test locally, too.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 24th April 2014 - 09:46