Nero 48 kbit/s HE-AAC (+v2) Test - STARTED! |
  |
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED! |
 Jan 24 2006, 12:13
Post
#1
|
|
|
Nero MPEG4 developer Group: Developer Posts: 1466 Joined: 22-September 01 Member No.: 8  |
Dear All,
I would like to announce that the 48 kbps test of upcoming Nero AAC encoder has just started. The purpose of this test is to determine which bit allocation mode gives highest quality, and how big the quality difference is between ABR, CBR and VBR. With this test, I wish to include true audiophile and scientific community into making the HE-AAC /v2 solutions even better in the future. Many thanks in advance for your help! Test runs for 10 days, Until february 3rd 2006. It should be much easier and more interesting to the wide audience than the last 128 kbps test, since I hope many more people would be able to actually judge the quality  So, What's in? There are two tests actually - Test 1 and Test 2. - Test 1 contains HE-AAC v1 tracks (AAC+SBR) - Test 2 contains HE-AAC v2 tracks (AAC+SBR+Parametric Stereo) The reason why they are split is that rating HE-AAC vs HE-AAC v2 is not the point of this test (it is the point of the next public 48 kbps multivendor AAC test coming in February) Which codecs are tested? For both tests, following codecs are being tested: - HE-AAC (or v2) CBR - Constant Bit Rate, optimized for online streaming - HE-AAC (or v2) ABR - Average Bit Rate, little bit more pre-buffer delay, more room for "hard to encode" frames - HE-AAC (or v2) VBR v1 - Quality iterated VBR 1 - HE-AAC (or v2) VBR v2 - Quality iterated VBR 2 - MP3 at 128 kbps (LAME -b128) *** High Anchor *** - MP3 at 48 kbps (LAME -b48) *** Low Anchor *** Difference between quality iterated VBR modes is in the bit allocation strategy. All encoding modes (except high anchor) should produce clips of average bit rate close to the 48 kbps. How To Test 1. Download test1.rar and test2.rar (links in the next message) 2. Extract these two RAR files in separate directories (for example, C:\TEST1 and C:\TEST2) 3. Download samples for the test, and extract them in the directory of the test (so, test1_atrain.rar should go to C:\TEST1 and test2_atrain.rar should go to C:\TEST2) 4. When you download all the samples, in each test directory (C:\TEST1 or C:\TEST2 in example) execute the decode_all.bat files. This will decode all the files and prepare the test. Note that you have to do it for both tests separately. 5. Alternatively, you can download and decode each sample, and execute individual bat files to decode just that sample (e.g. for test1_atrain.rar, unpacked in C:\TEST1, you have to execute C:\TEST1\atrain.bat) 6. Now everything is ready for the test, and the file structure should look like: CODE C:\TEST1\ C:\TEST1\ATRAIN\ atrain.wav atrain_codeca.wav atrain_codecb.wav atrain_codecc.wav atrain_codecd.wav atrain_codece.wav atrain_codecf.wav 7. Open ABC/HR by executing abchr.jar - and, to test each sample, open up the individual .ecf files. Each test set (test1 or test2) has an .ecf file for each sample. 8. From now on, typical ABC/HR methodology applies, you need to judge the quality of individual files - save the results after you're done, and send the files to me, either by PP or directly to: idimkovic (at) nero.com (replace at with @ to get an e-mail address) Samples coming in the next post... Happy Testing
This post has been edited by Ivan Dimkovic: Jan 24 2006, 12:47 |
 |
 
|
|
Jan 24 2006, 14:24
Post
#3
|
|
 Group: Members Posts: 1189 Joined: 19-May 05 From: Montreal, Canada Member No.: 22144 |
Finally it started! I'm going to take part in this ASAP!
|
|
|
|
|
Jan 24 2006, 15:18
Post
#4
|
|
|
Group: Members Posts: 1315 Joined: 3-January 05 From: Argentina, Bs As Member No.: 18803 |
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_atrain.rar.rar link is broken
Doble .rar.rar This post has been edited by IgorC: Jan 24 2006, 15:19 |
|
|
|
|
Jan 24 2006, 15:21
Post
#5
|
|
|
Nero MPEG4 developer Group: Developer Posts: 1466 Joined: 22-September 01 Member No.: 8 |
Thanks IgorC for noticing it - It is corrected now.
|
|
|
|
|
Jan 24 2006, 15:39
Post
#6
|
|
 Group: Members (Donating) Posts: 325 Joined: 5-April 04 From: Copenhagen, Denmark Member No.: 13246 |
QUOTE (Ivan Dimkovic @ Jan 24 2006, 03:13 AM) 8. From now on, typical ABC/HR methodology applies, you need to judge the quality of individual files - save the results after you're done, and send the files to me, either by PP or directly to: idimkovic (at) nero.com (replace at with @ to get an e-mail address)  What are the requirements for producing valid results? For example: - Will ranking any reference make the result invalid? - Will it be acceptable to mistake between the high anchor and reference? - Will results with 5.0 ratings be invalid? |
|
|
|
|
Jan 24 2006, 16:04
Post
#7
|
|
|
Nero MPEG4 developer Group: Developer Posts: 1466 Joined: 22-September 01 Member No.: 8 |
QUOTE - Will ranking any reference make the result invalid? I would say - yes. Ranking reference in such a low bit rate test is clearly a sign of either a mistake or lack of listening training. QUOTE - Will it be acceptable to mistake between the high anchor and reference? I will have to think about that - and discuss with some other people, too. QUOTE - Will results with 5.0 ratings be invalid? 5.0 results will be valid. |
|
|
|
|
Jan 26 2006, 00:12
Post
#8
|
|
|
Group: Members Posts: 830 Joined: 3-November 05 Member No.: 25526 |
Do we have to perform all the tests for the results to count or can we do say, only test2? (I'm of course planning to do both)
|
|
|
|
|
Jan 26 2006, 09:26
Post
#9
|
|
|
Nero MPEG4 developer Group: Developer Posts: 1466 Joined: 22-September 01 Member No.: 8 |
You can do both the tests, or a single test, or only few samples of any set (of course, I'd prefer full results, but we also accept single sample results, too
|
|
|
|
|
Jan 26 2006, 19:24
Post
#10
|
|
|
Group: Members Posts: 4 Joined: 20-January 06 Member No.: 27240 |
suggestion : a little shell script instead of the batch file, for linux users...
|
|
|
|
|
Jan 29 2006, 03:42
Post
#11
|
|
|
Group: Members Posts: 1315 Joined: 3-January 05 From: Argentina, Bs As Member No.: 18803 |
Maybe it is my ignorance but it seems to me there is no big difference between Nero VBR1/2 , ABR, CBR on some (not on all?) samples like Sadman and Beatles (You can do it)
I also tried to ABX between Nero's VBR1&2 /CBR/ABRsamples . It does not help. CODE ABC/HR Version 1.1 beta 2, 18 June 2004 Testname: Beatles 1L = C:\48\test2_youcantdothat\005 youcantdothat_codecD.wav 2L = C:\48\test2_youcantdothat\001 youcantdothat_codecE.wav 3L = C:\48\test2_youcantdothat\006 youcantdothat_codecF.wav 4R = C:\48\test2_youcantdothat\002 youcantdothat_codecA.wav 5R = C:\48\test2_youcantdothat\004 youcantdothat_codecC.wav 6R = C:\48\test2_youcantdothat\003 youcantdothat_codecB.wav --------------------------------------- General Comments: --------------------------------------- 1L File: C:\48\test2_youcantdothat\005 youcantdothat_codecD.wav 1L Rating: 2.3 1L Comment: --------------------------------------- 3L File: C:\48\test2_youcantdothat\006 youcantdothat_codecF.wav 3L Rating: 1.0 3L Comment: --------------------------------------- 4R File: C:\48\test2_youcantdothat\002 youcantdothat_codecA.wav 4R Rating: 2.3 4R Comment: --------------------------------------- 5R File: C:\48\test2_youcantdothat\004 youcantdothat_codecC.wav 5R Rating: 2.3 5R Comment: --------------------------------------- 6R File: C:\48\test2_youcantdothat\003 youcantdothat_codecB.wav 6R Rating: 2.3 6R Comment: --------------------------------------- ABX Results: Original vs C:\48\test2_youcantdothat\005 youcantdothat_codecD.wav 5 out of 5, pval = 0.031 Original vs C:\48\test2_youcantdothat\001 youcantdothat_codecE.wav 5 out of 8, pval = 0.363 Original vs C:\48\test2_youcantdothat\006 youcantdothat_codecF.wav 8 out of 8, pval = 0.004 Original vs C:\48\test2_youcantdothat\002 youcantdothat_codecA.wav 5 out of 5, pval = 0.031 Original vs C:\48\test2_youcantdothat\004 youcantdothat_codecC.wav 8 out of 8, pval = 0.004 Original vs C:\48\test2_youcantdothat\003 youcantdothat_codecB.wav 5 out of 5, pval = 0.031 CODE ABC/HR Version 1.1 beta 2, 18 June 2004
Testname: Sandman 1R = C:\48\test2_sandman\sandman\004 sandman_codecD.wav 2L = C:\48\test2_sandman\sandman\002 sandman_codecB.wav 3R = C:\48\test2_sandman\sandman\006 sandman_codecF.wav 4R = C:\48\test2_sandman\sandman\003 sandman_codecC.wav 5L = C:\48\test2_sandman\sandman\001 sandman_codecA.wav 6L = C:\48\test2_sandman\sandman\005 sandman_codecE.wav --------------------------------------- General Comments: --------------------------------------- 1R File: C:\48\test2_sandman\sandman\004 sandman_codecD.wav 1R Rating: 2.0 1R Comment: --------------------------------------- 2L File: C:\48\test2_sandman\sandman\002 sandman_codecB.wav 2L Rating: 2.0 2L Comment: --------------------------------------- 3R File: C:\48\test2_sandman\sandman\006 sandman_codecF.wav 3R Rating: 1.0 3R Comment: --------------------------------------- 4R File: C:\48\test2_sandman\sandman\003 sandman_codecC.wav 4R Rating: 2.0 4R Comment: --------------------------------------- 5L File: C:\48\test2_sandman\sandman\001 sandman_codecA.wav 5L Rating: 2.0 5L Comment: --------------------------------------- 6L File: C:\48\test2_sandman\sandman\005 sandman_codecE.wav 6L Rating: 4.7 6L Comment: --------------------------------------- ABX Results: Original vs C:\48\test2_sandman\sandman\004 sandman_codecD.wav 8 out of 8, pval = 0.004 Original vs C:\48\test2_sandman\sandman\002 sandman_codecB.wav 8 out of 8, pval = 0.004 Original vs C:\48\test2_sandman\sandman\006 sandman_codecF.wav 5 out of 5, pval = 0.031 Original vs C:\48\test2_sandman\sandman\003 sandman_codecC.wav 5 out of 5, pval = 0.031 Original vs C:\48\test2_sandman\sandman\001 sandman_codecA.wav 5 out of 5, pval = 0.031 Original vs C:\48\test2_sandman\sandman\005 sandman_codecE.wav 7 out of 8, pval = 0.035 This post has been edited by IgorC: Jan 29 2006, 03:49 |
|
|
|
|
Jan 30 2006, 20:20
Post
#12
|
|
|
Nero MPEG4 developer Group: Developer Posts: 1466 Joined: 22-September 01 Member No.: 8 |
QUOTE suggestion : a little shell script instead of the batch file, for linux users... Huh... must be because I am dumb windows user If someone is willing to share some shell scripts doing this job with me - I could update the archives of course QUOTE IgorC Maybe it is my ignorance but it seems to me there is no big difference between Nero VBR1/2 , ABR, CBR on some (not on all?) samples like Sadman and Beatles (You can do it) I also tried to ABX between Nero's VBR1&2 /CBR/ABRsamples . It does not help. Huh... "discussing of the results ist verboten" Anyway - this is the sole purpose of the test - to be honest, I am not at all amused in how would VBR perform on 48 kbps compared to ABR or even CBR - since there is not so much space to actually "adapt' the bit rate, and SBR anyway takes half of the spectrum (and SBR is not scalable) - but let's see after the test ends. Also, too bad there is no time to test 2-pass CBR, too - but maybe in one of the future tests later this year I would like to ask for more people to do the test, please I only got 2 incomplete results so far - and if this trend continues, there won't be too much to discuss and analyse... thanks in advance!
|
|
|
|
|
Jan 31 2006, 12:58
Post
#13
|
|
 Group: Members Posts: 23 Joined: 26-July 03 From: India Member No.: 8008 |
which which version of nero encoder is HE-AAC v2 supplied?
thanks |
|
|
|
|
Jan 31 2006, 13:17
Post
#14
|
|
|
Nero MPEG4 developer Group: Developer Posts: 1466 Joined: 22-September 01 Member No.: 8 |
Basically all recent Nero7 AAC encoders have HE-AAC v2.
What we are testing here is completely new AAC encoder - which will be releaesed in short time - so its performance is quite different (I believe) than the current public N7 AAC encoder. |
|
|
|
|
Feb 1 2006, 12:56
Post
#15
|
|
|
Nero MPEG4 developer Group: Developer Posts: 1466 Joined: 22-September 01 Member No.: 8 |
2 more days left for the test.
Due to lack of results (3 for test 1, 1 for test 2, many incomplete) I might consider postponing finishing of this test until Sunday evening (5th of February), hopefully more people will have time to do it during weekend. Please if you have time, do the test - I would expect 2 or 3 certain results for test 1 and test 2, but that is still not enough for reliable analysis. Thanks a lot in advance! |
|
|
|
|
Feb 1 2006, 13:42
Post
#16
|
|
 Group: Members Posts: 1303 Joined: 14-September 05 From: Helsinki, Finland Member No.: 24472 |
I am not promising anything, but I could give it a try next weekend. I have been very busy during the last few weeks and had no time for hobbies like this.
-------------------- http://listening-tests.freetzi.com
|
|
|
|
|
Feb 1 2006, 14:14
Post
#17
|
|
 LAME developer Group: Developer Posts: 2950 Joined: 1-October 01 From: Nanterre, France Member No.: 138 |
QUOTE Due to lack of results (3 for test 1, 1 for test 2, many incomplete) Does it means that right now you do not even have full results from you, Menno, Garf and Juha?
|
|
|
|
|
Feb 1 2006, 14:32
Post
#18
|
|
|
Nero MPEG4 developer Group: Developer Posts: 1466 Joined: 22-September 01 Member No.: 8 |
Actually the results I mentioned are excludig @nero.com results. I will refrain from testing the clips I authored because I can tell which codec is which and I feel I would be biased by that fact. |
|
|
|
|
Feb 2 2006, 01:51
Post
#19
|
|
|
Group: Members Posts: 830 Joined: 3-November 05 Member No.: 25526 |
I'm still working on it, I've been busy unfortunately... I've finished test1, done 2 samples in test 2, so I've got a ways to go. I'm hoping to finish tomorrow.
I have a question though, what's the point of the low anchor in listening tests? So you can establish a scale or...? Pardon the n00b question.  And when the test is finished, do you get to see individual results? (so you can see if you screwed up and so on) This post has been edited by Firon: Feb 3 2006, 03:32 |
|
|
|
|
Feb 3 2006, 10:38
Post
#20
|
|
|
Nero MPEG4 developer Group: Developer Posts: 1466 Joined: 22-September 01 Member No.: 8 |
Just a quick note - test is definitely extended until Sunday, 5th of February!
(I got more results, but still I need more for reliable analysis) - your feedback will be greatly appreciated! QUOTE I have a question though, what's the point of the low anchor in listening tests? So you can establish a scale or...? Pardon the n00b question. tongue.gif Yep - it is to establish a reliable scale, since IMHO most of the codecs could be very tied. QUOTE And when the test is finished, do you get to see individual results? (so you can see if you screwed up and so on) Yes, I will upload all individual results - and post the test keys, so everyone could check their own test locally, too. |
|
|
|
|
|
Lo-Fi Version | Time is now: 21st May 2013 - 20:08 |
