IPB

Welcome Guest ( Log In | Register )

15 Pages V  < 1 2 3 4 5 > »   
Closed TopicStart new topic
New Public Multiformat Listening Test (Jan 2014), Discussion on preparation
Kamedo2
post Dec 9 2013, 20:07
Post #51





Group: Members
Posts: 168
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (IgorC @ Dec 10 2013, 03:35) *
Well, a lot of suff is going on.

Kamedo2,
Until now one of the conditions of HA tests is "no less than 10 results per sample".
Please have a look through these 10 "full" contributors. http://listening-tests.hydrogenaudio.org/i...-a/results.html zip file.

Sadly some of them have got tired let's say after 10 samples and after that they have just rated the low anchor.

I said that "more than 10 results per sample is a bit overkill", but if many are 5.0, around 10 res/sample might be about right. I'll later try to dotplot it, rather than just average and CI95% errorbar, so that I can have a better grasp of the score distribution.
Go to the top of the page
+Quote Post
IgorC
post Dec 9 2013, 20:19
Post #52





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (Kamedo2 @ Dec 9 2013, 16:07) *
I said that "more than 10 results per sample is a bit overkill", but if many are 5.0, around 10 res/sample might be about right...

That's why a good low anchor with acceptable quality has to be included. So people won't just rate a low anchor and submit other results as unrated 5.0. I admit, our low anchor ffmpeg at 96 kbps was very bad.

faac at 96 kbps (CBR) should do better job.

This post has been edited by IgorC: Dec 9 2013, 20:21
Go to the top of the page
+Quote Post
Kamedo2
post Dec 9 2013, 20:22
Post #53





Group: Members
Posts: 168
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



I don't think the Apple AAC encoders shine at 80k. They shine at 96k. At 80k, I'm almost sure Opus would beat the AAC-LC encoders.
And 96k is likely to be the bitrate many people would use. For smartphones with speed limitation(128kbps is common), broadcasting at 96k seems natural.
http://www.hydrogenaudio.org/forums/index....howtopic=102876
Go to the top of the page
+Quote Post
IgorC
post Dec 9 2013, 20:23
Post #54





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



Kamedo2,

If here we're are at least 10 listeners then we can rise a number of samples or even work at what Jean-Marc has proposed.

This post has been edited by IgorC: Dec 9 2013, 20:24
Go to the top of the page
+Quote Post
IgorC
post Dec 9 2013, 20:54
Post #55





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (C.R.Helmrich @ Dec 9 2013, 07:20) *
I totally agree. Is there no interest in lower bit-rates? 48 kbps perhaps?

Everything is discussible. If people will prefer 48 kbps or any other rate, so be it.
Go to the top of the page
+Quote Post
eahm
post Dec 9 2013, 21:25
Post #56





Group: Members
Posts: 882
Joined: 11-February 12
Member No.: 97076



QUOTE (Kamedo2 @ Dec 9 2013, 12:22) *
I don't think the Apple AAC encoders shine at 80k. They shine at 96k. At 80k, I'm almost sure Opus would beat the AAC-LC encoders.
And 96k is likely to be the bitrate many people would use. For smartphones with speed limitation(128kbps is common), broadcasting at 96k seems natural.
http://www.hydrogenaudio.org/forums/index....howtopic=102876

Damn it poor IgorC, I am afraid Kamedo2 is right, AAC shines at 96, I was too excited to see it against Opus at 80 that I didn't even think about the older test already performed and AAC really didn't change much since.

Sorry again, revert back and last, definitive choice:

AAC-LC (Apple/qaac) VBR 96 kbps

AAC-LC (Fraunhofer/fhgaacenc) VBR 96 kbps

AAC-LC (Fraunhofer/fdkaac) VBR 96 kbps

Opus (1.1) VBR 96 kbps
Go to the top of the page
+Quote Post
Serge Smirnoff
post Dec 9 2013, 21:27
Post #57





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



I'm interested in Opus and fdk-AAC as they are new players with high popularity potential. And how they compete with Apple-AAC @96 or @80. So:
1. Opus
2. fdk-AAC
3. Apple-AAC


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
Steve Forte Rio
post Dec 9 2013, 21:45
Post #58





Group: Members
Posts: 432
Joined: 4-October 08
From: Ukraine
Member No.: 59301



QUOTE (IgorC @ Dec 8 2013, 17:13) *
I think we should probably keep in mind both TVBR and CVBR. Because if TVBR will end up with ~94 kbps and other codecs at ~96-100 kbps then we probably should go to CVBR ~100 kbps. Anyway both Apple TVBR and CVBR are great.


We'll see about that (bitrate for TVBR). But anyway TVBR is recommended mode and it is mostly used. So results for another algorithms will not be so useful and informative. This should be a decisive argument.

This post has been edited by Steve Forte Rio: Dec 9 2013, 21:45
Go to the top of the page
+Quote Post
IgorC
post Dec 9 2013, 22:01
Post #59





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



eahm, Serge Smirnoff,

Will update later.

Has FhG fdk VBR 80 or 96 kbps mode?


eahm,
QUOTE (eahm @ Dec 9 2013, 17:25) *
...
AAC-LC (Fraunhofer/fhgaacenc) VBR 96 kbps
...

Can I ask You what do You expect from testing FhG again? fhgaacenc uses the same FhG Winamp encoder that we have tested in the last public listening test. It came 2ēd, right after Apple AAC. The result will be same.
Anyway it's your choice.


QUOTE (Steve Forte Rio @ Dec 9 2013, 17:45) *
QUOTE (IgorC @ Dec 8 2013, 17:13) *
I think we should probably keep in mind both TVBR and CVBR. Because if TVBR will end up with ~94 kbps and other codecs at ~96-100 kbps then we probably should go to CVBR ~100 kbps. Anyway both Apple TVBR and CVBR are great.


We'll see about that (bitrate for TVBR). But anyway TVBR is recommended mode and it is mostly used. So results for another algorithms will not be so useful and informative. This should be a decisive argument.

Agree

This post has been edited by IgorC: Dec 9 2013, 22:03
Go to the top of the page
+Quote Post
greynol
post Dec 9 2013, 22:15
Post #60





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (IgorC @ Dec 9 2013, 13:01) *
It came 2ēd, right after Apple AAC. The result will be same.

First, Apple did not win.

Second, what makes you so sure the results will be identical? I can provide random data for the contenders and not have my results tossed so long as I don't do anything stupid in ranking the anchors with respect to the contenders. These tests are subjective, after all. Also, even if people who participated in both tests gave the same results (don't hold your breath on that) what about people who participated in one test, but not the other?

Third, I see no reason to dismiss this test which don't give the exact* same result:
http://www.hydrogenaudio.org/forums/index....howtopic=100525

(*) Maybe it did if you actually pay respect to the error bars for both tests (just between Apple and FhG, they are actually statistically tied over-all in both tests!).

This post has been edited by greynol: Dec 9 2013, 22:41


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
C.R.Helmrich
post Dec 9 2013, 22:20
Post #61





Group: Developer
Posts: 681
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



QUOTE (Garf @ Dec 9 2013, 14:36) *
If the Winamp encoder in whatever the latest Winamp release was is current, that's great.

Are there relevant differences between the libfdk_aac that you sold to Google and this encoder?

Yes, Winamp 5.666 has the latest AAC encoder quality-wise, no new quality tunings which are ready for release.

I'll let you know when quality is improved. Or just ask smile.gif

The Winamp/Sonnox/... encoder has a completely different code-base than fdkaac and is a bit better tuned, especially for VBR.

Chris


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
jensend
post Dec 9 2013, 22:42
Post #62





Group: Members
Posts: 140
Joined: 21-May 05
Member No.: 22191



greynol, what on earth are you getting at? Are you just trying to be a devil's advocate or troll the process?

Apple absolutely did win. p=.002.

IgorC didn't claim the individual test results will be identical, he claimed the overall ordering will be the same. Though exact scores would vary if you ran that test a thousand times, since Apple won the test by a statistically significant margin, it is to be expected that it would place first in a large majority of repeated tests.

There are tons of good reasons to value the last HA listening test higher than SoundExpert's unsound methodology. That has been discussed more than plenty already.

The last test had five codecs incl. anchor and several of those who did the test said it was too much. Now, at a higher bitrate, people are now trying to throw in every AAC encoder and mode under the sun. You won't get enough worthwhile participation to get any useful information out of the test if you do that.

This post has been edited by jensend: Dec 9 2013, 22:44
Go to the top of the page
+Quote Post
greynol
post Dec 9 2013, 22:46
Post #63





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (jensend @ Dec 9 2013, 13:42) *
Apple absolutely did win. p=.002.

I see it now, CVBR, not TVBR, though a retest could easily go the other way.

QUOTE (jensend @ Dec 9 2013, 13:42) *
There are tons of good reasons to value the last HA listening test higher than SoundExpert's unsound methodology. That has been discussed more than plenty already.

I've done my share of criticizing SE, TYVM. You apparently aren't familiar with the details of test I linked as none of the comments I've seen about SE's unsound methodology apply.

This post has been edited by greynol: Dec 9 2013, 23:04


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
binaryhermit
post Dec 9 2013, 22:53
Post #64





Group: Members
Posts: 36
Joined: 23-April 09
Member No.: 69212



I have to say that I'm most interested in Opus, Vorbis (aotuv), and mp3 (lame) around 80-96 kbps.
Go to the top of the page
+Quote Post
IgorC
post Dec 9 2013, 23:06
Post #65





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



Greynol,

Let me give You one example.
There were 3 public tests. Roberto's Sebastian's and mine. All of them have shown that HE-AAC is better than Vorbis.

The same way if there will be a new public test (well done, with enough people, with correct methodic). Apple AAC will be always be on top of other AAC encoders.
There can be variations but an average score of Apple will be always on top.

This post has been edited by IgorC: Dec 9 2013, 23:08
Go to the top of the page
+Quote Post
eahm
post Dec 9 2013, 23:26
Post #66





Group: Members
Posts: 882
Joined: 11-February 12
Member No.: 97076



QUOTE (IgorC @ Dec 9 2013, 14:01) *
Can I ask You what do You expect from testing FhG again? fhgaacenc uses the same FhG Winamp encoder that we have tested in the last public listening test. It came 2ēd, right after Apple AAC. The result will be same.
Anyway it's your choice.

I remember Chris said he/they did some tuning in the low bit rates, even on the very last release (.16). Wonder if anything changed up to ~96 kbps.

This post has been edited by eahm: Dec 9 2013, 23:26
Go to the top of the page
+Quote Post
Serge Smirnoff
post Dec 9 2013, 23:26
Post #67





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (IgorC @ Dec 10 2013, 01:01) *
Has FhG fdk VBR 80 or 96 kbps mode?

fdkaac.exe -m 1 (libfdk-aac 3.4.12) gave me 94.1 for the whole Shpongle album

This post has been edited by Serge Smirnoff: Dec 9 2013, 23:41


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
eahm
post Dec 9 2013, 23:36
Post #68





Group: Members
Posts: 882
Joined: 11-February 12
Member No.: 97076



Here is a document I made while ago about kbps and settings: http://pastebin.com/4HiD8juZ

This post has been edited by eahm: Dec 9 2013, 23:43
Go to the top of the page
+Quote Post
testyou
post Dec 10 2013, 00:11
Post #69





Group: Members
Posts: 90
Joined: 24-September 10
Member No.: 84113



I'm interested in:
Opus --bitrate 96
Apple -V 36 (target is ~96)
LAME -V 5 (target is ~128)

This post has been edited by testyou: Dec 10 2013, 00:13
Go to the top of the page
+Quote Post
IgorC
post Dec 10 2013, 02:13
Post #70





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (greynol @ Dec 9 2013, 19:27) *
The whole point of this is that FhG could beat Apple in a re-match, especially when it tied Apple in a perfectly valid test, personal attacks against me aside.

Well, I see You're not familiar enough well with the results http://listening-tests.hydrogenaudio.org/i...ous/results.zip

And that's why You can't imagine why somebody (me in this case) can be sure that "FhG can't beat Apple in a re-match". (always speaking of 96 kbps, LC-AAC, stereo, 44.1 kHz)

If You analyse the results enough close, You will see that there are two group of people. First, the bigger one, who have prefer Apple encoder over FhG with statistically important differences. And the second (a smaller one) who have very slightly but still prefer FhG encoder over Apple's but there was no statistically valid difference. Someone could argue about sample selection. Well, it was a very representative set of 20 (!) samples which were automatically randomly picked from different groups of samples.


And now You get the SE test as argument.
Accroding to HA public listening test 2011 there was no a single person who has prefer Nero over any other AAC encoder (excluding of course the low anchor). Not a single person.
But according to SE test, the average score for Nero was higher than Apple's. Sorry, this can't be right. No.
And I can even explain to You what has happened. Nero uses a lot of long frames that makes it quite good for tonal samples, SE is generally consist of these kind of samples and doesn't have any kind of other problematic samples.
That's why Nero does that good. While HA's balanced set of samples consists of different kind of samples, including a transients where Nero performs very bad.


P.S. Anyway I don't mind to include FhG. In fact in my opinion FhG has the best in class quality among HE-AAC encoders at 64 kbps. And at some point it will shine in some future public test at those rate.

This post has been edited by IgorC: Dec 10 2013, 02:31
Go to the top of the page
+Quote Post
greynol
post Dec 10 2013, 02:26
Post #71





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



SE's test results didn't agree with yours so they must be wrong, the graphical representation of the overall results of your test doesn't do justice to the test data, and you feel there is no point in looking into it any further; I get it.

The next time I see data from two different tests that aren't in agreement, I'll just ask you to point me in the right direction. The scientific method of repeating an experiment to confirm the results be damned.

This post has been edited by greynol: Dec 10 2013, 02:31


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
greynol
post Dec 10 2013, 02:28
Post #72





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (IgorC @ Dec 9 2013, 17:13) *
P.S. Anyway I don't mind to include FhG.

I think that would be great.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
jensend
post Dec 10 2013, 02:39
Post #73





Group: Members
Posts: 140
Joined: 21-May 05
Member No.: 22191



Ah yes, editing your posts after people have replied in an effort to cover your tracks. Classy move, that. "'Hey, wasn't Trotsky in that photo last time I saw it?' 'No, Comrade. Nothing to see here.'"

QUOTE (greynol @ Dec 9 2013, 15:27) *
You're dismissing the SE result which doesn't exactly agree. I guess I'll have to take you at your word as to why that is.
Rather than taking my word for it, you could actually read the thread you linked to, along with any number of other threads on the subject.

AFAIK the only thing done differently for that test as compared to most SE tests was to quit with the artifact amplification. Though that was by far the most obvious problem with SE's methods, there have been many other concerns. A few of those include only having five samples (which have been considered unrepresentative), fine-tuning the rate settings to try to achieve a target on those five samples rather than allowing VBR encoders to make intelligent rate decisions given their target, not following normal ABC/HR or other established test protocol, and some statistical methods concerns.

This isn't about people "having an axe to grind." When you leap to uninformed conclusions and criticize people in an uncivil fashion it's pretty absurd for you to turn around and be hypersensitive - "Oh my, somebody called me out on my bad behavior! how rude of them!"

Without bothering to actually read the test results to find out whether there was a discrepancy, and without bothering to ask about in a civil fashion, you leapt to telling the person who organized and did so much of the work on that test that he was lying about its results. You disparaged scientific statistical methods in an incoherent fashion. Rather than considering what people have said about SE, including on the page you linked, you leapt to telling me I'm just ignorant. And you didn't expect to get any pushback on any of this unless someone "had an axe to grind"? Is that because you expect to usually be able to just bin anybody who dares to disagree with you?
Go to the top of the page
+Quote Post
greynol
post Dec 10 2013, 03:50
Post #74





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



I merely questioned (and am still questioning) a blanket claim. The differences between encoders simply isn't as great as all this posturing implies. Meanwhile we are all to dismiss another set of test results as completely worthless. I for one find the tight grouping of samples encoded with FhG as being worthwhile to at least a few regular and outspoken members. Perhaps this might be taken into consideration for this current round of testing.

I will compromise my request so that we don't have to worry about the possibility of redundant results:
QAAC 80 kbits TVBR
Whichever FhG AAC encoder that will likely get used by the greatest number of people in a post-Winamp world
Opus
Any other non-AAC format that stands a chance at this bitrate.

This post has been edited by greynol: Dec 10 2013, 19:54
Reason for edit: Combined posts, removed a reply to a comment that was binned.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
testyou
post Dec 10 2013, 04:15
Post #75





Group: Members
Posts: 90
Joined: 24-September 10
Member No.: 84113



QUOTE (greynol @ Dec 9 2013, 18:50) *
QAAC 80 kbits TVBR

I think this would be qaac -V 27.
QUOTE
Whichever FhG AAC encoder that will likely get used by the greatest number of people in a post-Winamp world

fdkaac?
QUOTE
Any other non-AAC format that stands a chance at this bitrate.

I'm not sure what else there would be. I remember vorbis being lower than nero here.
Go to the top of the page
+Quote Post

15 Pages V  < 1 2 3 4 5 > » 
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 18th April 2014 - 01:15