IPB

Welcome Guest ( Log In | Register )

15 Pages V   1 2 3 > »   
Closed TopicStart new topic
New Public Multiformat Listening Test (Jan 2014), Discussion on preparation
IgorC
post Dec 8 2013, 17:57
Post #1





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



Hi, Guys smile.gif

It's time to discuss an upcoming listening test. It will be a multiformat test at 96 kbps as a logical continuation to the last public AAC listening test from 2011 year.


There are few things we need to talk about:

1.Amount of codecs.
I think the possible amount of them is around 3-5.

2. Selection of codecs.
Please, propose a codecs You want to test. AAC, MP3, Opus, Vorbis, ...
We already had some discussion here http://www.hydrogenaudio.org/forums/index....c=92490&hl=
But since it was 2 years ago it will be good to start from scratch. lalala.gif
I will be updating this list "choice of codecs"


Also I think it will be more interesting to compare MP3 128-135 kbps and AAC/AoTuV 96-100 kbps. Probably a lot of people are interesting to trade off between compability/compression efficiency. But it's my point of view.

Let's discuss.

Also this time Steve Forte Rio will be conducting the test. He has helped a lot to organize and conduct the last AAC public test.
I just organize the discussion and help him. He will receive a results from a participants and keep a dialogue with them when the test will be open.

This post has been edited by IgorC: Dec 8 2013, 18:09
Go to the top of the page
+Quote Post
Kamedo2
post Dec 8 2013, 18:31
Post #2





Group: Members
Posts: 168
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.
Also, AAC is a mature, strong codec, have good compatibility, and especially Apple AAC encoder is known for its very high quality. Will Opus beat Apple?
And MP3 128kbps is also interesting. It has the best compatibility, and many people know what to expect from MP3 128kbps, so it's a good "anchor".

And we need a low anchor as well. FAAC 96kbps cbr has a bad quality, good for a low anchor.

Go to the top of the page
+Quote Post
LithosZA
post Dec 8 2013, 18:35
Post #3





Group: Members
Posts: 180
Joined: 26-February 11
Member No.: 88525



My choices would be:
MP3 (Helix or LAME or Fraunhofer) - Which one is better at 96Kbps?
AoTuV Vorbis
Apple AAC
Opus

All of them running at 96Kbps
EDIT: And FAAC 96Kbps low anchor

This post has been edited by LithosZA: Dec 8 2013, 18:37
Go to the top of the page
+Quote Post
Steve Forte Rio
post Dec 8 2013, 19:23
Post #4





Group: Members
Posts: 432
Joined: 4-October 08
From: Ukraine
Member No.: 59301



My choice:

1. MP3@128 kbps (96 kbps will surely lead to complete defeat). We can use Fraunhofer IIS MP3 Surround encoder, which is sometimes better than LAME at low bitrates beginning from 128 kbps, but I consider LAME much more popular, so maybe we should use LAME 3.99.5 -b 128 -q0.

2. OGG Vorbis aoTuV b6.03

3. QuickTime AAC TVBR.

4. WMA Pro

5. Opus 1.1, of course.

This post has been edited by Steve Forte Rio: Dec 8 2013, 19:26
Go to the top of the page
+Quote Post
Kamedo2
post Dec 8 2013, 19:30
Post #5





Group: Members
Posts: 168
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



I think we should increase the number of samples. More samples leads to more statistically valid results.
And I think we should choose the samples so that the average bitrate of the samples tested, and average bitrate of albums, is roughly equal, like I did;
http://www.hydrogenaudio.org/forums/index....howtopic=100896
If the average bitrate of albums is 96k and the average bitrate of tested samples is 144k, the corpus is overrepresented by critical samples.
Go to the top of the page
+Quote Post
eahm
post Dec 8 2013, 19:33
Post #6





Group: Members
Posts: 882
Joined: 11-February 12
Member No.: 97076



Thank you for organizing this, my choices:

AAC (Apple/qaac)

AAC (Fraunhofer/fhgaacenc)

AAC (Fraunhofer/fdkaac)

Opus (1.1)

Vorbis (libvorbis 1.3.3)

Vorbis (aoTuV b6.03)

WMA Standard

WMA Pro


Don't care about MP3, don't care about MPC.

edit: good observation from IgorC, let me [i] the "optional" ones. I don't know much about WMA, just want to see how it performs even if nothing changed in the last few years (I'm not even really sure about this, Microsoft is a mess)? Is Stardard or Pro more compatible?

This post has been edited by eahm: Dec 8 2013, 19:46
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 19:40
Post #7





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



eahm,

OK. But some obseravtions.

We already know that Apple was a best AAC encoder. It represents all AAC format very well. No need to test again FhG encoder.
The same for Vorbis. Only aoTuv.

We can't test 10 codecs. smile.gif . The optimal number is 3-5.

Guys, correct me if I'm wrong.

This post has been edited by IgorC: Dec 8 2013, 19:41
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 19:48
Post #8





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (Kamedo2 @ Dec 8 2013, 14:31) *
I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.

Do You mean include Opus 1.0 and 1.1?

QUOTE (Kamedo2 @ Dec 8 2013, 14:31) *
Also, AAC is a mature, strong codec, have good compatibility, and especially Apple AAC encoder is known for its very high quality. Will Opus beat Apple?

Good question. Let's see.

QUOTE (Kamedo2 @ Dec 8 2013, 14:31) *
And MP3 128kbps is also interesting. It has the best compatibility, and many people know what to expect from MP3 128kbps, so it's a good "anchor".

And we need a low anchor as well. FAAC 96kbps cbr has a bad quality, good for a low anchor.

Agree.



Go to the top of the page
+Quote Post
Kohlrabi
post Dec 8 2013, 19:49
Post #9





Group: Super Moderator
Posts: 953
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



Opus, QT-AAC, Musepack and Vorbis (AoTuV). Musepack really deserves to be tested again vs. other modern codecs.

This post has been edited by Kohlrabi: Dec 8 2013, 19:54


--------------------
Audiophiles live in constant fear of jitter.
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 19:56
Post #10





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



I would like to point to one comment of member Gecko. He has participated in previous tests:

http://www.hydrogenaudio.org/forums/index....st&p=780195
QUOTE
+1 for keeping the number of codecs small. Consider only three perhaps? Four is stretching it. I really struggled with the five in the last test (AAC @ ~96 kbps [July 2011]).
Go to the top of the page
+Quote Post
Garf
post Dec 8 2013, 19:57
Post #11


Server Admin


Group: Admin
Posts: 4853
Joined: 24-September 01
Member No.: 13



Just to be difficult, I propose 80kbps instead of 96kbps (easier for testers), and Apple LC-AAC, Apple HE-AAC, FhG (libfdk) LC-AAC, FhG (libfdk) HE-AAC, Opus 1.1, Vorbis aoTuV.

The AAC encoders could/should be a seperate pre-test, especially if FhG wants to send in a newer encoder than what is in libfdk. I'd favor libfdk over anything else AAC as it's used a lot together with ffmpeg now.

Edit: This isn't 100% a serious suggestion, but I want people to think about some things.

This post has been edited by Garf: Dec 8 2013, 20:04
Go to the top of the page
+Quote Post
o-l-a-v
post Dec 8 2013, 19:59
Post #12





Group: Members
Posts: 117
Joined: 18-July 12
From: Norway
Member No.: 101545



Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).
MPC and WMA is not interesting imo
Go to the top of the page
+Quote Post
Kamedo2
post Dec 8 2013, 20:00
Post #13





Group: Members
Posts: 168
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (IgorC @ Dec 9 2013, 03:48) *
QUOTE (Kamedo2 @ Dec 8 2013, 14:31) *
I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.

Do You mean include Opus 1.0 and 1.1?

The latest one, Opus 1.1. Testing Opus 1.0 is likely to lead to the redundant duplicate of http://listening-tests.hydrogenaudio.org/igorc/results.html and http://www.hydrogenaudio.org/forums/index....showtopic=97913
Go to the top of the page
+Quote Post
greynol
post Dec 8 2013, 20:03
Post #14





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (Kohlrabi @ Dec 8 2013, 10:49) *
Musepack really deserves to be tested again vs. other modern codecs.

Doesn't MPC really ony shine at settings that are intended to deliver transparent results which are like 3x what is being proposed for this test?


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 20:08
Post #15





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (Kamedo2 @ Dec 8 2013, 16:00) *
The latest one, Opus 1.1. Testing Opus 1.0 is likely to lead to the redundant duplicate of http://listening-tests.hydrogenaudio.org/igorc/results.html and http://www.hydrogenaudio.org/forums/index....showtopic=97913

Agree, only Opus 1.1
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 20:13
Post #16





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (Steve Forte Rio @ Dec 8 2013, 15:23) *
My choice:

1. MP3@128 kbps (96 kbps will surely lead to complete defeat). We can use Fraunhofer IIS MP3 Surround encoder, which is sometimes better than LAME at low bitrates beginning from 128 kbps, but I consider LAME much more popular, so maybe we should use LAME 3.99.5 -b 128 -q0.

2. OGG Vorbis aoTuV b6.03

3. QuickTime AAC TVBR.

4. WMA Pro

5. Opus 1.1, of course.

I think we should probably keep in mind both TVBR and CVBR. Because if TVBR will end up with ~94 kbps and other codecs at ~96-100 kbps then we probably should go to CVBR ~100 kbps. Anyway both Apple TVBR and CVBR are great.
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 20:18
Post #17





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (Garf @ Dec 8 2013, 15:57) *
Just to be difficult, I propose 80kbps instead of 96kbps (easier for testers), and Apple LC-AAC, Apple HE-AAC, FhG (libfdk) LC-AAC, FhG (libfdk) HE-AAC, Opus 1.1, Vorbis aoTuV.

The AAC encoders could/should be a seperate pre-test, especially if FhG wants to send in a newer encoder than what is in libfdk. I'd favor libfdk over anything else AAC as it's used a lot together with ffmpeg now.

Edit: This isn't 100% a serious suggestion, but I want people to think about some things.

Why not?

I agree that it will be interesting to see it. So let's see what people propose.

However also let's see where we're coming from. We've tested AAC encoders at 96 kbps and it's logical to test the best AAC encooder, Apple, against the rest of codecs.

Personally it's hard for me to do pre-test and then test, but, yeah, let's test in future or even now . It's all up to people decision.

This post has been edited by IgorC: Dec 8 2013, 20:18
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 20:20
Post #18





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (o-l-a-v @ Dec 8 2013, 15:59) *
Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).
MPC and WMA is not interesting imo


MP3 at 96 or 128 kbps?
Go to the top of the page
+Quote Post
o-l-a-v
post Dec 8 2013, 20:27
Post #19





Group: Members
Posts: 117
Joined: 18-July 12
From: Norway
Member No.: 101545



QUOTE (IgorC @ Dec 8 2013, 21:20) *
QUOTE (o-l-a-v @ Dec 8 2013, 15:59) *
Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).
MPC and WMA is not interesting imo


MP3 at 96 or 128 kbps?


Let's say 128. If I were to convert music in low bitrate mp3, I would never even think about going lower than that.
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 20:36
Post #20





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



I will participate in this test too so here is my wishlist.

1. MP3 128 kbps. LAME 3.99.5 -V5 (high anchor)
2. MP3 96 kbps . LAME ABR is better than VBR (?)
3. Apple AAC 96 kbps (QAAC highest quality TVBR or CVBR.)
4. Opus 1.1 vbr 96 kbps.
5. Vorbis AoTuv 6.0.3 vbr 96 kbps.

low anchor - FAAC CBR 96 kbps, as Kamedo2 said. It has a reasonably low quality.
We had also discussion to have 2 low anchors. Actually low anchor and low-middle anchor. It's good to have two acnhors to validate results. Low-middle anchor should be better than low anchor.
It can be: FAAC 64 (low anchor) and FAAC 96 (low-middle anchor).

This post has been edited by IgorC: Dec 8 2013, 20:40
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 21:02
Post #21





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (Kamedo2 @ Dec 8 2013, 15:30) *
I think we should increase the number of samples. More samples leads to more statistically valid results.
And I think we should choose the samples so that the average bitrate of the samples tested, and average bitrate of albums, is roughly equal, like I did;
http://www.hydrogenaudio.org/forums/index....howtopic=100896
If the average bitrate of albums is 96k and the average bitrate of tested samples is 144k, the corpus is overrepresented by critical samples.

More than 20 samples? hm, maybe, I don't know.
20 is already enough high number. During the last we've waited a little bit more than month to get enough results.


What do other think about it?
Go to the top of the page
+Quote Post
IgorC
post Dec 8 2013, 22:10
Post #22





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (Steve Forte Rio @ Dec 8 2013, 15:23) *
... so maybe we should use LAME 3.99.5 -b 128 -q0.

What about vbr. V5 ~128 kbps?
Go to the top of the page
+Quote Post
TheBashar
post Dec 8 2013, 22:46
Post #23





Group: Members
Posts: 39
Joined: 4-November 01
Member No.: 401



What I'd like to see in the test:
  • LAME MP3 -V5
  • Apple AAC @96
  • Opus 1.1 @96
  • Muespack @96

Go to the top of the page
+Quote Post
lvqcl
post Dec 8 2013, 22:58
Post #24





Group: Developer
Posts: 3208
Joined: 2-December 07
Member No.: 49183



1) Musepack at 96kbps will have lowpass ~14kHz. That's too low IMHO.

2) Bitrates of WMA VBR:
WMA Std Q50: 74 kbps;
WMA Std Q75: 115 kbps;
WMA Pro Q25: 83 kbps;
WMA Pro Q50: 113 kbps.

None of them is close to the target bitrate.

3) IMHO for LAME 3.99.x low-bitrate VBR is better than low-bitrate ABR.
Go to the top of the page
+Quote Post
eahm
post Dec 8 2013, 23:41
Post #25





Group: Members
Posts: 882
Joined: 11-February 12
Member No.: 97076



For WMA you can also set the bit rate at 96.

OT
lvqcl, where did you get the bit rates for the WMA quality settings? Do you have all of them (Std and Pro)?

This post has been edited by eahm: Dec 8 2013, 23:42
Go to the top of the page
+Quote Post

15 Pages V   1 2 3 > » 
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th April 2014 - 17:55