Public MP3 Listening Test @ 128 kbps - FINISHED |
  |
Public MP3 Listening Test @ 128 kbps - FINISHED |
 Nov 27 2008, 03:42
Post
#151
|
|
|
Group: Members Posts: 42 Joined: 18-May 05 Member No.: 22125  |
@/mnt
Please; Could you perform the test again with the same samples? but now with this command line: -V122 -X2 -HF2 -SBT500 -TX0 -C0 Copy and paste EXACTLY as this is written. Thanks. This post has been edited by level: Nov 27 2008, 03:45 |
 |
 
|
|
Nov 27 2008, 04:00
Post
#152
|
|
|
Group: Members Posts: 88 Joined: 16-December 03 Member No.: 10473 |
QUOTE (Sebastian Mares @ Nov 26 2008, 13:50)  Full results are online now. Hopefully I didn't miss anything. Thanks so much everyone! Sebastian, I sympathize with you, after putting forth the effort and conducting a very nice test, there are so many arguments! I had agonized over whether the switch from Lame 3.97 to Lame 3.98 was worth it, and the answer appears to be yes, quite likely, very much so, for some types of samples. So I will keep using lame 3.98. Thanks to all of the lame developers for your good work. The helix results look very interesting. I am not much of an expert. I downloaded the hellix encoder from rarewares, but I don't see a guide on how to set the switches. I like to try new things just for fun. I'm odd that way. SO, if someone could help me, if I want to use helix in the low-to-mid 200 kbps range (which is what I use nowadays since memory and hard disk space are so much cheaper), what would be the standard-issue switch settings? I assume I could use it with EAC (which is what I use)? Is there a link to a guide for setting the switches? I can't find the documentation. For me a decent encoder at this range is fine. I'm interested in the speed and trying something new. And most of all, thank you SO MUCH to everyone who participated in setting up and conducting the testing. This post has been edited by Steve999: Nov 27 2008, 04:01 |
|
|
|
|
Nov 27 2008, 04:13
Post
#153
|
|
 Group: Super Moderator Posts: 9366 Joined: 1-April 04 Member No.: 13167 |
QUOTE (Steve999 @ Nov 26 2008, 19:00) I had agonized over whether the switch from Lame 3.97 to Lame 3.98 was worth it, and the answer appears to be yes, quite likely, very much so, for some types of samples. There really isn't much legitimate to argue about. The problem is that people seem to want to extrapolate the results for a single sample to the entire genre of that sample; and it extends beyond this particular thread. Without conducting tests on additional samples and showing correlation such claims simply aren't credible. -------------------- Everything sounds the same until it is proven otherwise.
|
|
|
|
|
Nov 27 2008, 09:45
Post
#154
|
|
|
Group: Members Posts: 454 Joined: 15-November 04 Member No.: 18143 |
QUOTE (Neasden @ Nov 27 2008, 01:53) Is this basically the end of mp3 by reaching its maximum level of quality being that all codecs are tied? Define "end of mp3". Judging from this listening test clearly there's room for quality improvement as some encoders perform clearly better on some samples and worse on others. |
|
|
|
|
Nov 27 2008, 10:29
Post
#155
|
|
 Group: Members Posts: 1303 Joined: 14-September 05 From: Helsinki, Finland Member No.: 24472 |
QUOTE (Alexxander @ Nov 27 2008, 10:45) Judging from this listening test clearly there's room for quality improvement as some encoders perform clearly better on some samples and worse on others. That is one of the reasons I started the sample specific threads. I hope the threads would help the developers (mainly the LAME developers who are active here) to better understand what kind of problems the testers noticed. I have added new threads for the samples #3 and #4. More is coming. http://www.hydrogenaudio.org/forums/index.php?showforum=40 -------------------- http://listening-tests.freetzi.com
|
|
|
|
|
Nov 27 2008, 10:39
Post
#156
|
|
 Group: Super Moderator Posts: 4887 Joined: 12-August 04 From: Exeter, UK Member No.: 16217 |
QUOTE (Neasden @ Nov 27 2008, 00:53) Is this basically the end of mp3 by reaching its maximum level of quality being that all codecs are tied?  You seem to enjoy throwing inflammatory one-liners into this thread. Nice avatar BTW.QUOTE (Steve999 @ Nov 27 2008, 03:00) Thanks so much everyone! Sebastian, I sympathize with you, after putting forth the effort and conducting a very nice test, there are so many arguments! I suspect that Sebastian is very pleased about the debate that his test has created.
-------------------- I'm on a horse.
|
|
|
|
|
Nov 27 2008, 11:02
Post
#157
|
|
|
Group: Members Posts: 2297 Joined: 9-October 05 From: Dormagen, Germany Member No.: 25015 |
QUOTE (guruboolez @ Nov 27 2008, 00:50) QUOTE (halb27 @ Nov 26 2008, 22:47) There is a chance that users can get significant encoder differentiation for his individual needs. A very small chance. ... You're right, as you can read in my last post. BTW I guess I can see what our differences are about: I care about other things than you do. For instance it's not a question to me whether encoder X is better than encoder Y, in the first place because of the difficulty in the meaning of 'better'. I think most members here including you are happy when there would have been a good ordering in overall results with well-separated confidence intervals. While in a formal sense this is meaningful (also to me) this is not the most important thing to me. If for instance the best encoder in this sense would have had modest scores on samples 3, 4, 7, 10 (those samples with very special importance to me - see my last post) this encoder would not be interesting to me. In the end it comes to that I like only results on those samples I care about (1-8, 10, 13, with special emphasis on 3, 4, 7, 10), and I'd like to see results which are pretty close to 5 (cause only in this cases there is a strong agreement among all the participants that the outcome of the particular encoder on this sample is good). Sure 'pretty close to 5' is a weak statement, but at 128 kbps I can't expect to give a strong formulation in case I want to get some answer. And I do want an answer cause the question about saving storage space has arrived at me as I plan to use a Meizu M6 SL DAP (guess I'll get it for Xmas) which unfortunatley has only 8 GB. This test came in quite handy, and I was surprised I could only ABX pretty few encoders on pretty few samples (not mentioning the very low anchor). So my very high quality demands I had so far (and which I could easily have with my 40 GB iRiver H140) are inappropriate. So I take these test results as a starting point for my encoder choice, I have to reconsider those problem samples I care about (I still do, but I will drop some of minor practical significance to me and will be content when an encoder gets at the non-obvious issue level status), and once I have found a good candidate based on this (shouldn't be too hard), I will do intensive listening tests with 'normal' music of my favorite kind. This post has been edited by halb27: Nov 27 2008, 11:16 -------------------- lame3100k -V0 --cvbr 9
|
|
|
|
|
Nov 27 2008, 11:04
Post
#158
|
|
 Group: Members Posts: 3621 Joined: 14-May 03 From: Bad Herrenalb Member No.: 6613 |
QUOTE (Synthetic Soul @ Nov 27 2008, 10:39) QUOTE (Steve999 @ Nov 27 2008, 03:00) Thanks so much everyone! Sebastian, I sympathize with you, after putting forth the effort and conducting a very nice test, there are so many arguments! I suspect that Sebastian is very pleased about the debate that his test has created.Of course. Otherwise, the test would be pretty much pointless and would give me the feeling that nobody cares. 
-------------------- http://listening-tests.hydrogenaudio.org/sebastian/
|
|
|
|
|
Nov 27 2008, 11:48
Post
#159
|
|
 Group: Members Posts: 691 Joined: 22-April 06 Member No.: 29877 |
QUOTE (level @ Nov 27 2008, 03:42) @/mnt Please; Could you perform the test again with the same samples? but now with this command line: -V122 -X2 -HF2 -SBT500 -TX0 -C0 Copy and paste EXACTLY as this is written. Thanks. Since am busy today, i can only do a few tests. Anyway, did a few tracks. CODE foo_abx 1.3.3 report foobar2000 v0.9.6 beta 3 2008/11/27 10:29:05 File A: C:\Temp\Helix Mp3\V122\Metallica - Ride The Lightning\04. Fade To Black.mp3 File B: C:\Rips\Metallica - Ride The Lightning\04. Fade To Black.flac 10:29:05 : Test started. 10:30:15 : 00/01 100.0% 10:30:29 : 01/02 75.0% 10:30:33 : 02/03 50.0% 10:30:39 : 03/04 31.3% 10:30:45 : 04/05 18.8% 10:30:50 : 05/06 10.9% 10:30:55 : 06/07 6.3% 10:31:00 : 07/08 3.5% 10:31:05 : 08/09 2.0% 10:31:10 : 09/10 1.1% 10:31:14 : 10/11 0.6% 10:31:22 : 11/12 0.3% 10:31:49 : 12/13 0.2% 10:31:54 : 13/14 0.1% 10:32:06 : 14/15 0.0% 10:32:08 : Test finished. ---------- Total: 14/15 (0.0%) Warbling is almost gone at 3:55, but i think i spotted a preecho though after the drum snare. But alot better. CODE foo_abx 1.3.3 report foobar2000 v0.9.6 beta 3 2008/11/27 10:32:57 File A: C:\Rips\Metallica - Ride The Lightning\05. Trapped Under Ice.flac File B: C:\Temp\Helix Mp3\V122\Metallica - Ride The Lightning\05. Trapped Under Ice.mp3 10:32:57 : Test started. 10:33:17 : 01/01 50.0% 10:33:24 : 02/02 25.0% 10:33:31 : 03/03 12.5% 10:33:50 : 04/04 6.3% 10:34:01 : 05/05 3.1% 10:34:07 : 06/06 1.6% 10:34:22 : 07/07 0.8% 10:34:31 : 08/08 0.4% 10:34:39 : 09/09 0.2% 10:34:47 : 10/10 0.1% 10:34:55 : 11/11 0.0% 10:34:56 : Test finished. ---------- Total: 11/11 (0.0%) Gutiar warbling from the start is still there, but alot better. CODE foo_abx 1.3.3 report foobar2000 v0.9.6 beta 3 2008/11/27 10:36:14 File A: C:\Rips\Fear Factory - Digimortal\05. Linchpin.flac File B: C:\Temp\Helix Mp3\V122\Fear Factory - Digimortal\05. Linchpin.mp3 10:36:14 : Test started. 10:36:30 : 01/01 50.0% 10:36:37 : 02/02 25.0% 10:36:48 : 03/03 12.5% 10:36:58 : 04/04 6.3% 10:37:05 : 05/05 3.1% 10:37:12 : 06/06 1.6% 10:37:20 : 07/07 0.8% 10:37:27 : 08/08 0.4% 10:37:32 : 09/09 0.2% 10:37:40 : 10/10 0.1% 10:37:48 : 11/11 0.0% 10:37:56 : 12/12 0.0% 10:37:57 : Test finished. ---------- Total: 12/12 (0.0%) Hardly any improvement with sample 11, warbling and smearing. -------------------- "I never thought I'd see this much candy in one mission!"
|
|
|
|
|
Nov 27 2008, 12:32
Post
#160
|
|
 Group: Members Posts: 1015 Joined: 4-March 03 From: Singapore Member No.: 5312 |
Maybe LAME 3.98.2 is far into the realm of diminishing returns at this stage (for 128 kbps), when taking into account the tradeoff in encoding speed.
Does LAME have to trade its quality to reach Helix's speed? Does Helix have to trade its speed for LAME's quality (consistency and less metal artifacting)? |
|
|
|
|
Nov 27 2008, 13:44
Post
#161
|
|
 Group: Members Posts: 304 Joined: 9-August 02 From: SoFo Member No.: 3002 |
QUOTE -V122 -X2 -HF2 -SBT500 -TX0 -C0 where did you get that command line from? I've been asking about this before, we should explore the existing switches to come up with the best possible. It's the best we can do, since tuning Helix doesn't seem like an option. This post has been edited by DigitalDictator: Nov 27 2008, 13:44 |
|
|
|
|
Nov 27 2008, 14:13
Post
#162
|
|
|
Group: Developer (Donating) Posts: 2332 Joined: 28-June 02 From: Argentina Member No.: 2425 |
Two things to consider is that this where all problem samples and that people where paying attention as much as they can to hear errors.
I think in normal conditions, people would not be so picky when listening. Like, when i'm listening to music and working, i don't pay not half the attention to the music. Same if i'm in a meeting with friends, and we have background music. I think in "reality" , encoders sound better than on tests like this one. -------------------- MAREO: http://www.webearce.com.ar
|
|
|
|
|
Nov 27 2008, 17:53
Post
#164
|
|
|
Group: Banned Posts: 185 Joined: 1-July 08 Member No.: 55148 |
QUOTE (Synthetic Soul @ Nov 27 2008, 06:39) You seem to enjoy throwing inflammatory one-liners into this thread. Nice avatar BTW.Did you find my one-liner inflammatory? I don't think it is. I think perhaps it was a naive question that was interpreted as sarcasm (which is quite used by other members, often unremarked) but I really meant in this way: if there were more room for MP3 improvements, because of the known limitations bounded to the format. |
|
|
|
|
Nov 27 2008, 18:47
Post
#165
|
|
|
Group: Members Posts: 195 Joined: 29-May 07 Member No.: 43837 |
Does this latest test suggest that the recommended settings for LAME be changed (or any other part of the ha wiki)? And one specific point: is -v2 still the recommended minimum setting for transparency for most listeners in good listening environments etc., or should that be lower now?
Thanks for any comments (and thanks to Sebastian Mares for conducting the test). |
|
|
|
|
Nov 27 2008, 19:04
Post
#166
|
|
|
Group: Members Posts: 64 Joined: 18-September 08 From: Sparta, Ontario Member No.: 58419 |
QUOTE (kwanbis @ Nov 27 2008, 08:13) Two things to consider is that this where all problem samples and that people where paying attention as much as they can to hear errors. I think in normal conditions, people would not be so picky when listening. Like, when i'm listening to music and working, i don't pay not half the attention to the music. Same if i'm in a meeting with friends, and we have background music. I think in "reality" , encoders sound better than on tests like this one. True enough, but do you really want your encoder to let you down when you do decide to pay specific attention? Imo, pick your encoder and bitrate on very critical listening, and tough samples. At least then you can be reasonably confident that you won't hear artifacts when you do crank it up. |
|
|
|
|
Nov 27 2008, 19:38
Post
#167
|
|
|
Group: Super Moderator Posts: 4887 Joined: 12-August 04 From: Exeter, UK Member No.: 16217 |
QUOTE (Neasden @ Nov 27 2008, 16:53) QUOTE (Synthetic Soul @ Nov 27 2008, 06:39) You seem to enjoy throwing inflammatory one-liners into this thread. Nice avatar BTW.QUOTE (Neasden @ Nov 24 2008, 22:00) Does that make Helix the new recommended MP3 encoder, or has it to be LAME because it's open source? Edit: Both are open source. QUOTE (Neasden @ Nov 24 2008, 22:25) They are all techincally tied, but Helix outperformed all of them. Also, the encoding speed compared to LAME is absurd faster. Could these two arguments qualify Helix for the new recommended MP3 encoder? (LAME being the second recommended) QUOTE (Neasden @ Nov 26 2008, 17:50) * Facts ... Helix performed a bit better than LAME in this test. QUOTE (Neasden @ Nov 26 2008, 18:51) I have nothing to add, like I am not saying Helix is BETTER than LAME, I didn't say that... but the numbers are there, and I am gonna stick with the numbers. You can't tell against the numbers. QUOTE (Neasden @ Nov 27 2008, 00:53) Is this basically the end of mp3 by reaching its maximum level of quality being that all codecs are tied? I'll say no more on the subject now.
-------------------- I'm on a horse.
|
|
|
|
|
Nov 27 2008, 19:41
Post
#168
|
|
 Group: Members Posts: 516 Joined: 4-June 02 Member No.: 2220 |
Interesting results. It appears 128kbps really is enough for general use MP3 encoding after all. I actually stick to lossless, but I was satisfied with CBR 128kbps Helix on the old P3-733 system because it's advantageous speed and acceptable quality.
If anything, these results should help rid those ridiculous threads about customized encoding setting (you know, the ones with "-V 0 -m s -b320 -B320" et cetera). -------------------- "Something bothering you, Mister Spock?"
|
|
|
|
|
Nov 27 2008, 19:50
Post
#169
|
|
|
Group: Super Moderator Posts: 9366 Joined: 1-April 04 Member No.: 13167 |
QUOTE (TechVsLife @ Nov 27 2008, 09:47) Does this latest test suggest that the recommended settings for LAME be changed (or any other part of the ha wiki)? No.QUOTE (TechVsLife @ Nov 27 2008, 09:47) is -v2 still the recommended minimum setting for transparency for most listeners in good listening environments etc., or should that be lower now? -V2 is not recommended as the minimum setting for transparency. If you're talking about the wiki which is intended only to be a general guideline the minimum setting is -V3. This is not to say that -V4 might also deliver transparency to a large number of people over a large number of tracks, let alone -V5. Consider that -V5.7 was used in this test for Lame 3.98
-------------------- Everything sounds the same until it is proven otherwise.
|
|
|
|
|
Nov 27 2008, 20:10
Post
#170
|
|
|
Group: Members Posts: 195 Joined: 29-May 07 Member No.: 43837 |
Thanks. I must have been mistakenly thinking of -v2 as the minimum standard for transparency from some even older recommendation. I'll consider -v3 as a conservative and safe min. standard level for transparency for most (the vast majority?) of listeners under even very good conditions (i.e. quiet, but not abx testing). I understand this is only a very general guideline, and listeners vary, etc.
The reason I mention this test as possibly changing the wiki, is that if the perceived quality of the mp3 encoders at 128kps is now higher than it used to be over a couple of years ago (e.g. getting closer to "5"), then that might shift the recommendations a bit. The graph on the wiki of LAME -v settings against both resulting quality and filesize is very useful; I'm assuming that's still consistent with the latest results. Scroll down a little from here for the graph: http://wiki.hydrogenaudio.org/index.php?ti...ate.29_settings QUOTE (greynol @ Nov 27 2008, 13:50) -V2 is not recommended as the minimum setting for transparency. If you're talking about the wiki which is intended only to be a general guideline the minimum setting is -V3. This is not to say that -V4 might also deliver transparency to a large number of people over a large number of tracks, let alone -V5. Consider that -V5.7 was used in this test for Lame 3.98
This post has been edited by TechVsLife: Nov 27 2008, 20:17 |
|
|
|
|
Nov 27 2008, 23:00
Post
#171
|
|
|
Group: Banned Posts: 185 Joined: 1-July 08 Member No.: 55148 |
QUOTE Yes, I do see the suggestion that this is the "end of mp3" as inflammatory. Sorry if I offended anyone with those one-liners. Exiting the thread. |
|
|
|
|
Nov 28 2008, 02:48
Post
#172
|
|
|
Group: Members Posts: 82 Joined: 25-May 06 Member No.: 31129 |
Has anyone else noticed that the itunes encoder has actually performed very well from looking at the results. I mean all this time members f this forum have strongly recommended against using the mp3 encoder in itunes, only to find that it performed just as well as LAME 3.97, which was considered to be a very good encoder for a long time. On top of that Apple have now updated itunes with the fix version of there mp3 encoder. From these results I think it should be ok to say that people who use itunes and would prefer the ease of use in using the built in mp3 encoder, go for it.
|
|
|
|
|
Nov 28 2008, 09:26
Post
#173
|
|
|
Group: Members Posts: 2297 Joined: 9-October 05 From: Dormagen, Germany Member No.: 25015 |
QUOTE (jimmy69 @ Nov 28 2008, 03:48) ... I mean all this time members f this forum have strongly recommended against using the mp3 encoder in itunes ... Yes, I've never liked tendencies like these on HA (I've never seen strong reason for this), and I like the outcome of this test so everybody can pick his favorite also for non-quality related reasons. -------------------- lame3100k -V0 --cvbr 9
|
|
|
|
|
Nov 28 2008, 12:42
Post
#174
|
|
|
Group: Developer (Donating) Posts: 2332 Joined: 28-June 02 From: Argentina Member No.: 2425 |
QUOTE (jimmy69 @ Nov 28 2008, 01:48) I mean all this time members f this forum have strongly recommended against using the mp3 encoder in itunes, Probably based on the last mp3 listening test where it SUCKED. It is really good that now it does not. -------------------- MAREO: http://www.webearce.com.ar
|
|
|
|
|
Nov 28 2008, 17:14
Post
#175
|
|
 Group: Developer Posts: 3036 Joined: 2-December 07 Member No.: 49183 |
QUOTE (kwanbis @ Nov 28 2008, 14:42) QUOTE (jimmy69 @ Nov 28 2008, 01:48) I mean all this time members f this forum have strongly recommended against using the mp3 encoder in itunes, Probably based on the last mp3 listening test where it SUCKED. It is really good that now it does not. It is really good that Sebastian have dualcore computer, and Alex B doesn't 
|
|
|
|
|
|
Lo-Fi Version | Time is now: 19th June 2013 - 19:16 |
