While we wait the test to begin it might be useful to revisit the comments that were posted in the 64kbps multiformat test's announce thread: http://www.hydrogenaudio.org/forums/index....showtopic=56397
In that thread I made some suggestions about how the test presentation and instructions could be developed further: http://www.hydrogenaudio.org/forums/index....st&p=509971
EDIT
In addition, the comments in the post-test thread make a good read:
http://www.hydrogenaudio.org/forums/index....showtopic=56851
Full Version: MP3 Listening Test at 128 kbps
I am afraid that a major "overhaul" of the listening test page and the instruction is not going to be ready in time for this test. I am planning to change the listening test pages and make them somewhat more attractive, but 1. I am not really talented in designing so I have to see who can help me and 2. this is going to take some time so it will be only available for the next teast at earliest. I hope to have less ranked references in this test. Most of the people who perform listening tests are HA readers anyways so I was actually assuming that they know perform a "correct" test.
I didn't want to put more pressure on you. 
I merely wanted to keep this discussion alive and posted the links for newcomers so that they can get a better impression of the complete testing process.
I merely wanted to keep this discussion alive and posted the links for newcomers so that they can get a better impression of the complete testing process.
Small status update: I am trying to get in touch with schnofler for some changes in ABC/HR. Everything is pretty much done except for the ABC/HR configuration files and is ready to be uploaded.
Here is the updated bitrate table that also contains the encoding speeds (except for iTunes that has to be tested once the fixed version is out).
I didn't want to redo all tests on the single-core machine and comparing iTunes' single-core results to the dual-core results of the other encoders is not fair (at least if any of the other encoders makes use of multi-threading).
Edit 1: IMPORTANT: Please notice that the low anchor wasn't taken into consideration for building the min, max, delta and average bitrates.
Edit 2: Sorry for breaking the layout - I used a codebox so I don't exactly know what is going wrong here.
I didn't want to redo all tests on the single-core machine and comparing iTunes' single-core results to the dual-core results of the other encoders is not fair (at least if any of the other encoders makes use of multi-threading).
CODE
Sample 1 Sample 2 Sample 3 Sample 4 Sample 5 Sample 6 Sample 7 Sample 8 Sample 9 Sample 10 Sample 11 Sample 12 Sample 13 Sample 14 Max Min Avg Speed 1)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LAME 3.97 | 97 126 138 149 146 150 95 147 109 158 149 194 133 159 194 95 139 18x
LAME 3.98.2 | 107 149 143 140 138 143 109 136 118 152 145 214 146 156 214 107 143 27x
FhG | 119 121 139 140 144 149 134 128 129 147 134 212 151 163 212 119 144 45x
iTunes | 118 117 139 145 133 125 141 120 126 148 151 192 159 158 192 117 141 N/A 2)
Helix | 114 110 126 151 149 151 131 138 97 152 117 228 143 173 228 97 141 90x
Low Anchor | 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 1,63x
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Max | 119 149 143 151 149 151 141 147 129 158 151 228 159 173
Min | 97 110 126 140 133 125 95 120 97 147 117 192 133 156
Avg | 111 125 137 145 142 144 122 134 116 151 139 208 146 162 142
Max - Min | 22 39 17 11 16 26 46 27 32 11 34 36 26 17
1) These results are only an approximate value and were obtained on a dual-core Intel E6550 with 2 GB RAM running on Windows Vista SP1 32-bit
2) Encoding speed will be measured with iTunes once the bug-fixed version was released. The buggy version encoded at 18x but produced lower bitrates.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LAME 3.97 | 97 126 138 149 146 150 95 147 109 158 149 194 133 159 194 95 139 18x
LAME 3.98.2 | 107 149 143 140 138 143 109 136 118 152 145 214 146 156 214 107 143 27x
FhG | 119 121 139 140 144 149 134 128 129 147 134 212 151 163 212 119 144 45x
iTunes | 118 117 139 145 133 125 141 120 126 148 151 192 159 158 192 117 141 N/A 2)
Helix | 114 110 126 151 149 151 131 138 97 152 117 228 143 173 228 97 141 90x
Low Anchor | 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 1,63x
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Max | 119 149 143 151 149 151 141 147 129 158 151 228 159 173
Min | 97 110 126 140 133 125 95 120 97 147 117 192 133 156
Avg | 111 125 137 145 142 144 122 134 116 151 139 208 146 162 142
Max - Min | 22 39 17 11 16 26 46 27 32 11 34 36 26 17
1) These results are only an approximate value and were obtained on a dual-core Intel E6550 with 2 GB RAM running on Windows Vista SP1 32-bit
2) Encoding speed will be measured with iTunes once the bug-fixed version was released. The buggy version encoded at 18x but produced lower bitrates.
Edit 1: IMPORTANT: Please notice that the low anchor wasn't taken into consideration for building the min, max, delta and average bitrates.
Edit 2: Sorry for breaking the layout - I used a codebox so I don't exactly know what is going wrong here.
Sorry to flutter the dovecotes perhaps, but wouldn't it be fairer to rename the test into 140k then, in stead of 128k?
QUOTE(Polar @ Oct 16 2008, 12:50) 
Sorry to flutter the dovecotes perhaps, but wouldn't it be fairer to rename the test into 140k then, in stead of 128k?
Short test samples do not represent the average bitrate behavior of complete tracks of various genres in a media library. In my tests the overall average was 127.8 kbps. I think it is close enough.
Since this thread is huge and things can easily get lost in it I'll replicate my test results here (once again):
QUOTE(Alex B @ Aug 20 2008, 17:44)
I finally had time to continue my bitrate tests with classsical music as I promised earlier in this thread (a thing called summer got in the way...)
After browsing through my lossless classical library I picked 25 "reference" tracks that should be quite representative. I avoided the extremely low and high lossless bitrates and tried to select tracks that have quite varied qualities.
Apparently iTunes has changed radically since my last test. Back then the 128 kbps VBR setting was suitable, but the 7.7 version uses bitrates in a more relaxed way and the 128 kbps VBR setting produces higher bitrates than before. Fortunately the 112 kbps VBR setting appears to be suitable for our test.
EDIT: As explained earlier in this thread, the fundamental difference in the bitrate behavior was found out to be caused by a bug in iTunes. However, the "iTunes 7.7" bitrates in my test are now assumed to be correct. Since my test Apple has released iTunes 8.0, but apparently its MP3 encoder has not changed. iTunes 7.7 and 8.0 create identical MP3 files.
In addition, I retested the "various" bitrates with the latest encoder versions when applicable.
Since I didn't have the old iTunes version installed I couldn't test the "classical" bitrates with it (which would have been unnecessary anyway).
FhG, iTunes and LAME 3.97 have only one suitable VBR setting for this test so I'd suggest to calculate the average bitrate of these encoders and adjust the Helix and LAME 3.98 settings to match this average. Helix -V60 and LAME -V5.7 appear to be pretty close to this average with my test tracks.
Here are the new results:
Summary
Various - table and chart
Classical - table and chart
EDIT
I forgot to mention that if anyone wants to test FhG's bitrate behavior the bitrates must be measured correctly. Most programs don't show accurate bitrate values because FhG doesn't write Xing headers to VBR files.
I used EncSpot Pro's "full scan" option.
After browsing through my lossless classical library I picked 25 "reference" tracks that should be quite representative. I avoided the extremely low and high lossless bitrates and tried to select tracks that have quite varied qualities.
Apparently iTunes has changed radically since my last test. Back then the 128 kbps VBR setting was suitable, but the 7.7 version uses bitrates in a more relaxed way and the 128 kbps VBR setting produces higher bitrates than before. Fortunately the 112 kbps VBR setting appears to be suitable for our test.
EDIT: As explained earlier in this thread, the fundamental difference in the bitrate behavior was found out to be caused by a bug in iTunes. However, the "iTunes 7.7" bitrates in my test are now assumed to be correct. Since my test Apple has released iTunes 8.0, but apparently its MP3 encoder has not changed. iTunes 7.7 and 8.0 create identical MP3 files.
In addition, I retested the "various" bitrates with the latest encoder versions when applicable.
Since I didn't have the old iTunes version installed I couldn't test the "classical" bitrates with it (which would have been unnecessary anyway).
FhG, iTunes and LAME 3.97 have only one suitable VBR setting for this test so I'd suggest to calculate the average bitrate of these encoders and adjust the Helix and LAME 3.98 settings to match this average. Helix -V60 and LAME -V5.7 appear to be pretty close to this average with my test tracks.
Here are the new results:
Summary
Various - table and chart
Classical - table and chart
EDIT
I forgot to mention that if anyone wants to test FhG's bitrate behavior the bitrates must be measured correctly. Most programs don't show accurate bitrate values because FhG doesn't write Xing headers to VBR files.
I used EncSpot Pro's "full scan" option.
Alex, I think this is the 4th time you quoted yourself. 
Is it? And they still don't learn...
Maybe I should put the test results presentation in my signature.
(Don't take this personally, Polar. It's just that the "test samples vs. complete library" bitrate difference issue has been discussed over and over again in the listening test threads.)
Maybe I should put the test results presentation in my signature.
(Don't take this personally, Polar. It's just that the "test samples vs. complete library" bitrate difference issue has been discussed over and over again in the listening test threads.)
http://www.listening-tests.info/mp3-128-1/
Please report any major bugs. I will make the public announcement tomorrow (actually today since it's after midnight already).
BTW, many thanks to rjamorim and Polar for hosting the samples!
Please report any major bugs. I will make the public announcement tomorrow (actually today since it's after midnight already
).BTW, many thanks to rjamorim and Polar for hosting the samples!
Just a grammar/punctuation nitpick:
The purpose of this test is to find out which popular MP3 VBR encoder outputs the best quality on bitrates around 128 kbps.
(removed comma and added MP3)
The purpose of this test is to find out which popular MP3 VBR encoder outputs the best quality on bitrates around 128 kbps.
(removed comma and added MP3
)
From the readme:
Apparently "wcmd" was renamed to "cmd" in wine 0.9.21 which was released 2 years ago.
Is there any description of the samples available? E.g. if I want to test a specific genre I'd have to download all samples. Also, if one likes a track, it might be interesting what's the artist/title.
QUOTE
Linux users are asked to use Wine with "wine wcmd /c DecodeXX.bat" from the "bin" directory.
Apparently "wcmd" was renamed to "cmd" in wine 0.9.21 which was released 2 years ago.
Is there any description of the samples available? E.g. if I want to test a specific genre I'd have to download all samples. Also, if one likes a track, it might be interesting what's the artist/title.
Thanks for the wmcd feedback. I will edit the readmes once I get home this evening.
As for the sample names - they will be published only after the test. With past tests I had the problem, that a lot of people focused on several samples for which I then had lots of results while other samples remained untested.
If a lot of people complain about this, I might publish them, though.
As for the sample names - they will be published only after the test. With past tests I had the problem, that a lot of people focused on several samples for which I then had lots of results while other samples remained untested.
If a lot of people complain about this, I might publish them, though.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.