Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 128kbps Extension Test - FINISHED (Read 73205 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

128kbps Extension Test - FINISHED

Reply #100
Yay, a 64kb test should mean I can take part  When you're ready I'll finally be there to help you Roberto
< w o g o n e . c o m / l o l >

128kbps Extension Test - FINISHED

Reply #101
Quote
Lies, damn lies and statistics come to mind, in that results can be adjusted up or down...make of it as you will

Quoting Congressman Alex Shrub:

"Those statistics are interesting, but like all statistics, they are also irrelevant. "


128kbps Extension Test - FINISHED

Reply #102
"Facts, shmacts... you can use facts to prove anything, that even remotely true!" - Homer J. Simpson.
The Plan Within Plans

128kbps Extension Test - FINISHED

Reply #103
Quote
If I ever compare several MP3 encoders at 128kbps (yes, I'm planning that)

Would be quite interesting...

128kbps Extension Test - FINISHED

Reply #104
Aren't there any tests yet?

128kbps Extension Test - FINISHED

Reply #105
Quote
Quote
Iam Happy that my Favorite Codec  WMA show the rest that he is better and not so Crappy like Everybody said.

But i have One little Problem a wma9pro  Encoded File dont be played in Winamp 2.95.

Is there a plugin that wma9pro  played without the Windows Media Player.

ThX

Unfortunately, that's the "one little problem" with using wma9pro right now.  Although it sounds better than the older wma format, it breaks compatibility with current decoders.  I'm sure that will change in due time.

ff123

Let's say I want to play wma9 Pro and older wma encoded files. Do I need the old AND the new codec in order to play it, or does wma9 Pro codec already include the 'old' codec as well? I mean in that case I wouldn't see too many problems...
--alt-presets are there for a reason! These other switches DO NOT work better than it, trust me on this.
LAME + Joint Stereo doesn't destroy 'Stereo'

128kbps Extension Test - FINISHED

Reply #106
Quote
Do I need the old AND the new codec in order to play it, or does wma9 Pro codec already include the 'old' codec as well?

I think you would need to ask that to Microsoft. :-/

128kbps Extension Test - FINISHED

Reply #107
Wma in all its confusing glory:

Basically everything wma that is not pro, lossless or voice (even WMA v9 standard) can be played by all WMA codecs including the first v2 codec.

To play pro, lossless and voice install the wma v9 codecs (which will play everything).

128kbps Extension Test - FINISHED

Reply #108
Quote
Ok doing a simple percentage alignment for any codec over 128Kbps (don't know if that is fair or not, a codec given 64Kbps would be 25% of one given 256Kbps?...):

Spoon: you are my hero.

But I thought that you people said that the grand total average bitrate of all tracks (i.e. add all average bitrates and divide by # of tracks, 12 I believe, for a specific codec) was tuned to all be -exactly- 128?? Or? If so then spoon is wrong.. (or Im just missing something obvious, wouldn't be the first time)

But if not then this is even more staggering.. (and.. exactly what I mentioned over and over again  ).. so byte for byte AAC beats MPC if  you extrapolate results from 'bitrate really used' to 'what it would be rated at when bitrate was really 128'?

Cool.  and here I was thinking that maybe the overall differences would be too small to influence score, but at least for specific tracks this correction factor could give hints about what codec is best suited for what type of sample. Incidentally I realise that you can't just do this correction without giving the realities of lossy coding a second thought, but this is quite some food for thought.

(Plus next time PuntCodec™ is going to be submitted in there.. Lossy to the max people..)

Sooo.. reorganising Spoon's list based on adjusted score we get the following ranking:

Quote
Adj Result = Result * (128 / Avg Bitrate)
Adjusted Results are   Adj Result   (Previous results):

1: AAC        4.38      (4.42)
2: WMA Pro  4.30      (4.30)
3: MPC        3.95      (4.51)
4: Vorbis     3.91      (4.28)
5: Lame      3.77       (3.66)
6: Blade      2.22       (2.22)


Lies, damn lies and statistics indeed...  B)

128kbps Extension Test - FINISHED

Reply #109
Incidentally, rjamorim, don't take my criticisms the wrong way, I think the test was pretty solid and the results interesting! Good job.

While we're critisizing though, is there any way to eliminate the 'listening to crap hardware' factor in this kind of test? For example vorbis introduces more noise than 'average' (whatever that is), but if you listen on a stereo that hisses already you might find vorbis the best codec by far..

If you listen to bad headphones you might lose all lows and rate codecs that reproduce bass well way too low..

And well I still would be interested in 160kbps listening tests.. As most people agree, abx'ing -q6 settings will probably be drowned in 'noise' from people (like me probably) who can't hear the difference with the original, but q5 might still be doable. Plus that really comes into the ranges the codecs were 'designed' for, right?

I mean I saw claims that musepack is the #1 codec for 160kbps or more, I know that vorbisenc GT3 mostly has tweaks for -q5 and better and so on..

128kbps Extension Test - FINISHED

Reply #110
Quote
AAC beats MPC if you extrapolate results from 'bitrate really used' to 'what it would be rated at when bitrate was really 128'

This whole disagreement is incredibly bogus... I thought it had gone away when rjamorim explained how he picked the q ratings for vbr codecs (the average bitrate for your entire collection will be 128kbps even though for these deliberately singled out difficult pieces it is higher).

I think rjamorim should force spoon and puntloos to do a listening test of his entire music collection, just so that the bitrates come out right.

If puntloos still thinks these bogus statistics mean anything at all then I can upload the test encoded with floggy: even if I only scored 0.2 I'd come out on the top of spoon's table, and could claim to have the best 128kbps codec.

128kbps Extension Test - FINISHED

Reply #111
Quote
But I thought that you people said that the grand total average bitrate of all tracks (i.e. add all average bitrates and divide by # of tracks, 12 I believe, for a specific codec) was tuned to all be -exactly- 128?? Or? If so then spoon is wrong.. (or Im just missing something obvious, wouldn't be the first time)

Note: This post is long and getting buchered, so I've broken it up into several.  Sorry.

I again suggest you read the pre-test thread to clarify things.  You can do that here.  Also, look at this thread if you want to see, in detail, how the settings were chosen.  Several settings were tried over a large number of whole albums.  The settings that came closest to averaging 128 over all those albums were the ones used for the test.

The bitrates of the chosen samples are not representative of the bitrates you would get on average over a typical music collection.  They are often higher because difficult portions of music are overrepresented.

I don't think anybody ever takes 20-30 second clips of several different pieces of music and encodes them several times at different VBR settings until they get 128kbps.  On the other hand, I'd wager almost everyone chooses settings that meet their filesize/quality tradeoff requirements then encodes all their music with the same settings (though their settings might change over time as their needs change).
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

128kbps Extension Test - FINISHED

Reply #112
I maintain that to repeatedly encode your collection tweaking settings on a file-by-file basis so that each song (or album) comes out to 128kbps makes as much sense as encoding longer songs at lower quality levels so they come out to the same file size as shorter songs.

Furthermore, I think adjusting the test scores by multiplying them by the ratio of actual bitrate to target bitrate makes no sense.  It can't really represent how well they would actually perform in a CBR/managed bitrate comparision.  There is no reason to believe, for example, that if MPC encoded a short sample at 256kbps and got a "5.0" would get a "2.5" if forced to use 128kbps.  In fact, there's no basis on which to make ANY assumptions about what score a codec would get at a particular bitrate (other than a higher bitrate probably corresponds to a higher score, and a lower bitrate corresponds to a lower score).

If you would like to run a CBR comparison, you may get the results you're looking for.  However, I don't know how much participation you'll get since the only legitimate use of CBR is streaming.  A 64kbps CBR test would be interesting, because that is a bitrate well suited to streaming.
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

128kbps Extension Test - FINISHED

Reply #113
Put it this way, no one expected MPC to win this test and when it did you could say an average combined bitrate of 140Kbps over 12 tracks helped, when something like AAC was 128Kbps spot on.

I think it is right to challenge convention, remember a test for 128Kbps is aimed at an end user who is limited by space, if AAC was allowed an average of 140Kbps then the initial results would have been different.

I have nothing against rjamorim, mpc or abx testing, infact I recommend people to use MPC above 160Kbps.

128kbps Extension Test - FINISHED

Reply #114
Quote
I think it is right to challenge convention, remember a test for 128Kbps is aimed at an end user who is limited by space, if AAC was allowed an average of 140Kbps then the initial results would have been different.

Yes, but if the end user chooses mpc q4 as his default setting, on a large scale the bitrate will average 128kbps. Thus he is using up practically the same amount of space as when using aac cbr @ 128kbps. That's why choosing this setting for the test makes perfect sense.

128kbps Extension Test - FINISHED

Reply #115
Noone would recode over and over again until 128kbit is achieved in a realistic scenario. But this is a -test-, isn't it? Phong gave the floggy example (which, c'mon.. would never be given even more than a score 1, even compared to the 5 of Blade)..

Saying 'floggy scores 1.1 at 4kbps and MPC scores 4.5 at 128kbps' so floggy wins if you multiply up to compensate for average bitrate isnt fair, the simple calculation curve I suggested before and Spoon used does not work for encoder settings that are in completely different leagues. Anyway I doubt we (humans) have the capacity to fairly rate and compare a floggy and a mpc-q4 encoding. Would floggy be 1.1? 0.5? 0.05? I really wouldn't know how to fairly rate floggy if it were used as the anchor test instead of blade.

The encoders tested here however are in the same league. They were designed for these kinds of bitrates (perhaps a bit higher) and didnt vary that much in resulting bitrates.

However.

In -this- test, evidentally Q4 on MPC turned out to be 140kbps average. What can we conclude? Nothing solid. But we can at least hypothesise that:

1/ "Q4" for MPC coders meant something slightly different than 'has to average at 128" for one I think Q5 was intended to be 'transparent', and wasnt set while trying to fit an average bitrate (160?). Who says Q4 was intended to be "128"?

2/ Maybe the settings rjamorim and the people who made the test were indeed giving them 128kbit average with their samples, but they didn't for the end result of the test. I think it is safe to say that this means MPC got unlucky and didn't find the test sample package 'averagely easy'. And this will skew the results compared to (say) AAC. Yes, MPC scored higher on sound quality at the expense of bitrate. MPC should not be allowed to get away with this since I refuse to believe we just compare quality, period.

If we did only compare quality, PuntCodec™ which strips only the wav headers and calls itself lossy should win. This is useless information. The results aren't useless but I say that for fairness purposes it is skewed.

128kbps Extension Test - FINISHED

Reply #116
Quote
Who says Q4 was intended to be "128"?

I REALLY think you need to read this again.

Quote
Yes, but if the end user chooses mpc q4 as his default setting, on a large scale the bitrate will average 128kbps. Thus he is using up practically the same amount of space as when using aac cbr @ 128kbps. That's why choosing this setting for the test makes perfect sense.


and this

Quote
Furthermore, I think adjusting the test scores by multiplying them by the ratio of actual bitrate to target bitrate makes no sense. It can't really represent how well they would actually perform in a CBR/managed bitrate comparision. There is no reason to believe, for example, that if MPC encoded a short sample at 256kbps and got a "5.0" would get a "2.5" if forced to use 128kbps. In fact, there's no basis on which to make ANY assumptions about what score a codec would get at a particular bitrate (other than a higher bitrate probably corresponds to a higher score, and a lower bitrate corresponds to a lower score).


and this one too for good measure.

Quote
This whole disagreement is incredibly bogus... I thought it had gone away when rjamorim explained how he picked the q ratings for vbr codecs (the average bitrate for your entire collection will be 128kbps even though for these deliberately singled out difficult pieces it is higher).


edit:  added more quotes
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame

128kbps Extension Test - FINISHED

Reply #117
Quote
In -this- test, evidentally Q4 on MPC turned out to be 140kbps average. What can we conclude? Nothing solid. But we can at least hypothesise that:

1/ "Q4" for MPC coders meant something slightly different than 'has to average at 128" for one I think Q5 was intended to be 'transparent', and wasnt set while trying to fit an average bitrate (160?). Who says Q4 was intended to be "128"?

2/ Maybe the settings rjamorim and the people who made the test were indeed giving them 128kbit average with their samples, but they didn't for the end result of the test. I think it is safe to say that this means MPC got unlucky and didn't find the test sample package 'averagely easy'. And this will skew the results compared to (say) AAC. Yes, MPC scored higher on sound quality at the expense of bitrate. MPC should not be allowed to get away with this since I refuse to believe we just compare quality, period.


Before the test, rjamorim asked people to encode one or more albums at a couple of different q values in this thread. Based on those results, he concluded that for the majority of albums Q4 was most likely to have an ABR of 128. So what if it didn't for the samples in this test? That indicates, as 264556 said, that these pieces are more difficult to encode (and better for use in tests), and that Musepack is a better overall encoder, since it is capable of raising the ABR that high when necessary, while still maintaining an overall ABR of ~128kbps for the large majority of music.
Happiness - The agreeable sensation of contemplating the misery of others.

128kbps Extension Test - FINISHED

Reply #118
Quote
That indicates, as 264556 said, that these pieces are more difficult to encode (and better for use in tests), and that Musepack is a better overall encoder, since it is capable of raising the ABR that high when necessary, while still maintaining an overall ABR of ~128kbps for the large majority of music.

Thanks for putting it EXACTLY the way I wanted to. 
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame

128kbps Extension Test - FINISHED

Reply #119
Wow, MPC tops the 128 kbps range as well?  Impressive stuff.  I might start playing with MPC more then.

Actually, I think some credit is due for Vorbis.  Even though it didnt do that well in the test, at least it proved true in their claim that Vorbis will do as good, if not better than mp3.  Also, if we view this from another perspective, Vorbis is a product of the free open source concept rather than intensive and sophicated research and development by the big companies like Fraunhofer, Dolby, or Microsoft.  Thus for it to keep up with the other state-of-the-art codecs is quite an achievement within itself.  Take for example gcc.  It used to as slow as Borland's compiler but now is achieving pretty impressive performance.  However, compared with sophisticated compilers from Intel, it isnt easy for open source gcc to surpass them.

If I can ever understand the Vorbis source code, 128 kbps is the range I'm gonna try tuning for.

Anyway, thanks to Roberto for this very informative test as well as all those listeners.

128kbps Extension Test - FINISHED

Reply #120
Quote
I don't think anybody ever takes 20-30 second clips of several different pieces of music and encodes them several times at different VBR settings until they get 128kbps.  On the other hand, I'd wager almost everyone chooses settings that meet their filesize/quality tradeoff requirements then encodes all their music with the same settings (though their settings might change over time as their needs change).

You should read hedline of this topic again. I quite sure that there is a text "128kbps Extrension Test". There is'nt mentioned  anywhere anything about testin certain quality settings of encoders. I thought we should do this test to see how different codeks can handel common problem samples at same average bittrate. I think almost everyone here who encodes music him self with lossy codek will not use 128kbps average size without any special reason.

128kbps Extension Test - FINISHED

Reply #121
To test efficiency, we need to have more closely similar bitrates.  Regardless of whether MPC is good at choosing higher bitrates for those seldom "difficult" tracks, there should be enough tracks that are NOT this difficult to encode for us to test with.  Neither known difficult tracks for certain encoders, nor known easy tracks for one encoder should be used that could skew results.  To test efficiency, which i believe is what should be the goal for a 128kps  listening test, the bps should be more similar.
WARNING:  Changing of advanced parameters might degrade sound quality.  Modify them only if you are expirienced in audio compression!

128kbps Extension Test - FINISHED

Reply #122
The arguments over the test method as it concerns bitrate have been hashed and rehashed many times over now, and the opinions have solidified.  Maybe it's time to close this particular topic.

I think that a 64 kbit/s test could be designed to move closer to happy medium, since difficult samples are not be needed to discriminate between codecs.  So it would be possible to try to choose samples such that the entire test suite averages about 64 kbit/s for each codec.

For this test, though, there's not much more to say.  It's done and over with.  I will add that I personally think Roberto's tests are of the highest quality.  There's not a comparable test out there on the net for 128 kbit/s codecs (cT's test is a joke by comparison).  For those who would have done it differently:  the opportunity is still there for you!  If you can dish out the criticism, can you stand to take it too?

ff123

128kbps Extension Test - FINISHED

Reply #123
...

128kbps Extension Test - FINISHED

Reply #124
Do not take my last post as criticism for the last test.  I am speaking only for the future.  I highly respect the quality of this last test, and I only want to make the future tests even better.  This test proved very informative, I only would like to see how much closer it would be if we fine tuned it in the future, that's all.

Quote
For this test, though, there's not much more to say. It's done and over with. I will add that I personally think Roberto's tests are of the highest quality. There's not a comparable test out there on the net for 128 kbit/s codecs (cT's test is a joke by comparison). For those who would have done it differently: the opportunity is still there for you! If you can dish out the criticism, can you stand to take it too?


ditto.
WARNING:  Changing of advanced parameters might degrade sound quality.  Modify them only if you are expirienced in audio compression!