http://www.pcmag.com/article2/0,4149,806392,00.asp
[img]http://common.ziffdavisinternet.com/util_get_image/1/0,3363,sz=1&i=19186,00.gif[/img]
Seems (??) that they used "strict bitrate management" (practically cbr) for Vorbis, which is of course not ideal for it. Especially the high bitrate results are not flattering for Ogg Vorbis.
Full Version: PC Magazine audio results in "Play Anything" test.
Well, in my opinion those results are pure bullshit...
Agreed. Vorbis is most superior at low bitrate encoding. AAC is great for midrange. MPC is excellent for higher bitrate. mp3 with the latest lame and the new "medium" profiles generated on the other thread have yielded pretty decent results too for 100-150kbit range. That's my two cents.
Vorbis is not superior to MP3pro at low bitrate. MP3pro was also on top (with Vorbis) in ff123's 64kbps test.
Although in ff123's test MP3pro used only CBR, Vorbis used vbr and cbr.
http://ff123.net/64test/results.html
Imo MP3pro is currently the best quality codec at 64kbps. Vorbis comes close, but it collapses the stereo separation quite a lot. Soon there will be another contester for low bitrate: MPEG4 AAC+ (SBR)
Although in ff123's test MP3pro used only CBR, Vorbis used vbr and cbr.
http://ff123.net/64test/results.html
Imo MP3pro is currently the best quality codec at 64kbps. Vorbis comes close, but it collapses the stereo separation quite a lot. Soon there will be another contester for low bitrate: MPEG4 AAC+ (SBR)
>_< Why is it that, for publications aimed at computer novices, the columnists also appear to be novices?
I think their Editor's Choice awards for media players says a lot: Real Player One Plus 2.0 and MusicMatch Jukebox 7.5. I mean, seriously, I thought Real Player was just a banner rotation program, now they're telling me it plays music?
So they did use CBR for Vorbis. All in all, this appears to be a poorly designed test, almost as bad as some of the "blacklisted" compressed audio tests floating around the 'net. But then again, this test isn't about obtaining best sound quality. It's a test done by people with totally untrained ears, using default encoder settings, because that's the way their target audience will encode music. So for complete n00bs, such as Mom and Pop or that executive who can barely operate Microsoft Word, perhaps PC Magazine has found the best solution.
I think their Editor's Choice awards for media players says a lot: Real Player One Plus 2.0 and MusicMatch Jukebox 7.5. I mean, seriously, I thought Real Player was just a banner rotation program, now they're telling me it plays music?
QUOTE
We tested each format using three music genres—classical, pop, and rock. We digitally encoded each selection to MP3, MP3Pro, Ogg Vorbis, Real, and WMA formats at a low-bit-rate setting (64 Kbps) and high-bit-rate settings (typically 128 Kbps, although we used 96 Kbps for MP3Pro—the format's maximum bit rate—and 132 Kbps for Real files, which was the closest available setting). The tracks were encoded with constant bit-rate settings.
So they did use CBR for Vorbis. All in all, this appears to be a poorly designed test, almost as bad as some of the "blacklisted" compressed audio tests floating around the 'net. But then again, this test isn't about obtaining best sound quality. It's a test done by people with totally untrained ears, using default encoder settings, because that's the way their target audience will encode music. So for complete n00bs, such as Mom and Pop or that executive who can barely operate Microsoft Word, perhaps PC Magazine has found the best solution.
First of all, MP3Pro is horrible, terrible, and horrifying at high bitrates. Meaning 128 and above, it's crap. I have NO IDEA how it got such a good grade. For vorbis they must have used the most shitty configuration ever, this is crazy.
And -Real- and WMA so good?? They must be kidding me...
And -Real- and WMA so good?? They must be kidding me...
QUOTE (JohnV @ Jan 14 2003 - 05:06 PM)
Vorbis is not superior to MP3pro at low bitrate. MP3pro was also on top (with Vorbis) in ff123's 64kbps test.
Although in ff123's test MP3pro used only CBR, Vorbis used vbr and cbr.
http://ff123.net/64test/results.html
Imo MP3pro is currently the best quality codec at 64kbps. Vorbis comes close, but it collapses the stereo separation quite a lot. Soon there will be another contester for low bitrate: MPEG4 AAC+ (SBR)
Although in ff123's test MP3pro used only CBR, Vorbis used vbr and cbr.
http://ff123.net/64test/results.html
Imo MP3pro is currently the best quality codec at 64kbps. Vorbis comes close, but it collapses the stereo separation quite a lot. Soon there will be another contester for low bitrate: MPEG4 AAC+ (SBR)
Agreed!
ff123's test results immediately popped into my head too.
I still believe a properly tuned aac+sbr will ultimately beat everyone at that bit range.
Easier said than done.
The low bitrate results look rather reasonable although MP3Pro's lead is a bit overdone.
I had tested out MP3Pro at 32kbps using MMJ and was very much impressed with the performance. We're talking 44x compression and the output was listenable, occasionally surprisingly so. They need to get MP3Pro decoders on flash memory based portables so that an inexpensive 32MB model could be usable. When limited to MP3 or WMA playback only, these players are shit.
I had tested out MP3Pro at 32kbps using MMJ and was very much impressed with the performance. We're talking 44x compression and the output was listenable, occasionally surprisingly so. They need to get MP3Pro decoders on flash memory based portables so that an inexpensive 32MB model could be usable. When limited to MP3 or WMA playback only, these players are shit.
fisrt
i think whenever someone does a listening test he must say
if it was made by headphones or speakers
the brand of headphoones and the speakers and the amplifiers
since then we can compare two litsening test
second
listening to highly compressed mp3 or vorbis files can be enhanced very seriously by using a sound enhancing on winamp like dsp fx or other dsp that are freeware and very efficient because at the end the goal is to listen to music
i think whenever someone does a listening test he must say
if it was made by headphones or speakers
the brand of headphoones and the speakers and the amplifiers
since then we can compare two litsening test
second
listening to highly compressed mp3 or vorbis files can be enhanced very seriously by using a sound enhancing on winamp like dsp fx or other dsp that are freeware and very efficient because at the end the goal is to listen to music
QUOTE (SK1 @ Jan 14 2003 - 04:52 PM)
Well, in my opinion those results are pure bullshit...
agreeIt's ok for the LOW BIT RATE
HIGH BIT RATE
???????
Real the best?
WMA 2nd?
Something must be wrong!
MP3PRO is better than OGG?
MP3 & WMA handles "rock" better than OGG?
impossibleMP3 & WMA handles 'rock" very badly to me even at 192 kbit/s
Overall HIGH BIT RATE should be like.......
MPC
1. OGG
2. MP3 (lame)
3. WMA
4. Real
5. MP3PRO
QUOTE (JohnV @ Jan 15 2003 - 02:06 AM)
Vorbis is not superior to MP3pro at low bitrate. MP3pro was also on top (with Vorbis) in ff123's 64kbps test.
Although in ff123's test MP3pro used only CBR, Vorbis used vbr and cbr.
http://ff123.net/64test/results.html
Although in ff123's test MP3pro used only CBR, Vorbis used vbr and cbr.
http://ff123.net/64test/results.html
By the way, I also tested the c't reference.wav with Vorbis 1.0 -q0 back then to make sure that I would not overlook any differences between a variable and a "managed" setting that c't seemed to have used in their 64 kbps test. There was only a very small plus of quality using -q0, the awful harshness and white noise (making Kylie Minogue almost sound like Tina Turner etc.) was still there, but maybe not as mean as with a constant bitrate. So I wouldn't have ranked Vorbis on another position than second worst (out of 6) again...
QUOTE
Imo MP3pro is currently the best quality codec at 64kbps. Vorbis comes close, but it collapses the stereo separation quite a lot. Soon there will be another contester for low bitrate: MPEG4 AAC+ (SBR)
Yeah, you tell 'em, John...
Do you know by any chance which WMA version was used in this test? If it was WMA8 instead of WMA9 again, I could better understand these results... And what do these percentages mean, the rate of people who preferred/disliked the format (like in the c't test)? Who participated in the test, also normal listeners or only PC journalists?
Independantly of the test results, I think some of the information in that PCMag online article is misleading.
Does someone know where I could send them feedback/constructive remarks ?
Thx.
Does someone know where I could send them feedback/constructive remarks ?
Thx.
QUOTE (NumLOCK @ Jan 15 2003 - 01:10 PM)
Does someone know where I could send them feedback/constructive remarks ?
Try here: http://www.pcmag.com/article2/0,4149,17,00.asp
"let the flames begin!"
honestly though, this test is the most complete bastardization of any claimed scientific measurement ive seen, especially for such a distributed magazine.
and what is the deal with real?? in my opinion Real is the worst codec on the planet. ESPECIALLY at high bitrates. I really dont have a clue how the framesize works... but it sounds like they increase the framesize with the bitrate, without bothering to shorten the frame length. And the frame length *sounds* like its already really damn long, even at 32kbps. So you end up with the worst pre-echo ive ever heard out of any codec
that test is bs
honestly though, this test is the most complete bastardization of any claimed scientific measurement ive seen, especially for such a distributed magazine.
and what is the deal with real?? in my opinion Real is the worst codec on the planet. ESPECIALLY at high bitrates. I really dont have a clue how the framesize works... but it sounds like they increase the framesize with the bitrate, without bothering to shorten the frame length. And the frame length *sounds* like its already really damn long, even at 32kbps. So you end up with the worst pre-echo ive ever heard out of any codec
that test is bs
QUOTE (kotrtim @ Jan 15 2003 - 08:26 AM)
Real the best?
WMA 2nd?
Something must be wrong!
WMA 2nd?
Something must be wrong!
Naw, it just means Emmett didn't pay them as much as Real and MS did
A test to satisfy LAMErs
QUOTE (jesseg @ Jan 15 2003 - 12:42 PM)
[...] I really dont have a clue how the framesize works... but it sounds like they increase the framesize with the bitrate, without bothering to shorten the frame length. [...]
Ummmm....
EDIT: oh.. maybe you meant "frame size" = "number of bits"
At first I didn't understand it like this
This is a little off topic, but what do you all think of the next results, the video format test?
In general, I have been very impressed with Quick Time's video at high bit rates. I found it surprising that an MPEG4 codec was rated so bad.
In general, I have been very impressed with Quick Time's video at high bit rates. I found it surprising that an MPEG4 codec was rated so bad.
QUOTE (salpro @ Jan 15 2003 - 08:32 AM)
fisrt
i think whenever someone does a listening test he must say
if it was made by headphones or speakers
the brand of headphoones and the speakers and the amplifiers
since then we can compare two litsening test
i think whenever someone does a listening test he must say
if it was made by headphones or speakers
the brand of headphoones and the speakers and the amplifiers
since then we can compare two litsening test
Actually, they did mention what equipment they used.
What they don't mention is how the test was performed. The only thing I can find is this:
QUOTE
To decide which format sounds best at a given bit rate, our jury listened to each of the selections using high-quality headphones in a series of head-to-head comparisons against other formats at the same bit rate. The percentages we've charted are approval ratings based on the number of wins for each format (think of a sports team's winning percentage).
They played two compressed formats along each other (I'm assuming this means non-blind even to what formats they are comparing) and chose the one that sounded best?
Uh. No, this doesn't work. Not if you're trying to get a serious result, that is.
As a side note... I've sent them some comments of my own. Here's a copy of the e-mail:
As you can see, I tried to stay courteous and to make constructive comments where possible
No laughs please.. but do you guys think this is fine ?.. btw, I'll post the response if possible (assuming there will be one)
QUOTE
Dear Sir, Madam,
- Please forward this to the online editorial team working on technical
articles. -
I have a few remarks and some feedback that I'd like to share with you,
regarding your article on "Audio Codecs". This article starts at the
following URL:
http://www.pcmag.com/article2/0,4149,806392,00.asp
First of all, I'll refrain from commenting too much on the result of your
listening tests, because this is up to each listener's preference.
I have noted a few inconsistencies in your article:
- You state that "96kbps is the maximum bitrate for the MP3Pro format". This
is untrue: some free encoders might be limited in this way; but the MP3Pro
format itself doesn't have this limitation at all.
- For 2-channel audio, 64..96 kbps can be called "low bitrate" I agree, but
128kbps is by no means a "high" bitrate. I'd rather call 170kbps "moderate",
and 256kbps "high".
- It would be good if you'd include the encoder versions and settings used for
your tests. There are two reasons for this: first, people can reproduce
listening easily, and second, the quality (especially for mp3) highly depends
on the encoder and settings used.
- About the results: with all due respect, there's simply no way that MP3 can
blow away Ogg Vorbis at 128kbps, on the music you used. Even against the best
mp3 encoder, Vorbis 1.0 should be superior in every respect (at this bitrate,
that is).
- A last suggestion, about the future: next time you choose the formats to
test, you might want to include AAC (included in new Nero versions) and maybe
Musepack (fast encoding, freely available, and superior sound than MiniDisc
from 140kbps upwards, on average !).
- On the page about comparing MP3 with CD, I think you should mention that
even 256kbps MP3 is lossy. Surprisingly, many people think that MP3 is
lossless at high settings.
Thank you for listening to my comments. I hope you can take some benefit from
my (constructive I hope) remarks for the future.
Please, feel free to answer this e-mail. I like to discuss opinions on audio
compression. By the way, if you want your response to stay private (ie: no
inclusion in discussions on audio-related discussion sites) please mention
it.
Again, thanks for your time.
Best Regards,
(blah blah)
- Please forward this to the online editorial team working on technical
articles. -
I have a few remarks and some feedback that I'd like to share with you,
regarding your article on "Audio Codecs". This article starts at the
following URL:
http://www.pcmag.com/article2/0,4149,806392,00.asp
First of all, I'll refrain from commenting too much on the result of your
listening tests, because this is up to each listener's preference.
I have noted a few inconsistencies in your article:
- You state that "96kbps is the maximum bitrate for the MP3Pro format". This
is untrue: some free encoders might be limited in this way; but the MP3Pro
format itself doesn't have this limitation at all.
- For 2-channel audio, 64..96 kbps can be called "low bitrate" I agree, but
128kbps is by no means a "high" bitrate. I'd rather call 170kbps "moderate",
and 256kbps "high".
- It would be good if you'd include the encoder versions and settings used for
your tests. There are two reasons for this: first, people can reproduce
listening easily, and second, the quality (especially for mp3) highly depends
on the encoder and settings used.
- About the results: with all due respect, there's simply no way that MP3 can
blow away Ogg Vorbis at 128kbps, on the music you used. Even against the best
mp3 encoder, Vorbis 1.0 should be superior in every respect (at this bitrate,
that is).
- A last suggestion, about the future: next time you choose the formats to
test, you might want to include AAC (included in new Nero versions) and maybe
Musepack (fast encoding, freely available, and superior sound than MiniDisc
from 140kbps upwards, on average !).
- On the page about comparing MP3 with CD, I think you should mention that
even 256kbps MP3 is lossy. Surprisingly, many people think that MP3 is
lossless at high settings.
Thank you for listening to my comments. I hope you can take some benefit from
my (constructive I hope) remarks for the future.
Please, feel free to answer this e-mail. I like to discuss opinions on audio
compression. By the way, if you want your response to stay private (ie: no
inclusion in discussions on audio-related discussion sites) please mention
it.
Again, thanks for your time.
Best Regards,
(blah blah)
As you can see, I tried to stay courteous and to make constructive comments where possible
No laughs please.. but do you guys think this is fine ?.. btw, I'll post the response if possible (assuming there will be one)
NumLOCK, i think your letter is very fine.
However, for some reason i have a strange feeling it will be ignored because they're lamers and don't care about quality anything, just about spreading information, no matter how flawed...
However, for some reason i have a strange feeling it will be ignored because they're lamers and don't care about quality anything, just about spreading information, no matter how flawed...
QUOTE (SK1 @ Jan 15 2003 - 04:25 PM)
NumLOCK, i think your letter is very fine.
However, for some reason i have a strange feeling it will be ignored because they're lamers and don't care about quality anything, just about spreading information, no matter how flawed...
However, for some reason i have a strange feeling it will be ignored because they're lamers and don't care about quality anything, just about spreading information, no matter how flawed...
Thanks
Well, we'll see what happens..
QUOTE
To whom it may concern,
We appreciate your inquiry and welcome the opportunity to serve your needs.
We have forwarded your comments to the appropriate department.
Thank you for your interest in our publication.
blw
We appreciate your inquiry and welcome the opportunity to serve your needs.
We have forwarded your comments to the appropriate department.
Thank you for your interest in our publication.
blw
Let's try to be a bit patient
QUOTE (mithrandir @ Jan 15 2003 - 01:14 AM)
They need to get MP3Pro decoders on flash memory based portables so that an inexpensive 32MB model could be usable. When limited to MP3 or WMA playback only, these players are shit.
Don't know if it's been said here, but I heard something a few months ago about an RCA player which played MP3pro...
i believe rca players have some serious drm issues. at least the model i looked at did.
Real Audio at high are damn bright, even brighter than MS WMA8
and it's the best encoder at mid bitrate kidding? and I don't see any Tiple-s format as PC mag. state that the decoder will be realesed officially at feb. 2003
and it's the best encoder at mid bitrate
kidding? and I don't see any Tiple-s format as PC mag. state that the decoder will be realesed officially at feb. 2003
QUOTE
and I don't see any Tiple-s format as PC mag. state that the decoder will be realesed officially at feb. 2003
i suppose you are referring to the triple s codec; then you should read this:
http://www.hydrogenaudio.org/forums/index....t=ST&f=1&t=5723
fragtal
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.