Greetings HA. Long time lurker checking in. Start with some kudos to HA for years of invaluable info and countless tips, tricks, and tweaks. Helluva resource for those of us stuck in that no man's land between the lunatic fringe of audiophilia and the bleeding edge of techno-wizardry.

I don't know how actively LAME is being developed these days (I've been out of the game for a couple years), but it's been my encoder of choice since the turn of the century. I've seen rivals come and go, and after years of doomsday prophesiers predicting the death of mp3, I think it's fair to say that the King will retain his throne for a few years yet (sorry Mr Jobs, cute phone though.) Alas, the time has come for me to leave to courtyard, as my inner ear has finally won the war with my inner geek. Several years back, one of my audiophile "friends" (the snotty, technophobic variety) was offering his usual scoffs at my then "hi-tech" setup (slack linux-based digital media server paired with mid-fi audio components) when I dared him to A/B a 256kbps mp3 (encoded by the trusty old v3.93) against the original source material (standard redbook CD). He proudly agreed, and 15 frustrating (for him) yet oh so enjoyable (for yours truly) minutes later, he proudly stuck his foot in his mouth. Previous comparisons from years prior were what sold me on lossy compression in the first place. Even some lower bitrate (192kb/s) material proved challenging, if not impossible to identify. Snobby audio purism notwithstanding, if *my* ears couldn't tell the difference, then why should I care? Besides, it was such good fun to trash-talk those holier-than-thou "Audiocrats," and lord knows they made it easy every time they uttered that Audiophile gibberish ("This driver has a timbrel airiness that is simply unmatched!")

Over the years, however, my own taste for high end audio grew considerably, as did my pursuit of that "perfect" sound. With each upgrade I would force myself to sample a few tracks and compare the mp3 against the waveform, as I couldn't stomach the thought of spending thousands on components only to feed them crappy source materials. And while countless auditions, purchases, and obnoxious pursuits gradually elevated my set-up to respectable reference levels, my mp3s rarely failed to stand up to empirical scrutiny. With the purchase of my current speaker line (Krix) a couple years ago, I did begin to notice some minute yet discernible differences on a few "problem" tracks, so I opted to brute force the issue by re-doing my entire collection at the best variable bit rate settings available, which proved indistinguishable from source on each of the offending tracks. Convinced that my overhauling days were over, I resigned myself to happily encode future discs at HQ VBR settings and live out my days in audio bliss.

Fast forward two years, and several K in further upgrades. I've known another "quality check" has been long overdue, and with the price of storage space at historical lows, I found myself now in a decent postion to accept bad news if and when it came. So I dusted off a few CDs, ripped a few tracks and sat down for listening sessions.

-Reference Gear-
Signal Processing/Conversion: Xenon Chip, 16/48 Sis DAC (Xbox 360); Crystal Semiconductor CS-49400 DSP, 24/192 Wolfson DAC (Arcam AVR-250)
Amplification: Arcam AVR-250; Bottlhead S.E.X. integrated tube amplifier
Drivers: Krix Lyrix, Equinox, and KDX-M/C; Klipsch Forte II; Grado S60s


First informal test: Sample from existing mp3 collection vs. original wavform

I chose Tina's "What's love got to do with it," an all time favorite. I started with the original mp3 (encoded using and older version of LAME). Pairing the Xbox (transport only) with my AVR-250 (DSP and conversion), I listened for about 30 seconds, and couldn't imagine the waveform sounding much better. 10 seconds into the latter, I was about ready to call it day. Then the snare drum hit. Damn that sounded crisp. Don't remember it sounding that crisp in mp3 format. Switch back. I'll be damned. On the wav file I hear a tight, precise "pshhht" when the snare drum snaps. On the mp3, it sounds almost identical, but it doesn't snap. Instead there's this drawn out, wavy echo that shadows the impact, making it sound something like "pshhhuwwwwww." Not good.

2nd informal test
No worries though, LAME's probably got a new version by now, I'll just download the latest (3.97) and try the highest VBR settings again. Crap, my old scripts in EAC are outdated. What happend to -alt preset? Oh I see, -V 0 it is then! Rinse and repeat. No change. Getting a little worried. Time to break out the big guns. Let's see you try that crap with -b 320, Tina. Yes! Wait, no. Shit. The artifact is much reduced at his bitrate, but still detectable. I've got to let the HA folks know. Time to register. What's this, a 5 day probation?! But I haven't even...Doh! There's got to be way around this....let's check the FAQ's....Hot damn those HA folks have gotten anal about empirics. A/B this! Show me proof noob! [Remembering I'm a scientist and that evidence is the coin of the realm], Sweet. It's about time we get an audio forum that puts proof ahead of posturing. Besides, I've got work to do. No way I'm going to drag my computer into the living room, and my laptop has no digital out. I'll have to identify the problems on my reference gear, and conduct formal testing on my PC. Bear in mind this ups the difficulty quotient considerably, as the DAC's on my bargain basement HP's onboard audio are probably leagues below the hardware in my Xbox and certainly the Arcam, not to mention that pre/post amplification duties now fall on my lowly Klipsch pro-media 4.1 (I know it's a solid multimedia set-up, but that's not saying much). Still, I'll be listening through a set of Grado S60s, the lowest entry in the Prestige Line, but some damn fine cans just the same. Assuming the cheap DAC and the Promedias don't smooth over the imperfections, I should still be able to detect the differences b/w the mp3 and the wav.

Formal Test 1: VBR MP3 (-V 0) vs. Wav

After downloading a clever little program that automates the A/B process (WinABX), I had to extract all samples to wav using Audacity, since WinABX simply refuses to load mp3s natively. I fire up the program, click ABX mode, and have at it. The differences are beyond pronounced. 1/1, 2/2, 3/3, 4/4...11/11. Too easy. Let's the pull the Grados and see if the differences are detectable using the mass market (I got them at Best Buy) Promedia speakers themselves. A little tougher, but still quite easy. 10/10 and I'd had enough. Regardless of which hardware I chose, the difference in the impact of the snare drum b/w the mp3 and wave file were unmistakable.

-------------------------------------
WinABX v0.42 test report
09/08/2007 12:11:41

A file: I:\t\MP3 testing\What's Love -V 0clip.wav
B file: I:\t\MP3 testing\What's Loveclip.wav

Start position 00:00.0, end position 00:05.0
12:12:32 1/1 p=50.0%
12:12:56 2/2 p=25.0%
12:13:20 3/3 p=12.5%
12:13:39 4/4 p=6.2%
12:13:57 5/5 p=3.1%
12:14:19 6/6 p=1.6%
12:14:40 7/7 p=0.8%
12:15:00 8/8 p=0.4%
12:15:10 9/9 p=0.2%
12:15:31 10/10 p< 0.1%
12:16:01 11/11 p< 0.1%
12:16:50 reset

12:17:18 1/1 p=50.0%
12:17:22 2/2 p=25.0%
12:17:25 3/3 p=12.5%
12:17:30 4/4 p=6.2%
12:17:35 5/5 p=3.1%
12:17:39 6/6 p=1.6%
12:17:41 7/7 p=0.8%
12:17:50 8/8 p=0.4%
12:17:56 9/9 p=0.2%
12:18:07 10/10 p< 0.1%
12:18:38 test finished


Formal Test 2: CBR MP3 (-b 320) vs. Wav

Here's where things get interesting. Though the artifacts were barely (but consistently) detectable on my reference system, even at 320 CBR, I simply could not reproduce the distinction using my PC hardware, despite knowing exactly what to listen for. See for yourself:

-------------------------------------
WinABX v0.42 test report
09/08/2007 12:23:25

A file: I:\t\MP3 testing\What's Love - b 320clip.wav
B file: I:\t\MP3 testing\What's Loveclip.wav

Start position 00:00.0, end position 00:05.0
12:26:02 0/1 p=100.0%
12:26:31 0/2 p=100.0%
12:26:38 reset

12:28:01 0/1 p=100.0%
12:28:05 reset

12:31:13 0/1 p=100.0%
12:31:23 0/2 p=100.0%
12:31:30 0/3 p=100.0%
12:31:34 0/4 p=100.0%
12:31:37 0/5 p=100.0%
12:31:40 0/6 p=100.0%
12:31:43 1/7 p=99.2%
12:31:45 2/8 p=96.5%
12:31:46 2/9 p=98.0%
12:31:48 3/10 p=94.5%
12:31:49 4/11 p=88.7%
12:31:55 reset

12:32:31 test finished



Formal Test 3: CBR MP3 (-b 320) vs. Wav, alternate sample

At that point I began wondering if any imperfections at CBR 320 were simply too minute to ascertain using anything but hi end components. So I decided to test another track. This time I chose one that had been particularly difficult years before, even prompting me to switch to higher quality encoding: Smooth Criminal by Michael Jackson. About 30 seconds in, things get pretty busy. Quick strumming bass, synthesizers, guitar, drums, and Michael's almost lispy voice. What?! Why is his voice that lispy? Bloody hell.

She Ran Underneath The Table
He Could See She Was Unable
So She Ran Into The Bedroom
She Was Struck Down, It Was Her Doom

Like the snare issue in Tina's track, there's a slight loss of coherence to the lyrics on the mp3 version. The dead give away, however, is that same wavy echo, this time in between the words. What's particularly troubling is that in some cases the echo seems to *precede* the lyric itself ("So she" in line 3 above, e.g), as if the sampling algorithm gets ahead of itself. To be sure, this distinction was by no means a cinch to identify, particularly on my PC hardware. Nonetheless, when I listened through the Grado's, it became fairly easy to distinguish between the wav and the mp3. Observe:

-------------------------------------
WinABX v0.42 test report
09/08/2007 12:32:34

A file: I:\t\MP3 testing\Michael Jackson - Smooth Criminalmp3clip.wav
B file: I:\t\MP3 testing\Michael Jackson - Smooth Criminalclip.wav

Start position 00:00.0, end position 00:08.0
12:34:42 1/1 p=50.0%
12:34:56 2/2 p=25.0%
12:35:30 3/3 p=12.5%
12:35:41 4/4 p=6.2%
12:36:36 5/5 p=3.1%
12:37:11 6/6 p=1.6%
12:37:20 7/7 p=0.8%
12:39:03 7/8 p=3.5%
12:39:09 reset

12:39:49 1/1 p=50.0%
12:40:08 2/2 p=25.0%
12:40:17 3/3 p=12.5%
12:40:25 4/4 p=6.2%
12:40:33 5/5 p=3.1%
12:40:41 6/6 p=1.6%
12:41:10 6/7 p=6.2%
12:41:27 7/8 p=3.5%
12:41:55 8/9 p=2.0%
12:42:04 9/10 p=1.1%
12:42:13 10/11 p=0.6%
12:42:43 11/12 p=0.3%
12:42:58 12/13 p=0.2%
12:43:15 13/14 p< 0.1%
12:43:27 14/15 p< 0.1%
12:43:32 test finished


While these results are disappointing to say the least, I have to balance mp3s imperfections agaisnt several factors:
-I only tried a few tracks
-320 CBR performed quite well
-Differences are fare less pronounced using mass market audio components

Given these findings, it's clear that I'll have to re-rip/encode my collection using some higher bitrate codec, perhaps even lossless. No intention of canning my mp3 collection, though, since it will serve more than admirably in my DAP and car audio setup. Hopefully the LAME gurus can put these findings to good use. Who knows, there may even be a way to tweak the sampling algorithms to eliminate these flaws. I'm happy to email the tracks that I used for testing (the sample clips are quite small), and I've even done a little non-parametric (read: eyeball) analysis of the waveform pairs (mp3 vs. wav) that shows some minute, though pervasive differences in the waveform.

Cheers,
TP

Hi TP,

If the clips are less than 30 seconds each, can you upload them to the uploads section of the forum please?

If they're longer than 30 seconds, please trim and upload.

Upload the original lossless wav (compressed with FLAC or similar lossless compression if you want) and your mp3.

Maybe also upload the decoded mp3>wav version you loaded into WinABX so people can check nothing has gone wrong there.

Cheers,
David.

It would be nice if you could also test 3.98 beta 5 on your samples. You can get it from www.rarewares.org.

Does WinABX take offset and gain into account?

David,

Just uploaded all six files. I had to use another program (mp3 direct cut) to get the mp3 clips, since Audacity seems to decode mp3s automatically upon loading them, and would thus require re-encoding (and resampling) to get back into mp3 format (a big no no). Hence, the native mp3 clips are longer than their respective wav clips, though the latter are nested within the former. The wav file pairs (decoded mp3 and orig wav) are, of course, identical in length however.

TP

Wombat,

Happy to try out the latest beta. I'll let you know what I find out.

Sebastian,

I cannot answer definitively, but I could perceive no discernible differences in volume during the A/B tests, and I sub-sampled both files (wav and decoded mp3) simultaneously in Audacity. My guess is that WinABX is probably solid, as it seems to have been developed under the auspices of HA regulars (Dibrom and JohnV)

Given that you've only uploaded short wavefiles, it's not possible for me to encode to mp3 myself and verify that your mp3 encodings match mine.

However, there are no gain issues or obvious errors in the encoding or decoding. Time aligning your samples and inverse mix pasting them reveals the kind of differences that I'd expect from 320kbps mp3 vs the original.

I couldn't ABX my own V0 encodings of the clips.

Cheers,
David.

I don't understand. Most mature people that have been in the audio scene know mp3 on any setting can never be fully transparent all the time. Dibrom , John V, F Klemm and others have said this many times. I said it many times : what is the point of 320k cbr mp3 when -V4 can give good performance on many music at half the bitrate ? Now you are stuck with 'crap' 320k cbr looking for the next big thing and more problem samples are around the corner. its worse having 320k non-transparent than 160k encoding IMO.

The end of mp3 ? - Far from it. IMO the real advantage of MP3 is portability, small file size and quality at V3, V4 even V5 for some.

The assumption, though, is that it's transparent on most samples for most people at "transparent" settings (e.g. V2 and better).

There are exceptions - whole genres of music (though lame improves all the time), and a few gifted individuals.

Cheers,
David.

nice testing, maybe the algorithm can be improved... but the topic heading is a bit harsh.

99% of people would not hear the difference between wav and mp3 on a high end system because they are not "trained" to listen to music like audiophiles are, and depending on which samples are used artifacts can or cannot exist.

go mp3 go

High end has nothing to do with it. The smearing on some samples at 320k doesn't require too much attention. Having said that most problems aren't obvious in normal listening unless its really nasty [rare]. At the end of the day I am listening to music not clicking A/B/X. I can listen to -V4 encodings at night through the Hifi and the Grados and wouldn't tell the difference unless I had a difficult clip and the reference.

I think you just volunteered to confirm PurpleHayz's ABX results . Seriously though, please try the lossyFLAC versions (in the uploads forum) if you have time.

Cheers,
David.

Redundancies aside, I think there are more than a few folks (including some who frequent HA) that are oblivious to this fact. Prior to my recent testing, I was one of them.



Pushing the envelope. That's the point. Some are happy with good, other's want a bit more. For most things enthusiasts get revved up about, inputs and outputs are related in a non-linear fashion. Most often, marginal returns decrease as costs rise. Top speed, for instance, increases with the cubed root of brake engine power (more or less), which is why most people recognize the vast performance differential b/w a car that traps at 100mph (like a 300 HP Mustang) and one that traps at 112 (like a 420 HP Aston). Audio's no different. You still get less bang for each additional buck, and with wattage and spl, the decay on returns is even worse (logarithmic function). By your logic, there's no point to owning even a 40w/channel amp, since
it would take just 32w to produce ear-damaging volumes (101db) on even the most inefficient bookshelf speakers (say 86db). In the case of mp3 bitrates, I would argue that a doubling in filesize can be justified, even if the increase in "quality" (however we choose to define it) is several orders of magnitude less. From the studio to the track to the gym, everyone knows that returns diminish at the margins, but that never seems to stop folks from going after more. I see you own pair of Grado's as well. Tell me, are they 10 times better, in any discernible way, from a set of off-the-shelf cans costing 1/10 as much? If not why would you buy them?

In my tests, 320k improved both samples dramatically, one to the point that it became indistinguishable from source on my lower grade sound system, and barely distinguishable on my reference system. Perhaps you won't hear the same differences, but I certainly did. For other folks in the same boat, 320k is good option, and certainly better than VBR, at any setting.

--abr 260 will give a little more headroom over -V0 with 25% space saving over 320k. Quality won't suffer much if at all. Like I said you don't need great equip to hear problems. If you take hihat.wav sample smearing is easily heard at 320 k .. Saying 320 k is good for people isn't right if they can hear / are badly affected by the smearing issue its like go for a lower bitrate or another format.

If this is real - I would like to see a MPC --quality 7 ABX with these samples or even OGG -q6

MPC / vorbis / AAC at 180 k would do better. That is the one of the reasons for mpc rise years ago. MP3 would need 256~320k and not even that was enough at times. The logical [efficient] path was to tune 190k or go to another format.

I mean... this guy must hear like a bat or something... I am wondering here if MPC would beat the hell out of MP3...

If you haven't looked in this thread...

http://www.hydrogenaudio.org/forums/index....showtopic=57464

...you won't have seen that lossyFLAC hit 342kbps and 377kbps on the short samples provided by PurpleHayz.

ABX results on that would be interesting Mind you, I still can't get over ABXing V0 - it's totally beyond me.

Cheers,
David.

With lots of attention on the snare isolated, "Smooth Criminal" is perfectly ABX'able in 320 or V0.
Couldn't make a perfect ABX on "What Love", but I scored 50% in 10 rounds.

After trying (and failing, miserably) to get my g/f to successfully replicate my results, I hope if you won't mind if I recruit you to convince her that

-I'm not crazy
-the differences do exist
-and no, I didn't "rig" the A/B software just to toy with her head.

Good ear bro

p.s. I can't put the worms back in the (mp3) can, but you might want to give David's lossy FLAC experiment a try (see uploads section). Though I only made a few quick passes (nor formal testing yet), I could not distinguish the "smooth criminal" clip from the original wave.

Hi PurpleHayz.

This kind of ABX needs total concentration, and when you are concentrated you definitely can hear the difference - this is not the first time I ABX'ed a 320kbps sample. So if the person is not concetrated, it is likely to fail. Some tests you just pick up the difference at the beginning, and after 5 tries you start losing concentration.

Can you ABX your samples in MPC --quality 7 ?
I couldn't! I am wondering if you could!


MP3
===

foo_abx 1.3.1 report
foobar2000 v0.9.4.3
2007/09/13 18:23:42

File A: D:\data\download\Smooth_Criminal__b_320_clip.wav
File B: D:\data\download\Smooth_Criminal_clip.wav

18:23:42 : Test started.
18:25:26 : 01/01 50.0%
18:25:41 : 02/02 25.0%
18:25:56 : 03/03 12.5%
18:26:18 : 04/04 6.3%
18:26:24 : 05/05 3.1%
18:26:30 : 06/06 1.6%
18:26:40 : 07/07 0.8%
18:27:08 : 08/08 0.4%
18:27:21 : 09/09 0.2%
18:27:44 : 10/10 0.1%
18:28:44 : 11/11 0.0%
18:28:59 : 12/12 0.0%
18:29:16 : 13/13 0.0%
18:29:31 : 14/14 0.0%
18:29:41 : 15/15 0.0%
18:30:15 : 16/16 0.0%
18:30:28 : 17/17 0.0%
18:30:40 : 18/18 0.0%
18:31:18 : 19/19 0.0%
18:31:42 : 20/20 0.0%
18:31:49 : Test finished.

----------
Total: 20/20 (0.0%)

-------------------------------------------

MPC
===

foo_abx 1.3.1 report
foobar2000 v0.9.4.3
2007/09/13 18:45:16

File A: D:\data\download\Smooth_Criminal_clip.wav
File B: D:\data\download\Smooth_Criminal_MPC_clip.wav

18:45:16 : Test started.
18:48:00 : 00/01 100.0%
18:49:22 : 00/02 100.0%
18:49:47 : 00/03 100.0%
18:50:01 : Test finished.

----------
Total: 0/3 (100.0%)

It may be worth noting the Xbox resamples everything to 48 kHz.

edit: Wait, is that an Xbox or Xbox 360?

It's a 360. I get lazy sometimes

Hey Bourne, can you try Vorbis (latest AoTuVb build) -q7 as well? Would be interested in seeing the results.

Hi Vinnie97... I can't ABX AoTuV b5. But then -q7 is just as high as MPC@BrainDead...

Interesting, my only hope is that this original discussion can branch out into a new 'codec war' style comparison (without the blood of course!).

It's been way too long since we've had a good lossy encoder comparison, I'd love a 180 (or similar) MPC/AAC/Vorbis/MP3/perhaps lossyFLAC comparison.

- Spike

You won't motivate much people, many whom have trouble with 130 k. Lossy flac and co. cannot really compete at those bitrates too.

Thanks for your effort ... your test indicates that you seem to have some above-average hearing.

IMO, Lossless ist the way to go for you ... as you have noticed, mass storage space is really affordable these days (even if you save your entire collection on 2 different types of media like I do with HDD and DVDR).

Hey, 128/130, whatever. As long as it's a VBR comparison and a proper comparison of how popular codecs are functioning at comparable bitrates, that's the main thing.

I don't use 128 (or 180), which is why I said 'whatever' in the original post I made. But either way, it seems a lot of people make claims of how acceptable 128 kbps is (and how pointless high bitrates are), yet I see little ABX'ing of said claims.

Although to be fair, my 'feeling' is that 128 and above are pretty hard to distinguish from each other without proper listening (that is to say, if you were doing anything but concentrating 100% on the track you probably wouldn't notice). But that's a feeling, not fact

I would do one today, but I'm sick and my sinuses are clogged- not an ideal time to ABX test fiddly things like codecs.

- Spike

Well, it's certainly easier to test for the opposite.

I've been able to ABX enough samples at -V5 to know I don't want to use it. With the hardware I currently own or ever plan on owning, ~128 mp3 will never interest me; no matter how distracting my listening environment.

Switching A, B, Y, X too fast may confuse your brain - you might even guess right up to 6 chances, but then you definitely start losing that sensitivity. Take time and hear the samples with a few 10 seconds before switching to the next. I guess that is why most people can never ABX properly.