WavPack Hybrid broken on (Arch?) Linux, non lossless and heavily distorted Options

[skamp]

I tried to encode and decode several different WAV files from CD rips with WavPack (version 4.60.1) in hybrid mode, on Arch Linux, on two different machines (Intel Core i7 and Xeon, x86_64). I always got the same result: the files are heavily distorted (groups of samples repeating themselves), and obviously non lossless. See samples.

I ran the following commands:

CODE

wavpack -b400 -c -m hybrid.wav
wvunpack -m hybrid.wv

wvunpack outputs the following:

CODE

WVUNPACK  Hybrid Lossless Audio Decompressor  Linux Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

original md5:  e180f2ec2c4c65fd214bc6c4f70c7988        
unpacked md5:  2d5edeb0d3d7ba971282428b77fd41c6        
MD5 signatures should match, but do not!

I tried the win32 binary, which worked fine (lossless result). I don't have another Linux distro installed anywhere to see if this problem is distribution-specific. I also tried three different builds (including one built from source by myself), always with the same result.

This post has been edited by skamp: Jul 5 2012, 18:59

[plus]

I can confirm this bug exists on Debian Wheezy amd64, but does NOT exist on Debian Wheezy kfreebsd_amd64.

On the Debian Wheezy amd64 (linux) machine:

CODE

plus@honoka:~$ wavpack -b400 -c -m hybrid.wav

WAVPACK  Hybrid Lossless Audio Compressor  Linux Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

original md5 signature: e180f2ec2c4c65fd214bc6c4f70c7988
created hybrid.wv (+.wvc) in 0.38 secs (lossless, 41.39%)
plus@honoka:~$ wvunpack -m hybrid.wv

WVUNPACK  Hybrid Lossless Audio Decompressor  Linux Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

overwrite hybrid.wav (yes/no/all)? yes

original md5:  e180f2ec2c4c65fd214bc6c4f70c7988
unpacked md5:  2d5edeb0d3d7ba971282428b77fd41c6
MD5 signatures should match, but do not!

On the Debian Wheezy kfreebsd_amd64 machine:

CODE

eric@nene:~$ wavpack -b400 -c -m hybrid.wav

WAVPACK  Hybrid Lossless Audio Compressor  GNU/kFreeBSD Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

original md5 signature: e180f2ec2c4c65fd214bc6c4f70c7988
created hybrid.wv (+.wvc) in 0.47 secs (lossless, 39.23%)
eric@nene:~$ wvunpack -m hybrid.wv

WVUNPACK  Hybrid Lossless Audio Decompressor  GNU/kFreeBSD Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

overwrite hybrid.wav (yes/no/all)? yes

original md5:  e180f2ec2c4c65fd214bc6c4f70c7988
unpacked md5:  e180f2ec2c4c65fd214bc6c4f70c7988
restored hybrid.wav in 0.35 secs (lossless, 39.23%)

Both computers are running the latest version of wavpack in the Debian repositories.

[2012]

Fellow Archer here.

I confirm the bug. I got non-matched MD5s (albeit the corrupted one is different from what's reported here).
I confirm that FLAGS set in the build environment are not the cause.
I confirm that this is not a compiler-specific bug (both gcc and clang yield the same result).

What I discovered is that building with "gcc -m32" or "clang -m32" fixes the bug. So this looks like an architecture-related bug.

[Angenial]

The bug itself appears not to be architecture-related, although due to implementation differences in glibc, it manifests itself (on Linux) only on amd64.

For those interested, memcpy() is being used to copy overlapping regions of memory. The result of such a copy is undefined, which means it might work, but it doesn't have to. glibc on x86 appears to copy in such a way that corruption does not occur, but on amd64 it does not. This is almost certainly because optimized versions of memcpy() are being used, and differences in the architectures mean differences in how to best implement the function. The fact that the C standard makes the copying of overlapping areas undefined allows for these optimizations, resulting in faster code. There is another function, called memmove(), that can copy between overlapping regions.

I've sent a patch off to David, assuming he hasn't already seen this thread and fixed the problem.

[2012]

Good catch Angenial.

Using memmove() instead of memcpy() did the trick indeed.

[lvqcl]

Links about memcpy() behaviour:

https://bugzilla.redhat.com/show_bug.cgi?id=638477
http://sources.redhat.com/bugzilla/show_bug.cgi?id=12518
(Torvalds vs Drepper, ...)

This post has been edited by lvqcl: Jul 5 2012, 22:39

[bryant]

Thanks guys! I have checked Angenial's patch into SVN and will post something to the mailing list.

I didn't read the whole redhat thread, but I agree with Linus that changing memcpy() to copy backwards is silly, especially if it isn't any faster and breaks things. Of course I know that memcpy() is not guaranteed to work with overlapping regions, but I'm sure that when I did that I was just lazy and assumed that I'd be okay if dst < src. Won't happen again!

[bryant]

I knew this sounded familiar!

http://www.hydrogenaudio.org/forums/index....st&p=788710

[Kohlrabi]

Links about memcpy() behaviour:

https://bugzilla.redhat.com/show_bug.cgi?id=638477
http://sources.redhat.com/bugzilla/show_bug.cgi?id=12518
(Torvalds vs Drepper, ...)

That was irritating, entertaining, scary and plain hilarious at the same time. Thank you for linking that.

This post has been edited by Kohlrabi: Jul 6 2012, 00:55

[skamp]

Any ETA on a bugfix release?

[bryant]

I guess I’ve sort of dropped the ball on this. I am always hoping that I’ll get a chance to do a major release soon and so I can skip a minor release for just this fix, especially since it doesn’t affect too many users (Linux, 64-bit, and a mode which isn’t even supported in gstreamer). But I have accumulated some other small fixes and it’s probably time for a minor release just to avoid the appearance of abandonware. I’ll start looking into this and shoot for sometime later this month.

Thanks for the reminder...

David

[BearcatSandor]

I'm a bit stymied as i've been using wavpack-4.60.1 since june 5 and i only ran into this problem for the first time last week (gentoo system).

Is there a patch available to fix this yet?

Thanks for all the great work!

[Destroid]

...it’s probably time for a minor release just to avoid the appearance of abandonware

No way, sir! The idea of WavPack being abandonware in the lossless circuit is laughable =D

I think WavPack is the most versatile of its kind, despite that I have a lot of favorite audio codecs (both lossless and lossy).

Putting all my cards on the table, my initial peeve with WavPack was separate binaries for encoding and decoding. I am sure there are bountiful reasons for this and I was able to compensate due to my continued interest.

I apologize if I interrupted the discussion (I personally run Windows and use .WV primarily for 32-bit float) but I could not resist to extend another "thanks" to David. IMO WavPack will be around for a long time

[Mr_Rabid_Teddybe...]

I'm a bit stymied as i've been using wavpack-4.60.1 since june 5 and i only ran into this problem for the first time last week (gentoo system).

Is there a patch available to fix this yet?

Thanks for all the great work!

It's fixed in SVN.

CODE

mkdir wavpack
cd wavpack
svn co http://svn.slomosnail.de/wavpack/trunk/
cd trunk
./autogen.sh
./configure --enable-mmx --enable-man
make
sudo make install

[BearcatSandor]

Thanks for the link to the SVN and instructions. I complied and installed and i still get the same stuttering and false starts. I double checked to make sure i was using the correct binary. Might be something else on my system.

[bryant]

Thanks for the link to the SVN and instructions. I complied and installed and i still get the same stuttering and false starts. I double checked to make sure i was using the correct binary. Might be something else on my system.

I assume that this bug can affect both the command-line programs and the installed libwavpack (which, for example. gstreamer uses). Do you see the problem when you decode from the command-line (and did that change when you built and installed the new version)? Or is this something that only happens in a player? If it only affects hybrid lossless (wv + wvc) and sounds like the sample at the top of the thread, I'm sure it's the same issue.

BTW, thanks Destroid!

[BearcatSandor]

Bryant,

Thanks for asking. It happens when using command line tools using both the package compiled from the gentoo repos and the package i compiled manually from svn. If i take a wav file and encode it using wavpack -b400 -c -m hybrid.wav (to use skamp's example for testing purposes) and then play the resulting file in any media player i get the stuttering and distortion. I've tried mplayer, mpc, and gstreamer just to try different libraries with it.

If i then decode the wv+wvc back to wav and play the wav in any of those it displays the same behavior. Sometimes the stuttering starts right away and sometimes it's seconds into the track but the stuttering is exactly the same in both the wv+wvc and the decompressed wav. I don't know that mplayer is actually utilizing the wvc file but mpd is and the same things happen in all 3 players.

What confuses me is that i've been using wavpack-4.60.1 since June 5th and this problem only started occurring on November 28th. The only thing that i'd updated on my system when it stopped working was virtual/libiconv-0. That's a virtual package for gnu charset conversion for libc. What that would have to do with it, i've no clue. It possible that i rebooted the pc and something that was updated a while ago got read into memory (i don't reboot for months at a time usually). The previous kernel i was using was 3.4.9 from august 27th, though on nov 21st i did go from linux-headers-3.4.r2 to linux-headers-3.6. I have recompiled the repo's wavpack package a few times and done a revdep-rebuild which checks for packages that need to be recompiled due to changed libraries and headers to no avail.


sudo uname -a:
Linux jaguarundi 3.5.7-gentoo #1 SMP PREEMPT Tue Nov 13 13:03:54 MST 2012 x86_64 Intel® Core™2 Quad CPU Q9300 @ 2.50GHz GenuineIntel GNU/Linux
(it's a preempt kernel but jack is not running)

If there's any more testing i can do or info i can provide, i'd gladly provide it. If there is a bug list that i can copy this post over to, i'd be glad to.

Thanks Bryant!


Bearcat

[bryant]

Well, the first thing is do you get the same MD5 mis-match message that skamp reported, no matter which version of wvunpack you use? And I assume that when you do a regular lossless encode, everything works fine? Also, have you tried to run the version you built locally directly from the build directory (with “cli/wvunpack” from “trunk”)?

I assume that when you play in the players, everything works fine if you move or rename the .wvc file, right?

At this point, the only thing I can imagine is that you aren’t really using the new library,. That seems much more likely to me than that you’ve discovered a new bug. Unfortunately, I am not real clear on exactly how Linux libraries work, so I can’t offer solid advise on checking or verifying that...perhaps someone else can chime in on the proper procedure.

Sorry you’re having trouble!

[BearcatSandor]

Got it! Thanks Bryant and Mr_rabid_teddybear for your assistance. The svn compile was throwing things into the wrong directories. After i figured out what the correct config flags should be all was well. Now i'm having an issue with files that have covers in the tags but that's another subject for another thread.


In short, the snv code works and fixes this bug!

Thanks!!

[skamp]

I found another bug (still with version 4.60.1, Linux x86_64) with a 24 bit / 88.2 kHz / 6 channel WavPack file (lossless, not hybrid).

CODE

$ wvunpack -v surround88.wv
verified surround88.wv in 0.49 secs (lossless, 91.44%)

CODE

$ wvunpack -m surround88.wv
original md5:  ea86d287d00cc8bf0d2cd1998642e595        
unpacked md5:  2b7a7604435f12e0ef2d5167fe088fc8        
MD5 signatures should match, but do not!

I transcoded it from this FLAC file, freely available online: http://download.linnrecords.com/test/flac/surround88.aspx

[bryant]

Thanks, Skamp!

Yeah, the 64-bit Linux memcpy() bug could also affect lossless multichannel files. I verified that your sample works correctly with the new version on my system (which exhibits the issue).

David

[skamp]

OK, cool!