Hi!

At the moment I'm encountering a strange problem with Wavpack: ~7GB WAV-Files packed with wavpack can't be restored; writing of the restored file stops at exactly 4.194.048KB, then memory usage in Taskmanager slowly rises to around 1,4GByte until "wvunpack.exe encountered a problem and needs to close".
The files are Audition 2.0 32bit-files (mentioned in another thread).

I've tried to restore with both 4.40 and 4.41b.

When encoding these big files wavpack tells me the compression ratio not as "% compressed%, but as 100-(% compressed); furthermore the encoder reaches "100% done" relatively quickly (when reaching the 4GB-"barrier"?) but it takes a healthy amount of time from then on until it really finishes.

Now I'm going to look into my backups and hope that I do have a copy of the original .wav-files left.

This is probably a silly question, but: you aren't writing to a FAT32 filesystem, are you?

-brendan

It's not a silly question , but: of course I use NTFS. FAT32 imposes a 2GB-file-limit IIRC.

If there's a limitation from the filesystem, then yes, using NTFS instead of FAT32 might help. But somehow, I think Paul Brain is already using NTFS; how could he have got the 7 GB+ audio files in the first place?

However, the WAV format has a 4 GB limit. If you had .wav files that were larger than 4 GB, then technically they weren't allowed.

http://www.hydrogenaudio.org/forums/index....showtopic=32422

Maybe you could decode to raw PCM instead?


OK so you do use NTFS:), but come to think of it, bhoar might have been thinking of multiple partitions, one NTFS and one FAT32...

I should have looked at that thread more carefully because I would have noticed that you said your WAV file was 7 Gig. I have no idea how Audition is formatting a WAV file that big, but as Kjoonlee says, that's not a valid WAV file.

What is probably happening is that Audition is writing two chunks of audio and WavPack is compressing the first one and storing the second one as RIFF data. This would explain why the percentage reaches 100% on encoding. If you do a -ss with wvunpack you should see the size of the RIFF trailer as being huge (and the audio length being shorter than the real file).

On decode, wvunpack attempts to read the RIFF data into memory, but it's too big for that (I never anticipated this situation).

If you try -i during encode you might be able to encode these files correctly, but you'd have to try it. But, of course, they're still not valid WAV files and I need to find some way to warn about this beforehand.

If you don't actually have those files backed up anywhere else, I could probably create up a special version of wvunpack that would restore them correctly (I suspect that the files are fine). I would also really like to get one of those files for examination, if possible. Will the compressed versions fit on a DVD-R?

Thanks (and sorry for the trouble)...

David

This exact same issue was also “discovered” during the alpha testing of TAK, and also initially assumed to be a limitation of TAK. I don't recall how Thomas has solved this, but the posts can be found very easily I guess since they're in the beta or alpha testing threads...

Thomas wanted to test the very large WAV files, too, of course. And my suggestion was to create a silent WAV file in a compressed folder, but he was still using Windows 98SE and therefore had no NTFS partition or something like that.

So if you want to test such large files, I think a compressed folder and a silent WAV file will do fine. Actually that compressed file will only consume a few hundred kBytes for real.

Mmm, sparse files.

-brendan

Not really, because at least 4GiB will be written to disc, when the second chunk starts. Besides there could be some markers bytes in the audio chunk even when it is digitally silent, that will probably trigger the file to be written to disc write. Besides you still cannot allocate a sparse file that is bigger than the free space left on the partition.

Using compressed files/folders only make sense, btw, if a) Brian has not enough disc space and/or b) he doesn't have Audition to create such an incompliant WAV file.

In case of b) using archive based compression is much better anyway. Someone could create the silent gigantic WAV file compress it with gzip or rar and simply upload it somewhere. I guess a silent audio will do for testing.

a) was the case with Thomas. Actually using compressed folder doesn't really help here I guess, since I rather suspect that Bryan wants to get such a file in the first place because he can't create them himself. Audition is a commercial software after all, even tho it is widely available in P2P networks...

Nevertheless using compressed folders is a "quick and cheap" way to handle large testing files when you simply don't want to waste disc space for real.

I made my four word post more as a silly aside commentary that using a compressed folder would be similar to using a sparse file if the WAV data is primarily 00s, since they way sparse files are implemented in MS's NTFS driver is tied strongly with the file compressor. I really had nothing important to add, and if my post causes confusion, then I apologize and please ignore my post.

-brendan

Please apologize my delay in response, but my PC is out of order at the moment ;-) (RMA'd graphics card) and I need some time to verify these things.


A wvunpack -ss leads to the same memory/termination-error that's happening when doing a "normal" unpack; as for the RIFF-header: how do I see it's length?

I'll try this when my PC is functioning again.

Thank you very much for your answer! You absolutely do not need to apologize as Wavpack is a great program, and as others stated, my WAV-files aren't standard-compliant. I'm pretty sure that I have those files backed up, so you do not need to waste your spare time on programming my very special wavpack version, limited edition

At the moment I do not have a DVD-writer, but I'd need someone with a DL-writer as ~5GBytes is more than what will fit on a standard DVD-R.
Perhaps I could send you such a special, compressed "silent" WAV-file? If this would fit your needs...

Thanks again for your efforts in this great piece of software!

Since I posted last on this thread I was informed of another case of this problem that wasn't exactly the same but was crashing wvunpack because audio data was mistaken for a huge RIFF chunk. So, now that I'm sure about what's going on I will be fixing the crashing issue so you'll be able to [hopefully] restore your files. And, yes, the -ss option doesn't work either!

I'm also going to put in some checks to not allow any WAV file over 4 GB and not allow large (>16 MB) RIFF sections (whether they're valid or not).

I still believe that the -i option will allow your file to compress correctly, but keep in mind that if the resulting WavPack file is over 4 GB then there will be trouble seeking. Since the part after 4 GB on your file was not being compressed at all, you should get a much smaller file than before (which probably explains the difference in compression that you were seeing between the short and long versions).

It would still be nice to get a sample 7GB WAV file from Audition. If you made it all silence, then it should compress with something like ZIP to a very small size. I actually have Audition 1.0 and I'll try the same thing with that, except that I have to backup some files before I have enough free space!

Thanks again,

David

Use a NTFS compressed folder like I said, it works great. But be sure to also compress the Temp folders. Just in case. And it's wise to disable undo, too, so hopefully the Temp folders won't get used in the first place.

I have Audition 2.0 and I can upload some Giga-WAVs...

Btw, Audition warns the user before saving a WAV that exceeds the 4GB limit:


I will upload a 9.8GB WAV (because of the second extra chunk) and a 6GB WAV. About the split version... looking at their headers there doesn't seem to be anything special about them, normal WAV files. When loading them up in Audition again, they won't get recognised as a multi-part set either.

But there was one problem: NTFS compression is pretty good (or rather "intelligent" when it comes to big blank files), in fact it compresses these files much better than for instance WinRAR's best method. The ~10GB WAV only takes up 16kb of real disc space in a compressed folder, but the RAR archive would be several MBs big. I had to find a compressor that gives similar results on "empty" files as NTFS' compression.

I guess bzip2 is pretty good and I'll use that, you can find binaries at http://www.bzip.org/ and some optimised versions can be found via the Wikipedia article http://en.wikipedia.org/wiki/Bzip2#Implementations

Ok, here are the two non-conform WAVs: http://www.hydrogenaudio.org/forums/index....showtopic=53611

Thanks for the files, Fandango, I got them. I wonder though why you didn't just upload them here when they were only a few K...?

Anyway, I'll be interested to see if my Audition 1.0 handles the "extended" wavs or whether this was something added later. It's nice though that they are finally warning the user that the wav files they are creating are non-standard. Back in the Cooledit days they had no hesitation in just making their own "standard" (i.e. with float data stored as ints).

There is the RIFF64 standard to allow such large files, I am guessing they are just setting the riff chunk size to -1 and have two data chunks.

FAT32 has 4 GB limit for file length. Buggy software might interpret the value as signed integer and function properly only up to half of that size.

Sorry for offtopic.

fixed

Haven't thought of the Upload section.

Thanks Guys for your efforts!

At the moment I can't use my PC (defective graphics card), nevertheless I'm watching this thread closely

As sort of a late follow-up on this topic, I have incorporated some changes into the latest beta (see this thread). Basically I no longer accept files over 4 GB (at least until I have full W64 support) and I don't allow over 16 MB of RIFF data (because this is certainly some sort of non-conforming or corrupt file).

I also corrected a bug in WvUnpack that will allow any WavPack files created with huge RIFF chunks to be correctly unpacked.

Sorry for the trouble and the delay...

Hi!

So if wvunpack is now able to process these files, why not let them be encoded?
I know, non-standard-conformant files aren't preferable of course, but sometimes they simply exist. In my opinion, a message like "You are about to encode a non standard compliant wav-file; there could be problems with decoding/playing this file. Do you still want to continue? Y/N" would be better, if wavpack/wvunpack is able to process these files correctly in almost any case.

But, of course, thank you very much for the improvements!

Well, the audio data is being processed as RIFF info instead of audio, which means it's not even being compressed. While these files can be restored to their original WAV format with wvunpack, they cannot be played or read by another application. I think it's best to simply refuse rather than deal with the other issues that will come up.

But files >4GB are compressed up to 4GB, only the rest of the file is a bit-per-bit-copy right?

I understand your decision of not wanting to deal with all the possible issues. If I feel adventuresome, I could still use anything up to 4.41b2...

I think it would be better to use the latest version and specify -i instead to force compression of the whole file as audio. Then you could at least play the whole thing and could always extract it as a "raw" file. Watch out though that the resulting WavPack is not > 4 GB because then it won't be seekable.

Enjoy your adventure!