The 2GB Wav-limit!? Options

[Quantos]

Hello everybody,

I just wanted to know if there is by any chance a program which can
record wav-files larger than 2GB. The Wavelab-lite version I'm using
right know stops any recording which is exceeding that limit.

Thanks for your help!

[CiTay]

There are a number of programs that have problems with files larger than 2 GB. The WAV format itself is officially limited to 2 GB.

You could use a program that records to a lossy format on-the-fly, like Messer, that can record to MP3. Or you cut the recording in between... i mean, 2 GB, that's over 3 hours in one piece...

[ChiGung]

The common wav header can only allocate 4 bytes to state the total wav length in bytes which will be why there's a two Gig limit. That detail could be depreciated (make -1 = read till the end of file) -would be nice, might crash old players hmmm...

Since
Linco.exe can write a raw PCM file (wav file without any headers) it may not have its own restriction on length. That would allow a continuous recording, then have to reencode or chop up and format the raw.

[jaybeee]

Hello everybody,

I just wanted to know if there is by any chance a program which can
record wav-files larger than 2GB. The Wavelab-lite version I'm using
right know stops any recording  which is exceeding that limit.

Thanks for your help!

Audacity can. And it's free... and very good.

[cliveb]

The common wav header can only allocate 4 bytes to state the total wav length in bytes which will be why there's a two Gig limit.

This isn't quite correct. It is true that the file length in the WAV header is a 4 byte integer, but it's an *unsigned* number, which means that the maximum size of a WAV file is actually 4GB, not 2GB.

Some audio programs get this wrong, and interpret the file length as a signed number, thus imposing a 2GB limit. I can confirm that Wave Repair supports WAV files up to 4GB, and if used just for recording it's free. And I see someone else has pointed out that Audacity (also free) will also go above 2GB.

[ChiGung]

The common wav header can only allocate 4 bytes to state the total wav length in bytes which will be why there's a two Gig limit.

This isn't quite correct. It is true that the file length in the WAV header is a 4 byte integer, but it's an *unsigned* number, which means that the maximum size of a WAV file is actually 4GB, not 2GB.

Theres no need to correct to clarify
Wether the number is read as signed or not only becomes an an issue beyond 2g.
The 'actual' maximum size neednt have a limit if handled ideally.

[kjoonlee]

... which begs the question, how do you overcome the 4GB limit?

Using -1?

[Sebastian Mares]

... which begs the question, how do you overcome the 4GB limit?

Using -1?

No, because -1 is signed and the size descriptor in the Wave header is unsigned. -1 would be 0xff 0xff 0xff 0xff which already stands for the unsigned 4294967295.

I guess there is no way, since it is a limitation of the format. You could use raw PCM and wrap it into a Matroska container (if supported). If I am not mistaken, the WAV container was invented by Microsoft and at that time, FAT16 was the only HDD file system supported by Windows - which by design has a file size limitation of 2 or 4 GB (not exactly sure).

[ChiGung]

... which begs the question, how do you overcome the 4GB limit?

Using -1?

No, because -1 is signed and the size descriptor in the Wave header is unsigned. -1 would be 0xff 0xff 0xff 0xff which already stands for the unsigned 4294967295.

If a program wont read over 2gigs, the size descriptor never gets to have a sign because the msb is never used. Data has no sign or specific meaning - theres what maybe it should mean, what it can, and what it usualy does...
By -1 i meant 0xffffffff to be used as an escape value.

I guess there is no way, since it is a limitation of the format.

Its a limit of how the format is usualy read. An ideal wav reading routine can read to the end of file or use filesize-headersize to compute the PCM length.

You could use raw PCM and wrap it into a Matroska container (if supported). If I am not mistaken, the WAV container was invented by Microsoft and at that time, FAT16 was the only HDD file system supported by Windows - which by design has a file size limitation of 2 or 4 GB (not exactly sure).

The wav format was invented - heh like monkey poke stick in log pickup ants

[cliveb]

I guess there is no way, since it is a limitation of the format.

Its a limit of how the format is usualy read. An ideal wav reading routine can read to the end of file or use filesize-headersize to compute the PCM length.

Actually, this won't work. A WAV file is free to have arbitrary non-PCM data at the end, and some audio processing programs do in fact place proprietory information there. As well as the file length near the front of the RIFF header, there's also a "chunk length" (another unsigned 4 byte number) at the front of the data chunk. Nothing beyond the defined length of the data chunk should be interpreted as PCM data.

If I am not mistaken, the WAV container was invented by Microsoft and at that time, FAT16 was the only HDD file system supported by Windows - which by design has a file size limitation of 2 or 4 GB (not exactly sure).

FAT32 also has a file size limit of 4GB. Even if the WAV standard was changed to support longer files, they would only be possible on NTFS partitions. Lots of people still use FAT32, so it's simplest to just accept that 4GB is the limit.

[xmixahlx]

http://www.uwsg.iu.edu/hypermail/linux/ker...912.3/0009.html

anyways, audacity supports >2/4GB thanx to filesystems in linux > kernel 2.2 (or with LFS patch).

see http://linuxreviews.org/sysadmin/filesystems/


later

[ChiGung]

Its a limit of how the format is usualy read. An ideal wav reading routine can read to the end of file or use filesize-headersize to compute the PCM length.

Actually, this won't work. A WAV file is free to have arbitrary non-PCM data at the end, and some audio processing programs do in fact place proprietory information there.

Thats a point - but it is also free not too eh? To expand the wav format, make some very unlikely filesize like max-3 equal an escape value which implies a PCM size 'overflow'.
-a proprietory deviation from the not so common compliancy ~?

As well as the file length near the front of the RIFF header, there's also a "chunk length" (another unsigned 4 byte number) at the front of the data chunk. Nothing beyond the defined length of the data chunk should be interpreted as PCM data.
FAT32 also has a file size limit of 4GB. Even if the WAV standard was changed to support longer files, they would only be possible on NTFS partitions. Lots of people still use FAT32, so it's simplest to just accept that 4GB is the limit.

Nice and comfy for programmers to keep the format capped.

This post has been edited by ChiGung: Mar 17 2005, 18:41

[NumLOCK]

Nice and comfy for programmers to keep the format capped.

Well, that's how it goes by Microsoft standards. They're designed to sell well, not to be future-proof.

Most modern formats use variable-length encoding for integers:
- you actually gain bits when dealing with small files
- you have virtually no limit on file size.

The wav format is so simple, that it would be better to just use something else when dealing with big files, instead of adding ugly special-cases that will work in 10% of players, and will make people believe their files are corrupt.

What will happen when a user creates a long WAV file for a family video, and it won't play in Windows media player ?

Even worse, if you use this kind of hack with a legacy audio editor, it will truncate your audio without warning !

Good standards deserve to last, but IMHO, this ain't one of them.

[ChiGung]

.....
What will happen when a user creates a long WAV file for a family video, and it won't play in Windows media player ?

Even worse, if you use this kind of hack with a legacy audio editor, it will truncate your audio without warning !

The max-3 escape 'implication' i mention could only possibly misinterprate PCM files of exact size max-3 (highly improbable length and impossible with 16bit or stereo) -which have also postfixed metadata.*

Good standards deserve to last, but IMHO, this ain't one of them.

Its so common, for a container of PCM I think it should evolve rather than be ditched and replaced. Its a choice between k.i.s.s for the programmers or the users

edit* -or just fail to read because of the filesize i admit - this happens already...

This post has been edited by ChiGung: Mar 17 2005, 21:00

[NumLOCK]

I completely agree: more things should evolve instead of getting replaced every few years, that would prevent much useless re-development...

But wave ? I mean, if you do anything than switching between sample formats, you need to rewrite all applications that read or write wave files...

The problem is that, in the first place, there was not much design to evolve from.

This page sums it up pretty well:

http://www.borg.com/~jglatt/tech/wave.htm

The format cannot be improved much, without breaking compatibility with most existing programs. So when new features are needed, why not use the better alternatives ?

[Mr_Rabid_Teddybe...]

I think I read somewhere that aiff does not have this restriction. Is this correct?

[ChiGung]

I completely agree: more things should evolve instead of getting replaced every few years, that would prevent much useless re-development...

But wave ? I mean, if you do anything than switching between sample formats, you need to rewrite  all applications that read or write wave files...

Not if the 'simple' form PCM size specifier is escaped for file sizes over 4 gig. Old players wouldnt play files that needed to use the escape, old editors couldnt work with the files or convert them to their own format, but they couldnt anyway cause the file is just too big.
Its backward compatibilty for all the files which dont use the extended feature.

Maybe its too much fuss, but its not a hard limitation of the format that it cant be extended past 4gig. In practice i guess theres enough work to do if many wav openers misinterpret the 4byte addresser as signed.

This page sums it up pretty well:

http://www.borg.com/~jglatt/tech/wave.htm

A Bastardized Standard
The WAVE format is sort of a bastardized standard that was concocted by too many "cooks" who didn't properly coordinate the addition of "ingredients" to the "soup". Unlike with the AIFF standard which was mostly designed by a small, coordinated group, the WAVE format has had all manner of much-too-independent, uncoordinated aberrations inflicted upon it.

I could have my stance skewed by my embryonic experiments in wav file handling but the Articles comments:
'much-too-independent'- is irrelevant, probably owing to an unrelated interest of the Authors; p (fine page though thankyou)
- If one wishes to inflict aberrations on a format, that is impossible to do independantly.
If a format can be worked within to include prefered capabilities without breaking its compatibilty (less the capabilities) -thats a programmers perogitive.
Good stuff should eventually get coordinated into the format

The common standard for packing chunks of pcm data could just accomodate simple wav formats with escape values to quirky ones. I dont see a point just now in sticking to origional limitations... hmm..

[NumLOCK]

CODE

typedef struct {
 ID             chunkID;
 long           chunkSize;

 short          wFormatTag;
 unsigned short wChannels;
 unsigned long  dwSamplesPerSec;
 unsigned long  dwAvgBytesPerSec;
 unsigned short wBlockAlign;
 unsigned short wBitsPerSample;

/* Note: there may be additional fields here, depending upon wFormatTag. */

} FormatChunk;

CODE

typedef struct {
 ID             chunkID;
 long           chunkSize;

 unsigned char  waveformData[];
} DataChunk;

You could always define wFormatTag=1, as if audio data were compressed, set chunkSize to 0xFFFFFFFF, and then store the actual size in the compressed file info.

But then, you could not store the real size in DataChunk, because there's not enough room. Or you could use a spacial value of wFormatTag, that would trigger various changes in the format.

Anyway, none of these changes would be compatible with existing apps

Then, instead of redefining two data structures in an incompatible way, why not use mka ?

This post has been edited by NumLOCK: Mar 18 2005, 09:41

[cliveb]

You could always define wFormatTag=1, as if audio data were compressed, set chunkSize to 0xFFFFFFFF, and then store the actual size in the compressed file info.

formatTag=1 means *un*compressed. (I take it this is a typo, and that's what you meant). It could work, because it's impossible for a chunk to have a size as large as xFFFFFFFF.

But then, you could not store the real size in DataChunk, because there's not enough room. Or you could use a spacial value of wFormatTag, that would trigger various changes in the format.

Anyway, none of these changes would be compatible with existing apps 

Let me first say that I feel the WAV format shouldn't be messed around with. 4GB gives you over 6 hours of 44.1/16, which seems quite a long time. The only reason for having bigger files strikes me as being able to make very long unattended recordings. The obvious solution is for the recording program to produce multiple WAV files.

However, if what people are trying to arrive at is a way to support single WAV files longer than 4GB while remaining compatible with existing programs, then the most straightforward approach seems to be:

1. Put the first 4GB of samples (minus however many bytes are required for headers) into the data chunk.
2. Put the rest into multiple proprietory chunks (let's call them "more" chunks) after the data chunk. Each chunk has a maximum length of 4GB (because its length field must be an unsigned 32 bit integer, so that these chunks can be ignored by compliant programs).
3. Include an additional proprietory chunk before the data chunk that indicates the presence (and number) of "more" chunks. Each pointer to a "more" chunk could include a file offset as large as we like (eg. using a 64 bit integer).
4. Leave the file_length field in the RIFF header set to *exclude* the "more" chunks. (It couldn't include them anyway, because the file is longer than 4GB). This means that compliant programs which don't know about "more" chunks will still work provided they ignore unknown proprietory chunks (albeit processing only the first set of samples, in the data chunk).

But in summary, here is not the place to be making off-the-cuff design extensions to a mature standard which actually works suprisingly well.

[NumLOCK]

formatTag=1 means *un*compressed. (I take it this is a typo, and that's what you meant).

Oops.. indeed -- and it proves my lack of practice with wav

3. Include an additional proprietory chunk before the data chunk that indicates the presence (and number) of "more" chunks. Each pointer to a "more" chunk could include a file offset as large as we like (eg. using a 64 bit integer).

Wouldn't this cause problems with some badly written, unmaintained programs ? I've seen some "optimized" programs which don't... umm.. parse everything

But in summary, here is not the place to be making off-the-cuff design extensions to a mature standard which actually works suprisingly well.

Agreed (was trying to discourage him a bit, actually)

[madoka]

I think I read somewhere that aiff does not have this restriction. Is this correct?

No, this is incorrect. AIFFs have a maximum size of about 2 GB, for the chunk size field in the chunk header is a 32-bit signed integer. Moreover, each AIFF can only have 1 sound data chunk.

http://www.tsp.ece.mcgill.ca/MMSP/Document.../AIFF/AIFF.html

[cliveb]

3. Include an additional proprietory chunk before the data chunk that indicates the presence (and number) of "more" chunks. Each pointer to a "more" chunk could include a file offset as large as we like (eg. using a 64 bit integer).

Wouldn't this cause problems with some badly written, unmaintained programs ? I've seen some "optimized" programs which don't... umm.. parse everything

All bets are off with regard to badly written programs. But all WAV processing programs really should at the very least ignore any chunks they don't recognise. So this extra chunk (which will contain the required 4 byte integer defining its size) should just be skipped. The very large offsets to other chunks that it contains would never be seen.

Actually, come to think of it, this additional chunk before the data chunk isn't necessary: a program using the extra "more" chunks could just notice that they are present and process them. Other programs will ignore them. And even badly written programs that assume the data chunk always comes right after the fmt chunk would still work.

(Question to self: why am I even thinking about this, when my opinion is that it shouldn't happen???!!)

[NumLOCK]

Ah, ok. Good idea..

Then the first, legacy chunk could perhaps be dummy (ie: of zero length), so that legacy programs (buggy or not) will see nothing at all, unless they are upgraded ?

But then, when opening one of these new wave files with an old program and saving it, all newer chunks would get destroyed

This post has been edited by NumLOCK: Mar 18 2005, 16:40

[ChiGung]

But in summary, here is not the place to be making off-the-cuff design extensions to a mature standard which actually works suprisingly well.

well, i just find it interesting to *try* and point out possibilities, the replies help my interests.

cheers

This post has been edited by ChiGung: Mar 18 2005, 19:53

[Kurt S]

You could always use Adobe's Audition in the multitrack mode. When the temp file reaches the 4 gig limit, it creates another one. You can keep recording untill your hard drive is filled up. Saving the file will be an issue but you could always save several 4 gig chnks that make up the whole recording session or save as mp3 as someone else mentioned earlier.