FLAC file size limit and workarounds Options

[zgx]

I have a ~7 GB wav file I would like to compress to FLAC. It's a 24 bit / 48 kHz multichannel file. My problem is that when the FLAC file reaches 2 GB the encode stops and I get this error:

-: ERROR during encoding
state = FLAC__STREAM_ENCODER_CLIENT_ERROR

An error occurred while writing; the most common cause is that the disk is full.

From reading other posts here on hydrogenaudio I understand that "flac.exe" has a 2 GB file limit even if the FLAC format supports larger files.

Does anyone have any good ideas how to do the encode?

One idea was to split the huge WAV file into two smaller files and encode the separate files. The question is what tool I could use to the splitting?

[jcoalson]

technically there is no such thing as a valid 7GB WAVE file, the limit ~ 4GB. how did you get it? if you can save it as raw PCM samples you can encode it raw with flac.exe + format options

Josh

[zgx]

technically there is no such thing as a valid 7GB WAVE file, the limit ~ 4GB. how did you get it? if you can save it as raw PCM samples you can encode it raw with flac.exe + format options

Josh

It started out as a LPCM track from a Blu-ray but then I have used sox to convert it to WAV and then used WaveWizard to move the channels from "L,R,C,BL,BR,LFE" to "L,R,C,LFE,BL,BR". At this point I'm not 100% sure it's a valid WAVE or not but the sound is there. I have encoded several other big LPCM tracks this way and it has never been a problem with 4+ GB files. The problem now is that FLAC.EXE gives me this error when the resulting FLAC file passes 2 GB. In WaveWizard I can pick the output file I want. I usually go for "raw samples" and then use this command to encode the flac:

flac -8 --force-raw-format --endian=little --channels=6 --bps=16 --sample-rate=48000 --sign=signed -o "final.flac" - < temp.raw

But it does not work with this file sinc FLAC.EXE only seams to support 2 GB ouput. So the problem is not with the source file.

[Ardax]

What's the filesystem of the disk you're saving the .flac file to? It almost sounds like you're saving to a FAT32 partition.

[Junon]

What's the filesystem of the disk you're saving the .flac file to? It almost sounds like you're saving to a FAT32 partition.

Probably not. If it had been a FAT32 partition he wouldn't have been able to store the 7 GB of wave data inside a single file. The problem might rather be related to restrictions in either the technical 4 GB limit for .wav files, as mentioned above, a corrupted input stream or an issue with the FLAC encoder. Since zgx mentioned that he had already encoded +4GB wav. files prior to this latest attempt I guess we can cross out the technical restriction possibility.

zgx, have you tried playing back the .wav stream with different players yet (those which can handle the 7 GB thing)? If yes, did one of them report any errors about possible file corruptions?

[jcoalson]

ah, ok, this is on windows... as far as I know, there is no way to support files >2GB with MSVC6 (what I still have to build with) without using windows-specific APIs, which flac.exe does not do. last I checked they still don't implement posix largefile support, off_t, fseeko()/ftello(), etc.

I haven't taken the time to research and implement a separate windows-specific largefile-capable i/o path. as usual, windows is the only platform where this is even necessary.

Josh

[zgx]

I found it very hard to work on my 6 +GB WAV file so my current plan is to keep it as RAW PCM, split it into smaller parts and then process and encode the parts and put them back together as a large FLAC file.

First thing I need is a tool to do the splitting.

- I have tried shntool but it wants a wave header and if I add one it thinks the file is 47 min long and a little largen then 2 GB.

- I have tried simply splitting the file with "hjsplit". It works but has some sideeffects. I will get damaged samples in the split. The first splitted file was fine but the second splitted file I got all the audio channels mixed around so the center sound came out from my left rear and so on.


If I give my file a wave header and use shntool to split it into smaller parts those parts are playable in any audio player and are just fine. So it's nothing wrong with the audio. It's just that shntool will only give me 47 min of sound. As soon as I adds a wave header everythings goes wrong so as I said above I will try to keep it as RAW PCM as long as possible until it is in smaller pieces and then I can work with them.

ah, ok, this is on windows... as far as I know, there is no way to support files >2GB with MSVC6 (what I still have to build with) without using windows-specific APIs, which flac.exe does not do. last I checked they still don't implement posix largefile support, off_t, fseeko()/ftello(), etc.

I haven't taken the time to research and implement a separate windows-specific largefile-capable i/o path. as usual, windows is the only platform where this is even necessary.

Josh

I can trick FLAC.EXE if it does not know the size it's fine. This command works just fine with other 4+ GB Blu-ray (16 bit) LPCM tracks:

flac -8 --force-raw-format --endian=big --channels=6 --bps=16 --sample-rate=48000 --sign=signed -o "file.flac" - < file.pcm

The problems I have is that when the resulting flac file hits 2 GB I get the error mentioned.

Do you think I could copy this track to a Linux computer and it would work better there?

[wraithdu]

What about creating a cue sheet for the file and 'virtually' split it up like an album with multiple tracks. Then use something like foobar2000 to convert each 'track' into a separate flac file.

[jcoalson]

Do you think I could copy this track to a Linux computer and it would work better there?

yes, it will work on pretty much any other OS except windows.

[zgx]

What about creating a cue sheet for the file and 'virtually' split it up like an album with multiple tracks. Then use something like foobar2000 to convert each 'track' into a separate flac file.

Foobar does not accept the RAW PCM file and if I add a WAVE header it shows me the wrong run time. Since 6 GB is above the legal limit for WAV I think it's just asking for trouble to add such a header.

Is it possible to create a CUE for a RAW PCM file?

Do you think I could copy this track to a Linux computer and it would work better there?

yes, it will work on pretty much any other OS except windows.

I would prefer do to this on my Windows machine but I do have a Linux server in the closet with some free space so that should work.

Before I can encode the FLAC I need to move the channels. The channels in the LPCM source are mapped "L,R,C,BL,BR,LFE" as opposed to "L,R,C,LFE,BL,BR". I usually correct this with a tool called Wavewizard but it does not work unless I have a valid WAVE file. Do you know a good way to do this in Linux or would it still be better if I splitted the file into smaller parts so I can work with them in Windows?

[zgx]

Ok think I have solved the problem now.

Each sample in my RAW PCM file is 18 bytes
(24 bit * 6 channels = 144 bit / 8 = 18 bytes)

Since the file has no header all I have to do is to make sure the size of each file in the split is dividable by 18. So I made it easy for me and got 4 files with the size of 1800000000 bytes each (the last one smaller). And FLAC.EXE reports no broken samples so I guess it was a success.

I used the linux command "split" for the split but I guess there are others tools that could have done the job under Windows.