Flac 1.2.1 - Still 2 GB size limit?, flac.exe with fb2k/Converter Options

[Squeller]

Hi,

I cannot convert to flac+embedded cue > 2GB.

It's been asked here in 2005, Flac changelog says (for version 1.1.3): "Large file (>2GB) support everywhere". I'm using the latest version 1.2.1 (flac.exe called from fb2k 1.0 Beta5) - Not a file system issue (Saving to an NTFS file system).

FB2K reports:

An error occurred while writing to file (The encoder has terminated prematurely with code 1 (0x00000001); please re-check parameters) : "M:\Musik\TEMP\CDImage.flac"
Additional information:
Encoder stream format: 192000Hz / 2ch / 24bps
Command line: "M:\Musik\foobar2000\encoders\flac.exe" -s --ignore-chunk-sizes -6 - -o "CDImage.flac"
Working folder: M:\Musik\TEMP\

File size at that moment is 2.147.494.397 Bytes

This post has been edited by Squeller: Dec 29 2009, 12:35

[GHammer]

To isolate the problem, why not call flac from a command prompt instead of from foobar?
If it still fails, look at flac. If it does not fail, look at foobar.

[insane_alien]

what is the filesystem on M: ?

if its FAT then there's your problem. switch to ntfs.

[Squeller]

Isolating a problem is a question of time As I wrote, target volume is NTFS. It's all NTFS here. Windows 7 Professional. OK I encoded the 7 GB Wave file via command line. It's 24bit/192khz audio.

c:\Users\Michel\Desktop>m:\Musik\foobar2000\encoders\flac.exe image.wav -s --ignore-chunk-sizes -0 -o image.flac
INFO: Make sure you know what you're doing when using --ignore-chunk-sizes.
Improper use can cause flac to encode non-audio data as audio.
image.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=24

Now the encoding process is running... And again, after 2GB (the exact file size is 2.147.484.028 Bytes) have been processed (No, the disk is not full):

image.wav: ERROR during encoding
state = FLAC__STREAM_ENCODER_CLIENT_ERROR

An error occurred while writing; the most common cause is that the disk is full.

This post has been edited by Squeller: Dec 29 2009, 17:21

[Nick.C]

Are you sure that it is a WAV file? The WAV structure only accommodates files up to 4,294,967,294 bytes long. Sounds like the file is RIFF64 or BWF.

[Squeller]

Are you sure that it is a WAV file? The WAV structure only accommodates files up to 4,294,967,294 bytes long. Sounds like the file is RIFF64 or BWF.

Hmm, I've already deleted it. I created this wave file only for testing purposes, which made my command line life easier... The original source are multiple, independent flac files, which I have merged to one 7 GB wav file... with fb2k's converter...
BTW I'd also guess there's a file system/block issue here, still maybe on the flac side. The file size we are seeing is very close to 2^31 (2.147.483.648)

This post has been edited by Squeller: Dec 29 2009, 18:28

[jcoalson]

libFLAC uses the C stdlib for file i/o. even on my XP box with VS 2005, microsoft's stdlib implementation is still limited to 2 GB (i.e. no 64-bit off_t).

I'm reluctant to add win-specific calls to libFLAC just because MS is intentionally sabotaging portability. every other build of flac works with large files.

[Squeller]

I'm reluctant to add win-specific calls to libFLAC just because MS is intentionally sabotaging portability. every other build of flac works with large files.

Regrettably. According to this and this, large file support still seems to be a difficult story.

[fauntleroy]Josh, as we know, you are the most great-hearted flac developer of all times, you'll give us a win32/2gb+ flac anyway, one day, am I right? [/fauntleroy]



This post has been edited by Squeller: Dec 30 2009, 08:57

[lvqcl]

Does MinGW has this limitation? If no, maybe libFLAC compiled with MinGW is the solution to this problem.

[chelgrian]

libFLAC uses the C stdlib for file i/o. even on my XP box with VS 2005, microsoft's stdlib implementation is still limited to 2 GB (i.e. no 64-bit off_t).

I'm reluctant to add win-specific calls to libFLAC just because MS is intentionally sabotaging portability. every other build of flac works with large files.

VS2005 and later have _fiseek64 and friends. These take an __int64. fseek is pain anyway because it's generally prototyped to take a long rather than a off_t so you need fseeko, where the size of off_t will depend on the predefines passed to the compiler, or fseeko64 which always takes an off64_t. Even getting it to work on different unicies can be a pain due to confusion over the size of off_t and the presence or lack of o64 versions of functions. I mean just look at all the crap that configure scripts test for an idea of the historical fragmentation of the unix syscall API.

There are a whole host of other reasons to write your own wrappers around CreateFile etc. For example MS cripple the standard C library calls to only deal with paths up to 254 characters long. You have to use CreateFile with a UNICODE escape sequence at the start of the file name to get 32768 characters and it comes with lots of limitations including globbing not working. How much faff to go into rather depends on how how portable you want FLAC to be? Do you need to be able to compile it on ancient versions of Solaris for example?

Basically you end up with compiler detection and preprocessor magic and wrapper functions to sort the mess out whatever you do. For example I have reams of preprocessor magic to deal with the fact that Microsoft refuse to provide support for C99 and all the new types it specifies including fixed width ones. In order to write portable code I have headers which provide all the missing types. I also have stuff to turn off the silly warnings about using "unsafe" API calls. Which is a blatant attempt by MS to get you to use their 'secure' _s replacements which are utterly non-portable.

The other thing you can do is use something like the Apache Portable Runtime to sort the mess out for you however that comes along with the baggage of a piece of code you don't control the copyright and therefore the license of and probably isn't appropriate for FLAC.

Perhaps the answer is to start a project to write a BSD licensed portability layer, I'm certainly game to help if you want to do so.

This post has been edited by chelgrian: Dec 30 2009, 16:01

[chelgrian]

Does MinGW has this limitation? If no, maybe libFLAC compiled with MinGW is the solution to this problem.

MinGW does provide fseeko64 etc. effectively by providing wrappers around MS's C library functions. However firstly FLAC may not compile with MinGW, you have to treat it as another platform as it's a bit of a frankenmix of GNUisms and things understood by the underlying MS libraries and secondly MSVC++ almost certainly produces faster code for Windows than GCC does. However the very latest versions of GCC may have closed the gap.