Hydrogenaudio Forums > files open in C

Hydrogenaudio Forums > Hydrogenaudio Forum > Scientific Discussion

wkwai

Mar 29 2004, 04:43

Hi,

I tried out this code segment is VC 6

for(i=0; i<1000; i++){
stream[i] = fopen("filename0xxx.txt", "w");
}

where "filename0xxx.txt", xxx is different for different value of i..

My program will crash! It seemed that there is a upper limit to the number of files that a program can open this way.. Is there any way to increased the number of files that can be opened .. Maybe 10 000 files ?

menno

Mar 29 2004, 04:56

QUOTE

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclib/html/_crt_file_handling.asp
The C run-time libraries have a 512 limit for the number of files that can be open at any one time. Attempting to open more than the maximum number of file descriptors or file streams causes program failure. Use _setmaxstdio to change this number.

Using win32 API may be the answer...

Menno

cabbagerat

Mar 29 2004, 07:06

QUOTE

http://msdn.microsoft.com/library/default....setmaxstdio.asp
Note This upper limit may be beyond what is supported by a particular Win32 platform and configuration.

Basically what this is saying is that having too many files open at the same time is not a good idea, it might break on some setups/platforms or Windows versions.

I am intrigued - what are you writing that could possibly require so many open files?

wkwai

Mar 29 2004, 08:12

Actually, for normal programming purposes, this limit will not be touch.. However, for R&D purposes, where "every gear change, every engine rev, the behaviour and performance of every component" has to be recorded and studied.. this limit will be exceeded very easily!

It is for designing somekind of "software telemetry".. for studying and debugging purposes..

niktheblak

Mar 29 2004, 17:13

Yes, for C stdio there is definetely a limit, probably because of stdio implementation. K&R suggests that there is a static, fixed-size array of FILE objects, to which fopen returns pointers to. Undoubdly a great number of library vendors have chosen to use this technique.

You probably need to use platform specific APIs directly, I believe that they have much higher limits on the number of open files. MSDN documentation for the Win32API's CreateFile function doesn't seem to mention that there is an upper limit at all.

kuniklo

Mar 29 2004, 18:14

QUOTE(wkwai @ Mar 29 2004, 10:43 AM)

Most operating systems allocate a table of file descriptors of fixed size to each process. There are ways to adjust the size of this table, which vary from one system to the next. In general, you shouldn't assume that you're going to be able to keep huge numbers of open files.

I'm having a hard time coming up with a good reason to do this though. The resources associated with each open file would probably be exhausted pretty quickly with so many open files (memory, etc).

wkwai

Mar 30 2004, 03:56

So, the limit is also imposed by the operating system ?

For serious R&D purposes, there is a need to keep thousands of files open ! In AAC for instance, the Main Predictors has 672 individual predictors and each predictor has many parameters which behaviors & performance needed to be tracked and analysed for every input audio frame.. You can end up with many files for a very complex software engineering project !

Of course, I could keep everything inside just one file and then design somekind of software tools to extract / interpret the individual data in this one single gigantic file ! But I thought that it is much more systematic to have all these data to be arranged in many different individual files..

wkwai

Mar 30 2004, 03:58

QUOTE(niktheblak @ Mar 29 2004, 03:13 PM)

CreateFile[/URL] function doesn't seem to mention that there is an upper limit at all.

I think I tried out a simple program with CreateFile and it also had a upper limit just as with fopen..

niktheblak

Mar 30 2004, 05:06

QUOTE(wkwai @ Mar 30 2004, 12:58 PM)

I think I tried out a simple program with CreateFile and it also had a upper limit just as with fopen..

Okay, so the operating system indeed does impose a limit. Pretty hard to circumvent that one

QUOTE

Of course, I could keep everything inside just one file and then design somekind of software tools to extract / interpret the individual data in this one single gigantic file !

This seems pretty logical to me. Design or use some existing file format from which you can extract individual data streams, maybe length-prefixed arrays of bytes or something. I don't think that should be exceedingly difficult.

However, if you could manage to cut down the number of simultaneously open files to a certain number, say 672, you could be able to work with individual files.

seanyseansean

Mar 30 2004, 05:31

QUOTE(wkwai @ Mar 30 2004, 09:58 AM)

QUOTE(niktheblak @ Mar 29 2004, 03:13 PM)

CreateFile[/URL] function doesn't seem to mention that there is an upper limit at all.

I think I tried out a simple program with CreateFile and it also had a upper limit just as with fopen..

It depends on the OS too, Windows 95 (and the variants) have far less handles than those built on the NT core, such as XP and 2000. Same goes for GDI handles.

mikeson

Mar 30 2004, 06:26

I can confirm there is no such limit in C# and .NET (I've created 10000 files at time).
I know that this thread is about C, but what about trying Managed C++ (or C# of course

) and .NET Framework?

wkwai

Mar 31 2004, 01:20

QUOTE(mikeson @ Mar 30 2004, 04:26 AM)

I can confirm there is no such limit in C# and .NET (I've created 10000 files at time).
I know that this thread is about C, but what about trying Managed C++ (or C# of course

) and .NET Framework?

Have you writtened a small program to test this problem in C# including writing some random data into each of the 10 000 files ?

mikeson

Mar 31 2004, 07:31

QUOTE(wkwai @ Mar 31 2004, 08:20 AM)

Have you writtened a small program to test this problem in C# including writing some random data into each of the 10 000 files ?

Yes, for example with this code:

CODE

namespace ProjectSoft.User
{
using System;
using System.IO;

public class Runner
{
public static void Main()
{
StreamWriter[] myWriters = new StreamWriter[10000];

for (int i = 0; i < myWriters.Length; i++)
{
myWriters[i] = new StreamWriter(String.Format("{0:0000}.log",i));
myWriters[i].Write(String.Format("{0:0000}.log",i));
myWriters[i].Flush();
}
}
}
}

[EDIT] Some cleaning, replaced .Close() with .Flush() and removed statement using(myWriters[i] = new...), because caused disposal of object (calling method Dispose()) in the end of scope and that was not intended (=close file)[/EDIT]

wkwai

Mar 31 2004, 07:47

hmmmm.. funny that it is not an operating system problem.. I have not tried out C# yet so I can't be sure.. but you close the stream before opening a new stream..

What we are talking here is keeping 10 000 streams open at all time !

mikeson

Mar 31 2004, 07:55

You can replace myWriters[i].Close(); with myWriters[i].Flush(); and keep stream open and it will still work.

Moitah

Mar 31 2004, 08:18

I tried with this code and it works fine:

CODE

StreamWriter[] myWriters = new StreamWriter[10000];

for (int i = 0; i < myWriters.Length; i++) {
myWriters[i] = new StreamWriter(String.Format(@"G:\test\{0:0000}.log",i));
}

for (int i = 0; i < myWriters.Length; i++) {
myWriters[i].Write(String.Format("Writing to log {0:0000}",i));
}

for (int i = 0; i < myWriters.Length; i++) {
myWriters[i].Close();
}

I'd like to try it with CreateFile in C++ as well.

Moitah

Mar 31 2004, 09:19

Here's the code I used to test CreateFile, I was able to open 10,000 files (didn't try any more than that):

CODE

#include <stdio.h>
#include <windows.h>

int main()
{
const int numFiles = 10000;
HANDLE hFiles[numFiles];
char buff[256];
int i;

for (i = 0; i < numFiles; i++) {
sprintf(buff, "G:\\test\\%05d.txt", i);

hFiles[i] = CreateFile(buff, GENERIC_WRITE, 0, NULL, CREATE_NEW, FILE_ATTRIBUTE_NORMAL, 0);
if (hFiles[i] == INVALID_HANDLE_VALUE) {
printf("Unable to create file %d\n", i);
}
}

for (i = 0; i < numFiles; i++) {
if (hFiles[i] != INVALID_HANDLE_VALUE) {
DWORD bytesWritten;
sprintf(buff, "Writing to file %d", i);

if (!WriteFile(hFiles[i], buff, strlen(buff), &bytesWritten, NULL)) {
printf("Unable to write to file %d\n", i);
}
}
}

for (i = 0; i < numFiles; i++) {
if (hFiles[i] != INVALID_HANDLE_VALUE) {
CloseHandle(hFiles[i]);
}
}

return 0;
}

I'm using Windows 2000.

wkwai

Apr 2 2004, 07:46

Thanks.. I think you solved my problem !! I must have used OpenFile instead of CreateFile sometime ago !

niktheblak

Apr 2 2004, 09:09

QUOTE(wkwai @ Apr 2 2004, 04:46 PM)

Thanks.. I think you solved my problem !! I must have used OpenFile instead of CreateFile sometime ago !

Yes, this could very much be the case. OpenFile was deprecated in favour of CreateFile for Win32.

From MSDN:

QUOTE

Note This function is provided only for compatibility with 16-bit versions of Windows. New applications should use the CreateFile function.

Apparently OpenFile has some serious limitations compared to CreateFile.

This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.