seekg and large numbers

[LwSiX]

Hello there, i just have a quick question. Im trying to parse large files 4gb+ and ive run into a serious problem with seekg, trying to start the pointer at 5,000,000,000 isnt working and it gives me a compiler warning....

warning C4305: 'argument' : truncation from '__int64' to 'long'

How can i work around this?

Thanks

[Link777]

According to what I've found...

"signed int s32" is a signed integer guaranteed to be 32 bits on all platforms. Values of this type can range from -2,147,483,648 to 2,147,483,647.

Therefore rewrite the seek function (you did mean seek, and not seekg, correct?) as something like "MySeek", and use a different type of integer, but it won't be 32 bits!


According to the helpfile of Irrlicht (1.3.1),

Code: Select all

32 bit signed variable. 

This is a typedef for signed int, it ensures portability of the engine. 

Definition at line 43 of file irrTypes.h.

And line 43 of irrTypes.h?

Code: Select all

typedef signed int              s32;

Also: If you look a little lower, there's a 64-bit type..

Code: Select all

00047 // 64 bit signed variable.
00048 // This is a typedef for __int64, it ensures portability of the engine. 
00049 // This type is currently not used by the engine and not supported by compilers
00050 // other than Microsoft Compilers, so it is outcommented.
00051 //typedef __int64                               s64;

(Experts feel free to correct me if I'm wrong, as I've only been learning C++ about two-three days now. Although, I hope that I'm touching close to the answer.)

Cheers!

Hope I helped.

// Erik

[hybrid]

We only support 32bit file adressing as of now, I've never used larger files. But I guess that we simply have to adapt to the actual fseek signature, which uses long instead of int. That way a 64bit system should get a 64bit value which is large enough. I'll check.

[vitek]

Well he's actually talking about the C++ standard library. This doesn't really have anything to do with Irrlicht [yet].

Anyways, just having a larger integer type will not help. The istream::seekg() method takes an istream::pos_type or off_type, which, when boiled down on most C++ standard library implementations will be a long. Even if you did change things so that those types were 64-bits, the problem is going to be that the basic_filebuf that actually accesses the file is not likely to be using 64-bit aware file access methods. Most likely it is implemented as a wrapper around the C file stream methods [fopen, fclose, fputc, fgetc, ...] which aren't 64-bit aware.

If you are going to be dealing with files that are bigger than 2GB you will need to do some work on your own. Things won't be super portable as each vendor has their own names for the large file access functions.

Anyways, you could either write your code to use the large file methods directly, or you could create your own streambuf derived type that used the large file functions.

Travis

[vitek]

Another node while I'm here. The size of int and long are implementation defined. _Most_ 32-bit desktop operating systems use a 32-bit int and a 32-bit long, and most 64-bit desktops use a 64-bit long, but that isn't required. The long long types are usually 64-bits.

If you really want 64-bits, you should use the types from inttypes.h, which have the size written into the type name [int32_t, uint8_t, and such]. The Irrlicht engine makes some assumptions about the sizes of the native types. For Irrlicht to detect this stuff correctly, it would need a configuration step or some fancy template stuff to correctly assign the types.

Travis

[drac_gd]

FOR WINDOWS
//------------------------------------------------------------------------------
/// File seek..
__int64 VxFileSeek ( HANDLE hFile, __int64 u64DistanceToMove, U32 u32SeekHow )
{
LARGE_INTEGER li;
li.QuadPart = __int64;
li.LowPart = SetFilePointer (hFile, li.LowPart, &li.HighPart, u32SeekHow);
if (li.LowPart == INVALID_SET_FILE_POINTER && GetLastError() != NO_ERROR)
{
li.QuadPart = -1;
}
return li.QuadPart;
}
FOR LINUX look up fseek64

[vitek]

Windows has _fseeki64.

[Link777]

I'd advise just making the file that you're reading spread apart in many files, like data0.dat, data1.dat, data2.dat, etc.

It'd make things quicker as well.