Binary to ASCII

[LunaRebirth]

Hello!
Sorry for this terrible of a question.
I've looked everywhere for an answer, and nothing seems to work!!!

I'm trying to make file sending. So what happens is, I could send a file with words and such. But when sending a .exe or .bmp or anything with random letters in it, it stops after the first space.
So I'm going to convert text to binary on the server, send to the client, then the client converts binary to ASCII and outputs it into a file.

I can easily send the binary code, and that's all complete. But my current problem is how do I convert the binary code to ASCII???

Here's what I'm doing right now:

Code: Select all

//fullFile2 is the received binary file!
std::string fullFile = "";
for (int i = 0; i < fullFile2.length()/8; i++) {
            char c = strtol((char*)fullFile2.substr(i*8,i*8+8).c_str(), 0, 2);
            std::cout << fullFile2.substr(i*8,i*8+8) << std::endl; //Outputs correct binary code!
            std::cout << c << std::endl; //Outputs INCORRECT letters! First two letters correct, rest are wrong
            fullFile += c;
}
 
std::ofstream myFile;
myFile.open("Test.bmp");
myFile << fullFile;
myFile.close();

What happens right now, is that everything is received correctly in binary format.
But char c = strtol((char*)fullFile2.substr(i*8,i*8+8).c_str(), 0, 2);
is returning mostly incorrect characters.
Anything I'm doing wrong here?? Please help!

Thanks!

Researched Links:
http://stackoverflow.com/questions/1082 ... -char-in-c
http://www.dreamincode.net/forums/topic ... y-to-char/
http://www.cplusplus.com/forum/general/1764/
etc..

[CuteAlien]

I'm not sure what you mean by converting. The easiest way to send a block of data is to send it with 2 parameters. First is the size of the block and second is the content. If the content is a string then it still will be a string on arriving - you don't need any converting there.

I have no example for network right now, but some example how I do that stuff with files or with memory which is similar:
streaming.h : http://ideone.com/TMdW7N
streaming.cpp: http://ideone.com/fhuAii
Check the WriteString/ReadString stuff.

Note1: I called it streaming back then, but probably "serializing" would have been the correct name.
Note2: I wrote that code a long time ago and would write it different today (using templates for most functions).

[LunaRebirth]

I'm not sure what you mean by converting. The easiest way to send a block of data is to send it with 2 parameters. First is the size of the block and second is the content. If the content is a string then it still will be a string on arriving - you don't need any converting there.

I have no example for network right now, but some example how I do that stuff with files or with memory which is similar:
streaming.h : http://ideone.com/TMdW7N
streaming.cpp: http://ideone.com/fhuAii
Check the WriteString/ReadString stuff.

Note1: I called it streaming back then, but probably "serializing" would have been the correct name.
Note2: I wrote that code a long time ago and would write it different today (using templates for most functions).

Sorry, I mean I know how to send the information then save the file and everything.
What happens is, if I try sending a text document that has:
"Test1! Test again
Test2
Test3"
In it, everything works correctly and without any trouble.

But if it has those random sort of letters from .exe's and images, like so:
"BM� 6 (BB �"
It stops after the first space. I'm not sure why, but the letters unknown are affecting it.
So I figured if I convert to binary, THEN send it, and have the client receive and convert it to ASCII text, it'd all work out fine.

So my problem is that it's not sending those weird letters and they act as a null-terminator. So changing from binary to ASCII could fix that. Can't convert to ASCII because I don't know how to properly do it in C++.

[kklouzal]

Why don't you look into using a stringstream with the binary flag, I believe it is ios::binary, some google searches on "stringstream ios::binary" may turn up answers

[CuteAlien]

I still don't get what you mean by "converting to binary". That makes sense only when you talk about numbers. For example for the number "255" you send a single byte with FF instead of 3 characters which would be send in this case as 32 35 35 (yeah, that's hex not binary, convert to base 2 if you want binary, but my point stays the same). But for a whole text - what do you mean with binary? There is no conversion of "hello" to binary except the one you already have in memory. It will always be send as 48 45 4c 4c 4f (again using hex) as that's what the string looks like in memory.

The problem you describe rather sounds like the opposite. You try to work with strings when you have random data blocks. And you can't do that. Because as you found out already any 0 in there will terminate the string. So if you have any kind of data don't use std::string like in your fullFile. Instead work for example with std::vector<unsigned char> or any other kind of array which has the same size as your file.
And you can still send strings as well that way - because you know a string is a block of data with size strlen+1 (the +1 because the final 0 is not part of strlen).

So I think your problem is really a misunderstanding of what binary means here. You probably got the idea because some file functions have that difference between O_TEXT and O_BINARY. Basically - staying clear of O_TEXT and forgetting about it is in general the best thing you can do because it's behaviour is platform specific and I'm not even sure if it's well defined (I think on Unix it behaves like O_BINARY and on Windows it ensures line endings are saved as 13 10 even if you send strings which only end in 13, but I might be wrong). Otherwise they will look the same on disk. Although it's expected that O_TEXT only uses ascii characters (I don't even know what it would do if you send characters out of the ascii range to it, but likely nothing you want). But really - if you want any such transformation you're likely better of doing it yourself - so just avoid O_TEXT. Especially in a network context.

edit: Actually std::string can probably even handle binary data as I think it ignores 0 bytes internally. Just any other c-string function will stumble upon it. But strtol should never be needed anyway if you send stuff binary. Because sending numbers binary means you can just copy the memory content back into your numbers. Please take a look at the class I posted above - it does that a lot.

[chronologicaldot]

I think I get it now.

What you want would probably be an array of characters that represent the "bits" that make the actual characters, so you would be sending "chunks" of eight 1s so-to-speak. You can use bit flags to form the array.

Code: Select all

 
 
array<char> bitString(8);
 
for ( int i = 0; i < fullFile.length(); i++ )
{
// inlined so you can see what I'm doing (you could put this in a for-loop if you know bit operators)
bitString[0] = fullFile[0] & 0x01 > 0 ? '1' : '0';
bitString[1] = fullFile[0] & 0x02 > 0 ? '1' : '0';
bitString[2] = fullFile[0] & 0x04 > 0 ? '1' : '0';
bitString[3] = fullFile[0] & 0x08 > 0 ? '1' : '0';
bitString[4] = fullFile[0] & 0x10 > 0 ? '1' : '0';
bitString[5] = fullFile[0] & 0x20 > 0 ? '1' : '0';
bitString[6] = fullFile[0] & 0x40 > 0 ? '1' : '0';
bitString[7] = fullFile[0] & 0x80 > 0 ? '1' : '0';
 
// then save this array to your fullFile2 (or whatever you send)
}
 

Now we do the reverse

Code: Select all

 
 
char c = 0;
int b = 0;
for ( int i = 0; i < fullFile2.length(); i++ )
{
if ( b >= 8 )
{
b = 0;
std::cout << c << std::endl;
c = 0;
}
// now I've inlined it
c |= (fullFile2[i] - '0') << b; // bitshifting
b++;
}
 

Note that the character '0' in C++ is not the integer 0. Integer 0 is NULL in C++, whereas character '0' is 42 or something like that. But the easiest way to set our character '1' or '0' to integer 0 is to subtract character '0' from it. Then we can bitshift it to the slot it needs to be in.

[LunaRebirth]

I still don't get what you mean by "converting to binary". That makes sense only when you talk about numbers. For example for the number "255" you send a single byte with FF instead of 3 characters which would be send in this case as 32 35 35 (yeah, that's hex not binary, convert to base 2 if you want binary, but my point stays the same). But for a whole text - what do you mean with binary? There is no conversion of "hello" to binary except the one you already have in memory. It will always be send as 48 45 4c 4c 4f (again using hex) as that's what the string looks like in memory.

The problem you describe rather sounds like the opposite. You try to work with strings when you have random data blocks. And you can't do that. Because as you found out already any 0 in there will terminate the string. So if you have any kind of data don't use std::string like in your fullFile. Instead work for example with std::vector<unsigned char> or any other kind of array which has the same size as your file.
And you can still send strings as well that way - because you know a string is a block of data with size strlen+1 (the +1 because the final 0 is not part of strlen).

So I think your problem is really a misunderstanding of what binary means here. You probably got the idea because some file functions have that difference between O_TEXT and O_BINARY. Basically - staying clear of O_TEXT and forgetting about it is in general the best thing you can do because it's behaviour is platform specific and I'm not even sure if it's well defined (I think on Unix it behaves like O_BINARY and on Windows it ensures line endings are saved as 13 10 even if you send strings which only end in 13, but I might be wrong). Otherwise they will look the same on disk. Although it's expected that O_TEXT only uses ascii characters (I don't even know what it would do if you send characters out of the ascii range to it, but likely nothing you want). But really - if you want any such transformation you're likely better of doing it yourself - so just avoid O_TEXT. Especially in a network context.

edit: Actually std::string can probably even handle binary data as I think it ignores 0 bytes internally. Just any other c-string function will stumble upon it. But strtol should never be needed anyway if you send stuff binary. Because sending numbers binary means you can just copy the memory content back into your numbers. Please take a look at the class I posted above - it does that a lot.

That's not the reason behind it.
What's happening is that files like .exes have terminators. If you try to store a small .exe program's .txt format into an array, it'll stop after the first unknown character (LIKE a null terminator).
When I'm sending data from the server to client, it needs to be in char array format -- so obviously I want the entire file and not to stop after the first unknown character. Therefore, I convert the .exe's content in a string to binary, this way it CAN be sent. Since rather than unknown characters, it's in 010101 format. Then I send these 0's and 1's to client and the client stores it into a string, then finally converts it to text and saves it in a file.

Basically if it's not in binary, it won't send the entire thing. Tested.
If I try sending a file without unknown text letters, it works like a charm. But the contents in a program nullifies the char and stops it after that letter.

[CuteAlien]

Uhm, if you do what chronologicaldot wrote and convert each byte before sending into 8 bytes... then that makes some sense. But that would be strange network protocol (might make a little sense for sending binary data over an ascii-line). But as you write your own protocol that would be a somewhat strange (and expensive). And not sure why you would use strtol then - that does not work with binary representations in ascii to my knowlege (or does it?).

But you can send binary data directly - as I wrote above. Just send the size + the data block as char array. No conversion whatsoever necessary. You can do simple memcpy's. That's pretty much the typical way (and what I do in the code I linked).

Or if that's not it then I think you still think wrong about strings and memory. Strings are binary in memory. Always - as that's how computers save strings. Stopping at line-termintors has nothing to do with the memory layout - this is something the _functions_ working with strings do. If you want to handle a block of memory as a block of memory instead of a string then you can't work with string functions (like strlen or strtol) but you have to work with memory functions like memcpy. The memory layout stays exactly the same - but the functions accessing it work different (string-functions stopping at the first 0 while memory functions don't care).

[LunaRebirth]

Ohhh okay I understand.
So converting the string's text to binary won't help in any way, shape, or form.

But I still don't understand why I can't send images or other files.

Let me give my code now:
Server:

Code: Select all

void sendFile(char* filePath, int ID) {
    std::ifstream file;
    file.open(filePath);
    if (file.is_open()) {
        std::string myFile2;
        std::string myFile3;
        while(getline(file, myFile2)) {
            myFile3 += myFile2; myFile3 += "\n"; //Get all lines in the file
        }
        std::string myFile = "";
        for (int i = 0; i < myFile3.length(); i++) { //This doesn't matter. Same as myFile = myFile3;
            //myFile += std::bitset<8>(myFile3.c_str()[i]).to_string();
            myFile += myFile3[i];
        }
 
        std::vector<std::string> fileParts;
        if (myFile.length() > 3000000) { //Split file in parts if the size is > than 3gb
            for (int x = 0; x < myFile.length()/3000000; x++) {
                fileParts.push_back(myFile.substr(x*3000000,(x+1)*3000000));
                myFile = myFile.substr((x+1)*3000000,myFile.length());
            }
        } else { //Otherwise if < 3gb, have all in one 
            fileParts.push_back(myFile);
        }
        int xyz = 0;
        while (xyz < fileParts.size()) { //Send file parts
            char* sendThisFile;
            sendThisFile = (char*)fileParts[xyz].c_str();
 
            memcpy(SD3.fileData, sendThisFile, strlen(sendThisFile)+1 );
 
            SD3.done = false;
            sendAll(sockvec[ID-1],(char*)&SD3,sizeof(SD3),0); //sendAll makes sure to send every piece of info, even if the file is too large
            xyz++;
        }
        memcpy(SD3.fileData,"",3000000);
        SD3.done = true;
        send(sockvec[ID-1],(char*)&SD3,sizeof(SD3),0); //Send that the file is complete and the client can stop saving the file
    }
}

Client:

Code: Select all

void receiveFile(SOCKET mySock) {
    std::string fullFile2 = "";
    while (!RD3.done) {
        recvAll(mySock,reinterpret_cast<char *>(&RD3),sizeof(RD3),0); //Receive everything from the received file
        fullFile2 += RD3.fileData; //fullFile2 += received text from file being sent
    }
    std::string fullFile = "";
    fullFile = fullFile2; //Got rid of some stuff. That's why this is here
 
    std::ofstream myFile;
    myFile.open("Test.bmp");
    myFile << fullFile;
    myFile.close();
}

With the above two codes, I can send a text file with the save text:
"Hello Test
Wow
Test again
Okay
Rofllll cool and stuff"

The server will send it, and the client receives:
"Hello Test
Wow
Test again
Okay
Roflll cool and stuff"

BUT
if the text file says:
"‰PNG
IHDR ;Ö•J"
The server sends it, and the client receives:
"‰PNG"
Missing everything else

I don't see any reason behind it. That's my problem.
So I had originally thought that changing ""‰PNG
IHDR ;Ö•J" to binary as text would work. But if you think it won't, I shouldn't try it.

[kklouzal]

If you didn't mind using another library, there is a serialization library on GitHub called Cereal, it allows you to serialize any type, so you could serialize an entire class instance like a specific ITexture or ISceneNode if you wanted to. The client would receive and deserialize it into the original state it was in when it was serialized.

[mongoose7]

Why are you using getline()? While all the fiddling around? All that you need is

Read 64k of file into buffer
Send buffer
Repeat.

Send chunks as large as you want. The receiver controls how much to read with recv(.. size ..).

[CuteAlien]

You still mix string and memory functions.

Do not use getline to read, that works with strings, use istream::read instead which works with memory. And yeah, it's slightly more complex - because it means you have to allocate the size into which you read it first (but you can just use any block-size and send it in blocks or you use the file-size and transfer the whole size). Adding several files into one block means you have allocate enough memory for all of them first and then always read for each file into "address of your allocated memory block" + "amount of bytes already filled in that block".

Do not work with functions like substr - as the name says it's again a string function, not a memory function. For memory you work with pointers (and do the calculation yourself which is pretty simple as you can just use char* pointers so you can use normal additions).

Do not work with strlen - again as the name says that works with strings. You have to know the sizes yourself (from the file-sizes and your block-sizes) and do not need it. On the receiver side you can know the size by sending that as first parameter. So any binary network protocol usually looks something like: First send buffer-size (as 32 or 64 bit value, just be sure to use the same size on sender on receiver). Then send a block of data with that size. And on receiver side your first read that size-parameter, then allocate a block large enough to receive the rest and then read the rest. More complex binary protocols might send more information describing the data in which case you don't just have a size-parameter to send but maybe a larger header first. It is for example always a good idea to send a version number as well, as your protocol might change one day. Also some protocols might have single header to send first while others might have a header for each block they send. The important part is that the header needs to have a fixed size so you know how much to read on the client side to get it's information (or to make it more complex - in some protocols headers send their own size as well and send that information in a fixed position - so the header has it's own header more or less).

Do not use the stream operators << and >> as those work with formatted strings. You have to use the read and write functions instead which work with memory blocks.

[LunaRebirth]

If you didn't mind using another library, there is a serialization library on GitHub called Cereal, it allows you to serialize any type, so you could serialize an entire class instance like a specific ITexture or ISceneNode if you wanted to. The client would receive and deserialize it into the original state it was in when it was serialized.

That sounds really cool actually. So I could take an ITexture from the client, send its' data to the server (In serialization), have the server receive and then send it to everyone, and then each client can un-serialize it to load that ITexture that the first client had?
That would be VERY useful. Does this require a .DLL? If so, I might have to turn to something else that may be similar.
Thanks for the awesome idea

Edit: Just found it's a Boost thing. I will not download the Boost library as I've have bad experiences with it in the past, and it's slow. Great idea to look at, though, thanks!

----

Edit:I tried using read rather than getting lines, then I outputted the read file into another file (NOT sent to client. Done on server only) and it's still getting the same things. Not every character it should be getting from files (But yes, a-z letters works fine still). I'm using write rather than outputting with >> or << as suggested, and still nothing is grabbing each specific character. <-- got it working

[LunaRebirth]

Sorry, put some edits above. My bad

[LunaRebirth]

Okay so I've restarted from scratch, and didn't do what I wasn't supposed to do.

Code: Select all

void sendFile(char* filePath, int ID) {
    char * buffer;
    long size;
    std::ifstream file (filePath, std::ios::in|std::ios::binary|std::ios::ate);
    size = file.tellg();
    file.seekg (0, std::ios::beg);
    buffer = new char [size];
    file.read (buffer, size);
    file.close();
    send(sockvec[ID-1],buffer,sizeof(buffer),0);
}
 
void receiveFile(SOCKET mySock) {
    char* buffer;
    recv(mySock,buffer,sizeof(buffer),0);
    std::string findSize = buffer; int len = findSize.length();
 
    std::ofstream myFile ("Test.png", std::ios::binary);
    myFile.write(buffer,len);
    myFile.close();
}

Again, received normal letters. But sending a file (A .png in this case) stops it after that first %PNG line.
I seriously think the characters that are in the .png are acting as null-terminators no matter what.
I don't understand why this is happening.