TCP Sockets - File and Data Transfer

 

Since Firmware version 003 (AA003A), µPanel implements functions for the direct use of TCP and UDP sockets, providing an easy and effective way to perform both low-level and high-level data transfer or to arrange networks of µPanel devices. In addition, the Socket functions can interact with the file system permitting the direct transfer of files over HTTP protocol (TCP), both for downloading and uploading data. The main features of TCP socket functions are:

  • Up to 2 TCP Sockets (operating either as client or as server)
  • Low level access to stream (either in binary or ascii mode)
  • Connect-and-Transfer or Listen-and-Transfer functions in a single command
  • HTTP support over TCP sockets for data transfer (both GET and POST methods supported)
  • Interaction with the uPanel File System for file transfer with HTTP protocol over TCP sockets
  • Both file upload and download, both with GET and POST methods
  • File Transfer protectable with custom password

The syntax of socket commands has been designed to minimise the micro-controller intervention during the implementation of simple data transfer. For instance a TCP server can be started with a preloaded first data packet to be transferred as soon as the client connects, or can be programmed to transfer a whole file with the remote device (both upload and download). TCP commands start with the prefix $TCP.

TCP Commands

TCP commands permit either to establish a connection with a server or to listen to a given TCP port (server mode). In both the cases, when the connection with the remote device is established, an automatic action can be executed, such as:

  • transmission of a preloaded message
  • transmission of a predefined file
  • execution of a simple HTTP daemon (server mode)


TCP Socket Initialisation 

The general syntax for initialising a TCP socket follows: 

TCP Command syntax

$TCP N_Socket :Connection_Flags :Remote_Address :Port,[Parameter] [:File_Name[:Password]] [:Message]


Where

  • [ ] brackets denote optional arguments
  • N_Socket is a digit in the range {0,1} indicating the number of the socket to use
  • Connection_Flags is a combination of one or more of the following characters, which specify the action to be performed with the socket:

    • {C, L, D, X, Q}: C for connect (initialise the client mode); L for listen (initialise the server mode); D for disconnect socket; X for destroy socket (disconnect, delete and free resources); Q for query socket status.
    • {A, B}: A for transferring data in ascii mode, B for binary mode. The A flag can be followed by a number indicating the maximum allowed line length. If the line exceeds the specified length the line will be truncated. When the maximum line is specified and the line is not truncated the sequence of characters \ and n is appended to the line. For example, the flag A100 will set the maximum line length to 100 characters. 
    • F to enable the File transfer mode. This flag can be followed by one or more lower-case characters that represent the file-mode (e.g. Fac3 means File Mode in "Append,Circular,3 kB"). Please refer to the File section to obtain the full list of file modes.
    • H to enable the HTTP protocol for data transfer (both raw data and files)
    • {D, U}: D for download mode (default in C client mode), U for upload mode (default in L server mode)
    • K to enable the keep-alive connection mode
    • Q to enable auto-disconnect (quit) mode (i.e. the socket will be automatically disconnected after the first data packet is correctly sent)

    Please note that the flag order in not important, with the exception of {C, L, D, X, Q} flags that must be the first ones in the sequence and of {A, B} that must be the second ones in the sequence. Please also note that the flags in curly brackets are mutual exclusive (i.e. you can not specify C and L simultaneously)

  • Remote_Address is either the domain name or the IP address of the remote device 
  • Port represents the destination port in client mode (connect) or the local port to listen to in server mode
  • Parameter represents the local port in client mode (connect) or the timeout in seconds (in server mode)
  • File_Name is an optional parameter that specifies the file name in "File Mode". The File name can be followed by a password to protect the transfer. 
  • Message represents the data to be sent to the remote device as soon as the connection is established

Please Note: File_Name and Message can not be both specified, but they are mutually exclusive.


Command results

The possibile responses to the Query command (Q) are:

$ERR-TCPn:Query   The socket could not be queried
$TCPn:IS-FREE    Not created
$TCPn:IS-DISCONNECTED
$TCPn:IS-CONNECTING
$TCPn:IS-CONNECTED
$TCPn:IS-ON-ERROR
$TCPn:IS-SENDING

Please refer to the end of this section for the list of the other messages

Basic Examples (a complete list of examples is provided at the end of this section)

To connect the TCP socket with ID=1 to the Google Web server on port 80 with ascii transfer mode

$TCP1:CA:www.google.com:80

$OK-TCP1-Connecting
$TCP1:CONNECTED

 


TCP Data Send (Write)

The syntax of the command for sending some data through an already initialised and connected socket is as follow:

Command syntax

$TCP N_Socket: W: DATA


Where:

  • N_Socket is a digit in the range {0,1} indicating the number of the socket to use
  • DATA is a string containing the data to send. Data can contain all characters with the exception of the following 4 bytes, which have to be escaped:
    • return carriage byte 13 (escape sequence \r)
    • new line byte 10 (escape sequence \n)
    • null byte 0 (escape sequence \0)
    • back slash byte 92 (escape sequence \\)

Command results

$OK-TCPn-Sending immediately after command reception to acknowledge the command execution (n represents the socket number)

$TCPn:ERR-BUSY immediately after command reception to indicate that the operation could not be executed because the socket is still busy with another operation (n represents the socket number) 

$TCPn:ERR-SENDING immediately after command reception to indicate that the socket is not ready for sending (n represents the socket number)

$TCPn:SENT:x  when the data has been transferred; n represents the socket number and x represents the number of the sent packet, counted from the socket initialisation

  

Example

To write the string "Hello World" terminated with a new line (\n) to the TCP socket with ID 1: 

$TCP1:W:Hello World\n

$OK-TCP1-Sending
$TCP1:SENT:1

 


TCP Data Receive (Read)

When some data is received from a connected socket, the following two messages are sent to the controller, containing Header and Data respectively:

Header Message format

$TCP N_Socket: #NNNN,IP :Port


Where:

  • N_Socket is number of the socket that received data
  • NNN  is the number of received bytes formatted on 3 digit (e.g. 5 bytes will be 005)
  • IP is the IP of the remote device that sent data (formatted as XXX.XXX.XXX.XXX)
  • PORT is the TCP port of the remote sender

Data message format

$TCP N_Socket: >X: [#NNNN:] [ASCII_DATA]

 

Where:

  • N_Socket is number of the socket that received data
  • X can be the character A (for ascii mode), and B (for binary mode). This character specifies how the received data will be sent to the micro-controller (the same mode used during the socket initialisation)
  • NNNN is a four-digit number that specifies the number of data bytes that follow the message (this field is sent only for Binary mode)
  • ASCII_DATA is the text line received in ASCII mode. If the socket is set in Binary mode, this field is empty and the binary message follows after a new line byte (\n).

Example

1) If the message "Hello\r\n" is received from the socket #0 configured in ASCII mode, the following output will appear:

$TCP0:#0007,192.168.1.103:64292
$TCP0:>A:HELLO
$TCP0:DISCONNECTED

where 192.168.1.103 and 64292 are the sender's IP and Port respectively.

2) If the message "Hello\r\n" is received from the socket #0 configured in Binary mode, the following output will appear:

$TCP0:#0007,192.168.1.103:64292
$TCP0:>B:#0007:
HELLO

$TCP0:DISCONNECTED

Please note that the binary message starts on a new line and that also the message terminators "\r\n" are sent through the serial line. 

 


TCP Sockets - Examples

 

Example 1

Download the file logo.png from the www.miupanel.com server and save it with the name picture.png, using the TCP socket ID #0:

Command

$TCP0:CBFH:www.miupanel.com:80:picture.png:http://www.miupanel.com/logo.png

Response

$OK-TCP0-Connecting
$TCP0:CONNECTED
$TCP0:SENT:1
$TCP0:FILE-LEN:14125
$TCP0:FILE-SAVED
$TCP0:DISCONNECTED

Explanation: use TCP #0 to connect (C) in binary (B) mode for File (F) transfer over HTTP protocol (H) with the server www.miupanel.com on port 80. Save the received data into the file with name picture.png. The URL of the remote data is http://www.miupanel.com/logo.png 

Please Note: the FILE-LEN value can be -1 if the server does not specify the content-length header. 

 

Example 2

Download and send to the micro-controller in ASCII mode the remote file robots.txt from the TCP port 80 of server  www.google.com, using the TCP socket with ID #0:

Command

$TCP0:CAH:www.google.com:80:/robots.txt

Response

$OK-TCP0-Connecting
$TCP0:CONNECTED
$TCP0:SENT:1
$TCP0:FILE-LEN:-1
$TCP0:#1440,149.003.177.026:00080
$TCP0:>A:User-Agent: *
$TCP0:>A:Disallow: /search
$TCP0:>A:Allow: /search/about
...
$TCP0:>A:D
$TCP0:#1440,149.003.177.026:00080
$TCP0:>A:isallow: /jsky/?
...
$TCP0:DISCONNECTED

Explanation: use TCP #0 to connect (C) and display in Ascii (A) mode the remote resource robots.txt available with the HTTP protocol (H) on the remote server www.google.com at port 80

Note 1: the FILE-LEN is -1 since the server does not specify the content-length header. 
Note 2: since the file is received in chunks (of 1440 bytes) the last line of the chunk can appear truncated. To avoid this problem define the maximum line length so that the line terminators will be marked as well (see the following example).

 

Example 3

Download and send to the micro-controller in ASCII mode with line terminators the remote file robots.txt from the TCP port 80 of server  www.google.com, using the TCP socket with ID #0:

Command

$TCP0:CA9999H:www.google.com:80:/robots.txt

Response

$OK-TCP0-Connecting
$TCP0:CONNECTED
$TCP0:SENT:1
$TCP0:FILE-LEN:-1
$TCP0:#1440,149.003.177.026:00080
$TCP0:>A:User-Agent: *\n
$TCP0:>A:Disallow: /search\n
$TCP0:>A:Allow: /search/about\n
...
$TCP0:>A:D
$TCP0:#1440,149.003.177.026:00080
$TCP0:>A:isallow: /jsky/?\n
...
$TCP0:DISCONNECTED

Explanation: use TCP #0 to connect (C) and display in Ascii (A) mode (using messages with no more than 9999 characters) the remote resource robots.txt available with the HTTP protocol (H) on the remote server www.google.com at port 80

Note 1: the FILE-LEN is -1 since the server does not specify the content-length header. 
Note 2: truncated lines at the end of the received chunks can be detected since they do not end with characters \ and n

 

Example 4

Save all the data sent by the remote Web server www.google.com into a local file named reply.txt, including the HTTP headers, sent as a reply to the GET / command, using the TCP socket with ID #0:

Command

$TCP0:CAF:www.google.com:80:reply.txt:GET / HTTP1.0\r\n\r\n

Response

$OK-TCP0-Connecting
$TCP0:CONNECTED
$TCP0:SENT:1
$TCP0:FILE-SAVED
$TCP0:DISCONNECTED

Explanation: use TCP #0 to connect (C) in Ascii (A) mode the server www.google.com on port 80 saving the whole reply to the file (F) name reply.txt. Since we want to save the HTTP headers too, we don't enable the HTTP parser (H) so that the headers will not be removed. However, we have to manually set the HTTP message to request the remote resource, so we set the automatic message to be sent to the server after connection to: GET / HTTP/1.0

 

Example 5

Connect to the remote server www.google.com and send to the micro-controller in binary mode the response to the request GET /, using the TCP socket with ID #0:

Command

$TCP0:CB:www.google.com:80:GET / HTTP1.0\r\n\r\n

Response

$OK-TCP0-Connecting
$TCP0:CONNECTED
$TCP0:SENT:1
$TCP0:#0488,149.003.177.050:00080
$TCP0:>B:#0488:
HTTP/1.0 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8

...
$TCP0:DISCONNECTED

Explanation: use TCP #0 to connect (C) in Ascii (B) mode to the server www.google.com on port 80 and to request GET /, sending the whole reply to the micro-controller. Since we want to receive the HTTP headers too, we don't enable the HTTP parser (H), so we have to manually send the request of the remote resource: we set the automatic first message to GET / HTTP/1.0

 

Example 6

Send to a custom server, with GET method, temperature and humidity, reporting to the micro-controller in ascii mode the server reply. In this example we suppose that a remote web server implements an asp/php page capable to read the parameters included in the requested URL. We suppose that the server will reply with a simple text line as well.

Command

$TCP0:CAH:www.miupanel.com:80:http://www.miupanel.com/measure.asp?t=23&h=53

Response

$OK-TCP0-Connecting
$TCP0:CONNECTED
$TCP0:SENT:1
$TCP0:FILE-LEN:15
$TCP0:#239,031.011.032.035:00080
$TCP0:>A:OK (T=23, H=53)
$TCP0:DISCONNECTED

File measure.asp on miupanel.com server

OK<% response.write " (T="+Request.QueryString("t")+", H="+Request.QueryString("h")+")" %>

Explanation: use TCP #0 to connect (C) and display in Ascii (A) mode the reply of the server www.myserver.com to the HTTP (H) requested URL measure.asp. The parameters (temperature 23 °C, and humidity 53%) are included in the URL.

 

 

Example 7

Install a simplified WEB server to accept incoming HTTP requests (on TCP port 5555) and transfer (upload) local files (without restrictions on the file name and without password). The file should appear in the browser window.

Command

$TCP0:LAFH::5555

Response

$OK-TCP0-Listen

Browser request (URL) to request the logo of example 1

http://192.168.1.100:5555/picture.png

Response after browser request

$TCP0:CONNECTED
$TCP0:FILE-SENT
$TCP0:DISCONNECTED

Explanation: use TCP #0 to listen (L) and send in Ascii (A) mode a local file (F) using HTTP (H) protocol on TCP port 5555. Please note that the file is always transferred in binary mode to the browser, but the ascii mode (A) has the effect to display the transferred file into the browser window. The binary (B) mode would trigger the download dialog instead.

 

Example 8

Install a simplified WEB server to accept incoming HTTP requests (on TCP port 5555) for transferring (upload) the local file picture.png with password secretword. The file should be downloaded on the remote computer and not displayed in the browser window.

Command

$TCP0:LBFH::5555:picture.png:secretword

Response

$OK-TCP0-Listen

Browser request (URL) to request the logo of example 1

http://192.168.1.100:5555/picture.png?secretword

Response after browser request

$TCP0:CONNECTED
$TCP0:FILE-SENT
$TCP0:DISCONNECTED

Explanation: use TCP #0 to listen (L) and send in binary (B) mode the local file (F) picture.png using HTTP (H) protocol on TCP port 5555 protecting the transfer with the password secretword. Please note that using the binary mode, the browser will trigger the download dialog instead of displaying the file in the browser's window.

 

Example 9

Install a simplified WEB server to accept incoming HTTP requests (on TCP port 5555) for transferring to the module's flash memory (download!!) the file abc.txt with password secretword. The file is transferred from the remote computer to the module's disk.

Command

$TCP0:LBFHD::5555:abc.txt:secretword

Response

$OK-TCP0-Listen

The browser can open a local html page to upload the file (source code follows)

http://127.0.0.1/upload.html

Response after browser request

$TCP0:CONNECTED
$TCP0:FILE-SAVED
$TCP0:DISCONNECTED

HTML File upload.html that can be used by a computer browser for selecting and transferring a file to the module's disk

<html>
  <body>
<form id='uPost' method="post" enctype="multipart/form-data" action="http://192.168.1.100:5555/?file=abc.txt&secretword">
<input type="file" name="ufile">
  </form>
<button onclick='document.getElementById("uPost").submit();'>SEND</button>   </body>
</html>

Explanation: use TCP #0 to listen (L) and in binary (B) mode for downloading (D) a file (F) using the HTTP protocol (H) a file with name abc.txt on TCP port 5555 protecting the transfer with the password secretword. Please note that on the remote computer a browser can use the upload.html page reported above to perform the transfer.

 

Example 10

Install a simplified WEB server to display into a browser's window the user defined message "Hello World" (or any other content). The system will accept incoming HTTP requests (on TCP port 5555).

Command

$TCP0:LAHQ::5555:HTTP/1.0 OK\r\n\r\nHELLO WORD

Response

$OK-TCP0-Listen

Browser request (URL)

http://192.168.1.100:5555/

Response after browser request

$TCP0:CONNECTED
$TCP0:>H:GET / HTTP/1.1
$TCP0:SENT:1
$TCP0:DISCONNECTED

Content of the browser window after the request


HELLO WORLD

Explanation: use TCP #0 to listen (L) in ascii mode (A) on the TCP port 5555, managing the HTTP protocol (H) of incoming requests, replying to them with the predefined message and quitting the communication immediately after the reply (Q). Please note, that in this specific case, the HTTP (H) mode has the effect to display only the first line of the HTTP request from the browser, while all the other lines are hidden (the first line is sent as: $TCP0:>H:..... where >H means received header)

 

Example 11

Install a simplified WEB server to display into a browser's window the message the micro-controller decides depending on the required content. In this example, the a browser will request the content results (on TCP port 5555) and the micro-controller will reply with the message Result 1351

Command

$TCP0:LAHQ::5555

Response

$OK-TCP0-Listen

Browser request (URL)

http://192.168.1.100:5555/results

Response after browser request

$TCP0:CONNECTED
$TCP0:>H:GET /results HTTP/1.1

The microcontroller processes the request and execute the following command


$TCP0:W:Result 1351

Response after write request

$OK-TCP0:Sending
$TCP0:SENT:1
$TCP0:DISCONNECTED

Content of the browser window after the request


Result 1351

Explanation: use TCP #0 to listen (L) in ascii mode (A) on the TCP port 5555, managing the HTTP protocol (H) of incoming requests, quitting the communication immediately after the reply (Q). Please note, however, that no message is defined so the reply must be managed by the micro-controller. The HTTP (H) mode has the effect to display only the first line of the HTTP request from the browser, which is useful for the micro-controller to read the requested data. Please also note that the micro-controller reply does not include the HTTP response. A full compliant reply would require to send all the HTTP headers, for instance:

$TCP0:W:HTTP/1.0 200 OK\r\n\r\nResult 1351

The communication is automatically closed after the reply (Q), however, consider that, the socket is not deleted, so it is ready to receive other requests.


TCP Socket Messages and Errors

Different messages can appear after the execution of TCP commands. The full list of message follows (where n is the socket ID number)

$TCPn:ERR-NET (err) Network error (number err)

$TCPn:FILE-SAVED The received file has been saved

$TCPn:DISCONNECTED The socket has been disconnected

$TCPn:FILE-SENT The local file has been sent

$TCPn:SENT:x The packet x has been sent

$TCPn:CONNECTED The socket has been connected

$TCPn:FILE-ERR-REPLY Malformed or unknown HTTP server response

$TCPn:FILE-ERR-RESP (x) The remote server responded with HTTP error x

$TCPn:FILE-LEN:x The server responded with Content-Length of x bytes

$TCPn:ERR-DNS The remote server name could not be resolved

$TCPn:ERR-CONNECT (x) Failed to connect to the remote server (error x)

$TCPn:ERR-SOCKET-BUSY The socket is already busy with the previous operation