A word of warning, this post will be kind of long. I will try to cover the binary communication over serial port basics in 2 languages (C & perl), but I hope every one will bare with me until the end.
I'm almost ready to go into some of my "real" projects. In order to do that all that needs to be done is to talk a little about SLIP protocol. SLIP protocol is used to synchronize data between receiver and transmitter on a serial line (not only RS232, ethernet as well and many other mediums). SLIP is one of the most commonly used option, the other one is Modbus RTU, which describes some timing constraints and requirements during the data transfer. I haven't tried Modbus RTU yet, since SLIP is doing pretty good job for me.
First of all why do we need to synchronize the data at all and what does it mean in details in the first place ? The reason for that is simple (maybe not so obvious in the first place). Consider the following situations:
We want to send 8 bytes of binary data from A to B over Serial line:
- B is not yet ready to listen for data
- A already send the first byte
- B become ready to receive data
- A is sending more data
- B received only 7 bytes out of 8. 1 byte has been LOST
What comes to one's mind straight away when analyzing this example, let's force A to wait for some time so B for sure will become ready. Let's think about it:
A sends 8 bytes of data in two packs to B:
- A waits a second (waiting for B to become ready to receive)
- B becomes ready to listen
- A sends first pack (4 bytes of data) to B
- B receives 4 bytes and process it
- A sends another pack of 4 bytes
- B has not finished processing of previous package and lost some of the data from the second pack
As time decreases and transmission speeds are getting higher this leads to more and more problems. As long as we can get away with sending text data between peers without any kind of synchronization. It's very difficult to do that and to have a stable reliable binary data transfer without any.
SLIP is a very simple, almost primitive solution, which solve that kind of problem. I'm not going to duplicate the description here, please refer to the rfc1055 for more details regarding it, briefly SLIP defines two special characters: an END byte and ESC byte. Receiver knows about the end of the data stream by listening for an END byte, when the data stream contains the END or ESC byte in it, then transmitter replaces it with the ESC byte and the ESC_END or ESC_ESC byte - those are replaced to the valid data by the receiving side. That's all - as I mentioned it's very simple. SLIP defines the exact values for END, ESC, ESC_ESC & ESC_END but those can have any values - it doesn't really matter:
/// 192 -> 0xc0
#define SLIP_END 0300
/// 219 -> 0xdb
#define SLIP_ESC 0333
/// 220 -> 0xdc
#define SLIP_ESC_END 0334
/// 221 -> 0xdd
#define SLIP_ESC_ESC 0335
For details of implementation (just as the protocol itself it is very simple), have a look into libpca (src/slip.c). One more thing must be mentioned, SLIP implementation from libpca can be used over many different links. One must only define the IO routines which will be used to retrieve data. Let's have a look on the fragment of slip.h header from the library:
By default libpca use the serial port IO routines, but this can be overridden if the user will provide it's own definition for symbols SLIP_CHAR_RECV/SLIP_CHAR_SEND.
Let's consider how we can use it, for that I'll define a simple software timer and a data model for our binary informations which will be exchanged on the serial line in order to modify the timer parameters. For the sake of simplicity we'll use LED on one of the port which will be toggled on every timer overflow. The "software timer" is defined as follows:
It's pretty simple we have the cnt variable itself and the maximum (max) value to which the timer can count, before being reseted to zero. Everything will get more obvious in the code using it. Let's define the data-model for the messages:
I use "packed" attribute here. It is needed (especially for the host machine) to prevent the compiler from aligning the structure to the machine architecture size. It doesn't have any influence on Arduino since it's an 8-bit machine, but since the header is shared between Arduino and the host machine when trying to compile the code on host machine without "packed" attribute applied to the structures probably size of struct timer_data would be 8 instead of 5 (alignment to the nearest 4-byte boundary, since most x86 are 32-bit machines). Of course the crc16 field is not mandatory in our communication, but just for demonstration purposes it's nice to know that if the data interchanged is really crucial it may be verified using CRC (I use crc16 but it's possible to use any other CRC32, CRC8, by default I decided to implement crc16 only in libpca). The host application will send new timer_data structure with new setting to reprogram the timer and in result the blinking speed will change. One thing must be mentioned here, Arduino will reset itself after establishing the serial connection, it's because of the board design itself - one should not worry about that. This behavior can be disabled by disabling the jumper on Arduino UNO rev 3. board (there are plenty of solutions available in the web for different boards - I don't want to go into details here).
Let's have a look on the Arduino program first, and do a little walk-through:
It looks pretty complicated at first glance but that's only the appearance. First of all the usual stuff, I configure Timer 0 to CTC mode and unmask it's interrupt (one must remember to disable the Timer's 0 interrupt implementation in libpca - #define TDELAY_IMPLEMENT_T0_INT 0, since we're defining our own custom handler). In the while loop, Arduino is waiting for a complete SLIP frame, any junk frames (which size is smaller than the struct timer_data) are discarded, the last step is to verify the data checksum of the frame and to send a SLIP frame containing struct response with NACK status in case the CRC doesn't match. slip_verify_crc16() takes the buffer, it's length and third argument which defines a position in the buffer of the CRC checksum - in our case the CRC is placed at the very beginning, so the position equals zero. If the CRC verification is successful then at line 104 we can be sure that the data is genuine and not corrupted in communication. Next step is to reprogram the timer and finally send the SLIP frame with struct response with ACK status. This frame has a checksum calculated as well, and we will be able to verify it on the other side. The code is more or less straight forward.
The conclusion is that the LED attached to the Arduino will blink with different speed, and the "timing" details are programmed remotely through the serial line and the binary messages exchanged using SLIP protocol.
We can now focus on discussing the details of the host application. To be honest writing an IO application to communicate through the serial port in C for the host machine is a little bit tedious and quite time consuming, that's why it is far more convenient and simply quicker to create such tool in a scripting language of choice (Perl, Python, Lua) since most of the dirty work has been already done under the hood. For the purpose of comparison and education I'll present a script in Perl which will talk to our Arduino as well as in C.
Let's start with Perl since it's far more simpler and I believe it will be easier to understand.
The biggest advantage of Perl over any other scripting language is CPAN and the amount of modules available. I'll use some of the already available building blocks in order to make my life a little bit easier. First of all we need to somehow use the Serial Port, the second thing is the CRC checksum calculation. We will need the following modules (they are available in Arch Linux package manager - which is my distro of choice, the package names are listed as well - those can be named similarly in your distribution):
We've got the tools, we can do the work finally. Let's have a look on the following code:
For those who are not familiar with Perl it can be hard to get used to it. I'll try to be very verbatim. Starting from the top, the first interesting line is the declaration of hash %g_dev which holds the details of the Arduino's Serial Port. Again, for those unfamiliar with Perl hashes are like associative arrays, next I define the SLIP special characters as constants. Next are two important subroutines which realize the communication. First of all, Perl does not define something like a structure and does not provide a direct access to the memory underlying the variables - just like in C, how can we talk then about binary data interchange in a format described previously by the C structures ? It happens in a little bit different way. Perl provides two functions pack/unpack to pack the variables into a string accordingly to the given template and to unpack the string to variables accordingly to the given template (please read the details in the perl documentation, perldoc -f pack, perldoc -f unpack, perldoc perlpacktut). In order to make it more reasonable let's have a look on the struct timer_data once again. We need to send a byte stream constructed the following way:
That means that if we would write:
the pack function will assemble our variables into a string which can be then directly send through the serial port. We need something more here, the SLIP_END characters, so the complete call could look like this:
my $data = pack ("CSC*", (SLIP_END, $crc, $prescaler, $ocr, $max_st, SLIP_END));
That's exactly what is happening in the slip_send subroutine. First the data passed to it is scanned for any SLIP special characters in order to escape them. Once that is done, the original data (unescaped) is passed to the crc16 function to calculate the checksum. A word about line 32. First a byte string is constructed containing two zeros in the beginning - which are a placeholder for the CRC, and the rest of the data. The placeholding zeros are replaced with actual CRC once it is calculated. The receiver will calculate the CRC the same way in order to verify the data. It will copy the received CRC from the frame to the temporary variable and zero it's bytes in the frame - now the frame is in the same state as on the transmitter side before the transmitter calculated the CRC. If that operation wouldn't have place of course the checksums would be different.
The slip_recv() subroutine is exactly implemented the same way as in libpca. One thing of explanation here, slip_recv() has been implemented in more or less universal fashion - it takes a references to a getchar routine (line 38) which can be implemented over any kind of link. Next the command line input is parsed into global variables (lines 68-74), the serial port is initialized (lines 76-84) and we wait for 4 seconds. Now why is that ? As mentioned previously, after establishing a serial connection RS232 control lines will be toggled which in result will reset an Arduino, so we need to wait some time in order for Arduino to become ready and operational. By default (as it is visible in the code), the Arduino's Timer0 and Software timer will be initialized so the overflow happens with a frequency of ~ 1 Hz. The LED will blink with that frequency until the SLIP frame will be received with new configuration values for the timer. In lines 100-101 the SLIP frame containing our binary data is send over the serial line next the script waits for the SLIP frame from Arduino containing the response. The struct response is very simple, it contains only the CRC16 (2 bytes) and the status. It is unpacked after reception (line 107):
# unpack the serialized data to variables
my ($crc, $status) = unpack("SC", $data);
# calculate the checksum of the response for verification
my $crc_calcd = crc16(pack("SC", (0,$status)));
 |
| This connection must be disabled in order to disable auto-reset on serial connect. |
Arduino Side
Let's have a look on the Arduino program first, and do a little walk-through:
It looks pretty complicated at first glance but that's only the appearance. First of all the usual stuff, I configure Timer 0 to CTC mode and unmask it's interrupt (one must remember to disable the Timer's 0 interrupt implementation in libpca - #define TDELAY_IMPLEMENT_T0_INT 0, since we're defining our own custom handler). In the while loop, Arduino is waiting for a complete SLIP frame, any junk frames (which size is smaller than the struct timer_data) are discarded, the last step is to verify the data checksum of the frame and to send a SLIP frame containing struct response with NACK status in case the CRC doesn't match. slip_verify_crc16() takes the buffer, it's length and third argument which defines a position in the buffer of the CRC checksum - in our case the CRC is placed at the very beginning, so the position equals zero. If the CRC verification is successful then at line 104 we can be sure that the data is genuine and not corrupted in communication. Next step is to reprogram the timer and finally send the SLIP frame with struct response with ACK status. This frame has a checksum calculated as well, and we will be able to verify it on the other side. The code is more or less straight forward.
The conclusion is that the LED attached to the Arduino will blink with different speed, and the "timing" details are programmed remotely through the serial line and the binary messages exchanged using SLIP protocol.
Host side
We can now focus on discussing the details of the host application. To be honest writing an IO application to communicate through the serial port in C for the host machine is a little bit tedious and quite time consuming, that's why it is far more convenient and simply quicker to create such tool in a scripting language of choice (Perl, Python, Lua) since most of the dirty work has been already done under the hood. For the purpose of comparison and education I'll present a script in Perl which will talk to our Arduino as well as in C.
Let's start with Perl since it's far more simpler and I believe it will be easier to understand.
Host Slip Communication in Perl
The biggest advantage of Perl over any other scripting language is CPAN and the amount of modules available. I'll use some of the already available building blocks in order to make my life a little bit easier. First of all we need to somehow use the Serial Port, the second thing is the CRC checksum calculation. We will need the following modules (they are available in Arch Linux package manager - which is my distro of choice, the package names are listed as well - those can be named similarly in your distribution):
- Device::SerialPort (perl-device-serialport)
- Digest::CRC (perl-digest-crc)
We've got the tools, we can do the work finally. Let's have a look on the following code:
For those who are not familiar with Perl it can be hard to get used to it. I'll try to be very verbatim. Starting from the top, the first interesting line is the declaration of hash %g_dev which holds the details of the Arduino's Serial Port. Again, for those unfamiliar with Perl hashes are like associative arrays, next I define the SLIP special characters as constants. Next are two important subroutines which realize the communication. First of all, Perl does not define something like a structure and does not provide a direct access to the memory underlying the variables - just like in C, how can we talk then about binary data interchange in a format described previously by the C structures ? It happens in a little bit different way. Perl provides two functions pack/unpack to pack the variables into a string accordingly to the given template and to unpack the string to variables accordingly to the given template (please read the details in the perl documentation, perldoc -f pack, perldoc -f unpack, perldoc perlpacktut). In order to make it more reasonable let's have a look on the struct timer_data once again. We need to send a byte stream constructed the following way:
- crc16 - unsigned short 16 bit variable - pack template: S
- prescaler - unsigned char 8 bit variable - pack template C
- ocr - unsigned char 8 bit variable - pack template C
- max_st - unsigned char 8 bit variable - pack template C
That means that if we would write:
my $data = pack ("SCCC", $crc, $prescaler, $ocr, $max_st);
or shorter:
my $data = pack ("SC*", $crc, $prescaler, $ocr, $max_st);
the pack function will assemble our variables into a string which can be then directly send through the serial port. We need something more here, the SLIP_END characters, so the complete call could look like this:
my $data = pack ("CSC*", (SLIP_END, $crc, $prescaler, $ocr, $max_st, SLIP_END));
That's exactly what is happening in the slip_send subroutine. First the data passed to it is scanned for any SLIP special characters in order to escape them. Once that is done, the original data (unescaped) is passed to the crc16 function to calculate the checksum. A word about line 32. First a byte string is constructed containing two zeros in the beginning - which are a placeholder for the CRC, and the rest of the data. The placeholding zeros are replaced with actual CRC once it is calculated. The receiver will calculate the CRC the same way in order to verify the data. It will copy the received CRC from the frame to the temporary variable and zero it's bytes in the frame - now the frame is in the same state as on the transmitter side before the transmitter calculated the CRC. If that operation wouldn't have place of course the checksums would be different.
The slip_recv() subroutine is exactly implemented the same way as in libpca. One thing of explanation here, slip_recv() has been implemented in more or less universal fashion - it takes a references to a getchar routine (line 38) which can be implemented over any kind of link. Next the command line input is parsed into global variables (lines 68-74), the serial port is initialized (lines 76-84) and we wait for 4 seconds. Now why is that ? As mentioned previously, after establishing a serial connection RS232 control lines will be toggled which in result will reset an Arduino, so we need to wait some time in order for Arduino to become ready and operational. By default (as it is visible in the code), the Arduino's Timer0 and Software timer will be initialized so the overflow happens with a frequency of ~ 1 Hz. The LED will blink with that frequency until the SLIP frame will be received with new configuration values for the timer. In lines 100-101 the SLIP frame containing our binary data is send over the serial line next the script waits for the SLIP frame from Arduino containing the response. The struct response is very simple, it contains only the CRC16 (2 bytes) and the status. It is unpacked after reception (line 107):
# unpack the serialized data to variables
my ($crc, $status) = unpack("SC", $data);
# calculate the checksum of the response for verification
my $crc_calcd = crc16(pack("SC", (0,$status)));
... and repacked again with CRC set to zero - the CRC is recalculated again for verification purposes. The last line simply presents the results.
Host Slip Communication in C
As far as our simple communication program was relatively short in Perl. We will have to dedicate much more effort in order to achieve the same in C (I imagine that everyone straight away lost their good mood now :)), but it's possible and sometimes it can be a better solution than any scripting language (small systems without any kind of interpreter available). Let's try to create functionally identical program in C. As previously we need some tools before we start
- Serial Port IO and init routines
- CRC16 implementation
The bad news is that we need to create those ourselves. Let's start with the CRC16 algorithm.
The best and most comprehensive source for any algorithm or data structure implementation is ... linux kernel. Yes, that's right, everything that you will probably ever need is already implemented there and tested, just have a look at kernel's lib/ directory and pick what you need. I'm sure it will be there. The crc16 implementation is there for sure, we need the crc16.c file, slightly modified since we don't need the kernel specific stuff, have a look bellow:
Next, the serial port IO and initialization.
Serial port is a tty device, we need to initialize the speed, parity, flow control and other stuff through a termios API. I found this code some time ago and simply copy and paste it to my projects, there's no philosophy behind it, the tty_attrib_conf() initializes the port speed and other IO specifics, the tty_block_conf() configure, whether the calls to read() should block until the requested amount of data is collected. We will use the port by simply opening it (as like a file) and write/read to/from it using standard file IO API.
The last thing is the SLIP implementation itself. Of course I'm not going to duplicate it for the host application. Since it is already implemented in the libpca I'll use that implementation. Some preparations are needed though. libpca SLIP module expects to see common.h header file and config.h header file. We'll create dummy versions of those. config.h will contain:
#define SLIP_IMPLEMENT_CRC16 0
unsigned int slip_sendc(unsigned char a_char) {
/* printf("Sending char: %02x\n", a_char); */
return write(g_fd, (void *)&a_char, 1);
}
unsigned char slip_getc(unsigned char *a_data) {
unsigned char x = read(g_fd, a_data, 1);
/* printf("Recv: %02x\n", *a_data); */
return x;
}
CRC16 checksum calculation
The best and most comprehensive source for any algorithm or data structure implementation is ... linux kernel. Yes, that's right, everything that you will probably ever need is already implemented there and tested, just have a look at kernel's lib/ directory and pick what you need. I'm sure it will be there. The crc16 implementation is there for sure, we need the crc16.c file, slightly modified since we don't need the kernel specific stuff, have a look bellow:
Next, the serial port IO and initialization.
Serial port initialization
Serial port is a tty device, we need to initialize the speed, parity, flow control and other stuff through a termios API. I found this code some time ago and simply copy and paste it to my projects, there's no philosophy behind it, the tty_attrib_conf() initializes the port speed and other IO specifics, the tty_block_conf() configure, whether the calls to read() should block until the requested amount of data is collected. We will use the port by simply opening it (as like a file) and write/read to/from it using standard file IO API.
The last thing is the SLIP implementation itself. Of course I'm not going to duplicate it for the host application. Since it is already implemented in the libpca I'll use that implementation. Some preparations are needed though. libpca SLIP module expects to see common.h header file and config.h header file. We'll create dummy versions of those. config.h will contain:
#define SLIP_IMPLEMENT_CRC16 0
to disable the CRC implementation - it relies on the functions from avr-libc which of course are not available in libc/glibc for the standard GNU/Linux system. The common.h will contain the overriden IO functions:
#define SLIP_CHAR_SEND(__x) slip_sendc(__x)
#define SLIP_CHAR_RECV(__x) slip_getc(__x)
unsigned int slip_sendc(unsigned char a_char);
unsigned char slip_getc(unsigned char *a_data);
unsigned int slip_sendc(unsigned char a_char) {
/* printf("Sending char: %02x\n", a_char); */
return write(g_fd, (void *)&a_char, 1);
}
unsigned char slip_getc(unsigned char *a_data) {
unsigned char x = read(g_fd, a_data, 1);
/* printf("Recv: %02x\n", *a_data); */
return x;
}
The slip_send and slip_recv functions in the host application will use those to perform IO. Of course we need a Makefile to solve the inclusion paths and to make the whole project compilable. Finally our C application to send the binary data will look the following way:
Let's shortly walk through the code. First I open and create the file descriptor for the serial port (lines: 14-17), next two lines configure the port attributes and set the port to blocking mode. Next 5 lines (22-27) simply set the data in the struct timer_data which will be sent through the serial port to the Arduino. In line 30 and 31 the crc of the data packet is calculated and placed in the packet itself, finally on lines 42,43 and that's it.
How to use it ?
That's really simple. In order to test the examples presented bellow one can obtain etheir a project snapshot from here - it contains a snapshot of libpca and the source for the Arduino firmware (arduino_slip) - simply navigate to this directory and type make to build it and make install to flash it to Arduino. On the host side one can use either the host_slip.pl Perl script or build the application in host_slip directory by typing make. The application can be used the same way as host_slip.pl script.
It's possible to obtain all the sources from my public repositories from github in order to do that, one should
- clone the newest version of libpca
- clone my Arduino projects repository
- navigate to projects/binary_slip
- build both arduino_slip (remember to disable the TDELAY_IMPLEMENT_T0_INT in libpca config.h file since we are defining our own custom interrupt handler) and host_slip application
How to use the application ? After flashing the Arduino, regardless if we'll gonna use the perl script or the C app, the syntax is the same:
$ ./host_slip.pl
host_slip.pl <prescaler> <ocr> <max_st>
The first parameter is the prescaler value (0-5), the second one is the value which will be placed in the OCR, and the last is the maximum value for the software timer, by invoking:
$ ./host_slip.pl 5 255 64
we'll program the timer to blink with a frequency around ~ 0.5 Hz. Experiment with the values.
I hope that this article shed some light on the aspects of binary data exchange between the Arduino and any other machine and I hope that it proved that it's not that very difficult. Next time I'll try to present a project using this knowledge to build some more useful stuff.
No comments:
Post a Comment