Jan 8, 2012

Sending Large Strings of Data to Arduino

For one of my projects I needed to send between 200 and 500 bytes of data to my Arduino quickly over a serial connection. The problem I faced is that the Arduino only has a 128 byte read buffer. Sending strings of data longer than 128 bytes would cause unpredictable results, especially at higher speeds. Sometimes the Arduino would receive all the data successfully and sometimes it would fail.

Specifically, my project involved sending commands to the Arduino with the following requirements:

All commands are simple ASCII strings, terminated with a newline character
Most commands are small, only 1~2 characters in length
Few commands are over 128 characters in length, causing the Arduino’s receive buffer to overflow
All commands have to be sent to the Arduino with absolutely no loss of data
Latency kept to a minimal
When the Arduino detects a serial link, it sends OK which the host checks to make sure the serial link is usable

I met the above requirements by creating a custom protocol for sending larger commands:

All small commands are read as usual, introducing no unneeded latency
If a command starts with RCV, then the custom protocol is invoked to read 128 bytes at a time until the full command is received

Perhaps the best way to illustrate the custom protocol is with an example of sending 200 bytes of ‘a’s to the Arduino:

# The host initiates a new command 200 bytes in length
host> RCV 200
# The Arduino states that it is ready to receive the data
arduino> RDY
# The host sends the first 128 bytes
host> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
# The Arduino acknowledges that it successfully received 128 bytes of data
arduino> ACK 128
# The host sends the remaining 72 bytes:
host> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
# The Arduino acknowledges the 72 bytes of data, and the transfer is completed
arduino> ACK 72

I created a simple Arduino test program which can receive any string up to 1024 bytes in size and echoes it back over the serial connection:

#include <stdio.h>

void setup() {
	Serial.begin(115200);

	Serial.println("OK");
}

char command[1024];
char commandBuffer[128];
int commandBufferSize = 0;

void readCommandBuffer(int bytesToRead) {
	int i = 0;
	char c = 0;
	while (i < 128 && (i < bytesToRead || bytesToRead <= 0)) {
		while (!Serial.available())
			;
		c = Serial.read();
		if (c == '\r' || c == '\n') {
			break;
		}
		commandBuffer[i] = c;
		i++;
	}
	commandBufferSize = i;
}

void readCommand() {
	command[0] = '\0';
	readCommandBuffer(0);
	if (strncmp(commandBuffer, "RCV", 3) == 0) {
		commandBuffer[commandBufferSize] = '\0';
		int expectedSize = atoi(commandBuffer + 4);
		if (expectedSize <= 0 || expectedSize > 1024) {
			return;
		}
		Serial.println("RDY");
		int bytesRead = 0;
		while (bytesRead < expectedSize) {
			readCommandBuffer(expectedSize - bytesRead);
			memcpy(command + bytesRead, commandBuffer, commandBufferSize);
			bytesRead += commandBufferSize;
			Serial.print("ACK ");
			Serial.println(commandBufferSize);
		}
		command[bytesRead] = '\0';
	} else {
		memcpy(command, commandBuffer, commandBufferSize);
		command[commandBufferSize] = '\0';
	}
}

void loop() {
	if (Serial.available()) {
		readCommand();
		// "command" now contains the full command
		Serial.println(command);
	}

To test the stability of this implementation, I also created a Python script which creates a random string between 1 and 1023 characters in length, sends it to the Arduino, and verifies that it was sent correctly by comparing the string echoed by the Arduino to the original string. It repeats 1000 times and prints the amount of successes and failures at the end. You’ll need pySerial for this.

import math

import serial
from threading import Thread, Lock
class Serial:
	def __init__(self, port='/dev/cuaU0', rate=9600, timeout=10):
		self._serial = serial.Serial(port, rate, timeout=timeout)
		self._mutex = Lock()
		self._mutex.acquire()
		response = self._serial.readline().strip()
		if response != 'OK':
			raise Exception("Failed to communicate with the serial device!")
		self._mutex.release()

	def _shortCommand(self, command):
		self._serial.write(command)
		response = self._serial.readline()
		return response.strip()

	def _longCommand(self, command):
		response = self._shortCommand('RCV ' + str(len(command)) + "\n")
		if response != 'RDY':
			return None
		for i in range(int(math.ceil(len(command) / 128.0))):
			c = command[128*i:128*(i+1)]
			response = self._shortCommand(c)
		return self._serial.readline().strip()

	def command(self, command):
		self._mutex.acquire()
		if len(command) < 128:
			response = self._shortCommand(command + "\n")
		else:
			response = self._longCommand(command)
		self._mutex.release()
		return response

import random, string
def main():
	serial = Serial(port='/dev/cu.usbserial-A6008ikf', rate=115200)
	passed = 0
	failed = 0
	for i in range(1000):
		l = random.randint(1, 1023)
		s = ''.join(random.choice(string.ascii_uppercase + string.ascii_lowercase + string.digits) for x in range(l))
		o = serial.command(s)
		if o == s:
			print (i + 1), 'OK', l, 'bytes'
			passed = passed + 1
		else:
			print (i + 1), 'FAIL', l, 'bytes'
			failed = failed + 1
	print 'Passed:', passed
	print 'Failed:', failed

if __name__ == '__main__':
	main()

I found that the implementation was completely stable even at 115200 baud. The Serial class from the above snippet should even work in a multithreaded Python script with no problems. It’s the same code I’m using in my project.

Hope this is of some use to someone!