REDIS

by Dermot Tynan in Microtransat

---

Posted on Saturday, October 26, 2013 at 08:00

---

After a lunchtime conversation with a friend of mine, I ditched some of the earlier design considerations around message-passing and MQ-based systems in favour of the NoSQL database Redis.

Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

That's according to the official blurb on the web site, at least. It has several interesting components which are useful for Beoga Beag. It has a rudimentary pub/sub architecture pattern, it is extremely lightweight, it stores its database in memory rather than on a disk (so it's less likely to burn through the Compact Flash file system), and it has an "append-only file" (AOF) archive mechanism.

The AOF system means that updates to the database are recorded sequentially in a journal file. Normally this wouldn't be a great way to operate as the journal file can get quite large. In this case, however, it's ideal. By journalling all of the database updates, I can examine the database at any moment in time, and recreate the decisions made by the boat during its missions. Also, even though the journal can use a lot of space, it is only writing the disk blocks once. For contrast, imagine a traditional database which is storing the latitude/longitude at a particular block on the file system. If the lat/long is updated once per second, or 86,400 times per day, unless the OS does something clever then the lifetime of that particular disk block will be exceeded in a day or two.

I will still need to retain the classic Redis disk dump, but the write frequency of the dumps can be tuned to be no more frequent than every 60 seconds or so, which reduces the load on specific disk blocks. Furthermore, I will be looking to time-stamp the disk dump so that the system will use a new file for each day.

There are a number of language interfaces to Redis, which means I can use more than one language on the main system. For example, I can use C code to read and parse the GPS data, saving the latitude and longitude (in radians) directly in the Redis database. I can then use Ruby to process the GPS position and determine a new course.

---