rqlite – replicated SQLite with new Raft consensus and API

Raft consensus protocolrqlite provides robust replication for SQLite databases using the Raft consensus protocol. Coded in Go it ensures that all changes made to the leader SQLite database are replicated to all other nodes in the cluster, providing fault-tolerance and reliability.

It’s been 18 months since development of rqlite first started and it’s time for version 2.

New Raft consensus module

V1 of rqlite used go-raft as its Raft consensus module. However go-raft is no longer maintained, and it was time for a new module. I am quite familiar with the Hashicorp Raft consensus module due to my work with InfluxDB and hraftd, so swapping in that new module was straightforward.

rqlite cluster
A rqlite cluster, which fully replicates a SQLite database

This new Raft module also uses persistent TCP connections between the nodes; in contrast go-raft used HTTP requests and responses. The persistent connections lowers latency and increases throughput. Hashicorp Raft is used in production by many other systems, such as Consul and Nomad, and should provide enhanced stability and performance.

A better API

The API offered by version 1 was rather simplistic, and didn’t make it easy to build libraries on top of rqlite. Column order of returned results was not deterministic, and important information like last_insert_id was not available.

This has all been addressed in the new v2 API, and the changes should make it much easier to build libraries that use rqlite. A Python driver is already in development.

The rqlite API is documented via examples in the README. It explains how to read and write to the cluster, shows the format of the responses, and how to control such functionality as transactions.

Backup Support

The latest version of rqlite also supports backing up the database. A consistent snapshot of the SQLite database can be retrieved at any time, and used to restore a node later.

Deploying rqlite

You can find the source code for rqlite, and instructions on how to build and deploy it, here on github. I hope to continue developing this software, as databases and distributed consensus systems are immensely interesting.

5 thoughts on “rqlite – replicated SQLite with new Raft consensus and API”

  1. Kudos for implementing distributed sqlite. I really wanted to use it for my hobby project. I see all read req/queries must go through leader. This will not scale. What is your opinion about using a commit log?

    1. Cheers devendra. The point of rqlite is not scale for performance, it’s for replication. I am going to enhance rqlite such queries do not need to go through the leader. Even when they do go through the leader, they will be fast since they will just need to hit the local database (in default mode).

      I don’t follow your comment about the commit log. If you have ideas about enhancing rqlite, a ticket on Github would be best.

  2. How interchangeable or different is this from Consul? Does it play nice/along side it?

    I currently use Consul for service and server announcements in a distributed design. Works great for name resolution, etc as it’s designed, but I also hit the issue of how to provide that distributed redundant sqlite backend. I was looking a things like sqlproxy just to sync data, but releveraging something already used would be better…

  3. Malcolm — it’s a different system to Consul. Apart from the fact that both systems use the same Raft implementation, rqlite and Consul are not compatible.

    1. Ok. Thanks for that. I’ll still check it out – I didn’t dig too deep into the workings straight awsy. It still may do what I want though.

Leave a Reply

Your email address will not be published. Required fields are marked *