rqlite is a lightweight, easy-to-use, open-source, distributed relational database. It’s written in Go and uses SQLite as its storage engine.

Release 8.0 is out now and brings substantial advancements to its handling of large data sets. It also adds many new features, and brings improvements in simplicity and operation — all the while maintaining rqlite’s focus on robustness and quality.

Before we get into how all this was done, let’s look at what’s new in 8.0.

What’s New?

Enhanced Large Data Set Support: 8.0 is designed to handle very large data sets efficiently, marking a step forward in rqlite’s capability to manage substantial volumes of data.

Refined Clustering Mechanism: All internode communications now occur over dedicated network connections, streamlining secure cluster creation.

On-Disk Database by Default: Keeping the focus on operational simplicity, 8.0 runs exclusively with an on-disk database without any performance hit. Removing in-memory database support also reduces development complexity going forward.

Seamless Upgrade Path: Upgrading from the 7.x series is straightforward. It should be mostly a seamless transition, though backing up your data before upgrading is always recommended.

Improvements and Features

• Upgraded SQLite to 3.44.0, ensuring up-to-date database features.
• SQLite FTS5 support, enhancing rqlite’s full-text search capabilities.
• Improvements to the rqlite CLI, including mTLS support.
• Added straightforward CORS (Cross-Origin Resource Sharing) support, increasing web compatibility.
• Support for automatic VACUUM during backup operations.

A new approach to Raft snapshotting

How does rqlite 8.0 achieve its support for very large data sets?

Central to this advancement is a fundamental redesign of the snapshotting process within rqlite’s Raft consensus system. Traditionally, Raft implementations use log truncation, a method where a snapshot is taken of the managed data (in rqlite’s case, a SQLite database). This snapshot then serves as a point-in-time record, allowing the removal of prior entries from the Raft log. This process ensures that the Raft log does not grow indefinitely.

In previous versions of rqlite the Raft snapshotting process involved making a copy of the entire SQLite database. But as the SQL database size increased, snapshotting became progressively more memory-intensive and time-consuming. This set a practical upper limit on the size of the SQLite database. In reality databases couldn’t get much bigger than a couple of gigabytes — not unless you had a lot of memory and plenty of processing power.

With 8.0, rqlite adopts a new strategy: instead of snapshotting the entire SQLite database, it now copies just the Write-Ahead Logging (WAL) file, incrementally updating the Raft snapshot using that data. And once the WAL has been snapshot, rqlite then checkpoints the WAL, ensuring all subsequent writes will be in the next WAL — and this cycle repeats every snapshot. The end result is that the snapshot process only depends on how much data was written to the database since the last snapshot — and has no bearing on the total amount of data in the SQLite database.

Of course, the details are crucial — snapshotting and log truncation are critical to get right — and rqlite must remain stable and correct even if it crashes in the middle of this entire operation. From Raft’s point of view this entire process must be atomic — snapshotting either runs to completion or be as if it never occurred. However, it actually involves multiple steps, sometimes involving large amounts of data.

Snapshotting the WAL was inspired by the design of Ben Johnson’s Litestream which uses a similar approach for backing up SQLite databases to the Cloud.

Why snapshot the WAL?

Snapshotting the WAL offers several advantages. Firstly, it significantly reduces the overhead involved in managing large datasets. Instead of dealing with the entire database, we’re now dealing with a more compact and manageable subset of data changes. This approach also enhances performance, as the rqlite is less resource-intensive and more efficient in general. The data read from the WAL can also be compacted making it even more efficient to process.

Using the WAL also increases write performance, and along with setting Synchronous mode to OFF, write performance now matches that of an in-memory database.

Downloading rqlite

Version 8.0 has been 6 months in the making, and I hope the community will find it useful. Download the latest release and be sure to join the rqlite Slack channel.

Acknowledgments

A big thank you to Ben Johnson, whose work on Litestream has inspired these advancements in rqlite.  And credit also goes to Joe Mordica who first suggested I remove the limits on data set sizes.