rqlite is a distributed relational database, which uses SQLite as its storage engine. rqlite is written in Go and uses Raft to achieve consensus across a set of SQLite databases.
With the v3 release series, rqlite can now replicate SQLite databases on a global scale. Let’s see it in action using the AWS EC2 cloud.
rqlite is a distributed relational database, with SQLite as its storage engine. v3.0.1 has been released and it is a significant upgrade relative to the 2.0 series. The 3.0 series allows more sophisticated clusters to be built and simplifies rqlite client coding requirements.
Thanks to Zac Medico, there are new Python clients available for rqlite — a pure Python client, and a SQLAlchemy mapper.
Check them out on GitHub.
To help with the growth of rqlite, it’s been moved to a new repository, under a dedicated organization. Github made this transfer very easy.
I decided on a new logo too.
rqlite is a replicated relational database built on SQLite, with distributed consensus provided by the Raft consensus protocol. Written in Go, v2.2.1 is out now.
I made a presentation on rqlite tonight at the San Francisco Go Meetup. It was an enjoyable evening, and I had a chance to discuss why I built rqlite, how it works, and where it might go in the future.
Programming a database is fascinating work. I’ve been deeply involved with developing open source databases for the past two years and programming a database is possibly the most instructive project one can ever complete as a software developer.
What’s really striking however, is how much my attitude towards databases has changed over the past 6 years. From a state of disinterest, I’ve come to think of these systems as a pinnacle of software engineering.
Pre-built binaries for rqlite and Ekanite, for both Linux and OSX, are now available. That way users can get up and running quickly. Check out the Releases tab of each project on Github for more details.
rqlite is a replicated SQLite database, with distributed consensus provided by the Raft consensus protocol. Written in Go v2.0 is now out. This release is a significant upgrade relative to v1.0.
rqlite provides robust replication for SQLite databases using the Raft consensus protocol. Coded in Go it ensures that all changes made to the leader SQLite database are replicated to all other nodes in the cluster, providing fault-tolerance and reliability.
It’s been 18 months since development of rqlite first started and it’s time for version 2.
I’ve started replacing go-raft within rqlite with the implementation from Hashicorp. go-raft is no longer maintained, and I’ve good experience with the Hashicorp code, due to my work with InfluxDB and hraftd. I’m also going to change the API, so it’s more useful. The existing implementation and API has been tagged as v1.0, so it’s still available.
You can follow the work on this branch, and I hope to merge it to master in the near future.
Ekanite is an open-source Syslog server with built in log search. Thanks to some nice work by Fabian Zaremba, Ekanite now supports searching your logs via a browser.
If you’d like to understand more about the design and development of Ekanite, check out this series of posts.
“Bad money drives out good.”
When is the last time you spoke with your fellow developer? I mean actually spoke? Or was it just over Slack?
I like really Slack, Flowdock, Hipchat and the like. I couldn’t do my job without them. But as with Gresham’s law, bad communication is driving out good.
It’s been 18 months since the first commit to my first significant Go project — syslog-gollector. After an initial burst of activity to create a functional Syslog Collector that streamed to Apache Kafka, the source code hadn’t been updated much since. But today I received a report that it no longer built, so I spent some time porting the code to the latest Shopify Sarama framework.
It was amusing to see how naive much of my early Go code was.
While running services is hard, it’s also fun. It confronts you with a whole set of engineering challenges because you’re building — and running — large scale systems.
So after 18 great months working on the InfluxDB source, I’m returning to Services and have started a new role with Percolate, as Director of Engineering on their Data Platform team.
I recently presented at the InfluxDB San Francisco Meetup, on InfluxDB and the Raft consensus protocol. My talk was about the fundamental problems of distributed systems, and how InfluxDB uses Raft to solve these issues.
This is the last part of a 3-part series “Designing and building a search system for log data”. Be sure to check out part 1 and part 2.
In the last post we examined the design and implementation of Ekanite, a system for indexing log data, and making that data available for search in near-real-time. Is this final post let’s see Ekanite in action.
This is the second part of a 3-part series “Designing and building a search system for log data”. Be sure to check out part 1. Part 3 follows this post.
In the previous post I outlined some of the high-level requirements for a system that indexed log data, and makes that data available for search, all in near-real-time. Satisfying these requirements involves making trade-offs, and sometimes there are no easy answers.
This is the first part of a 3-part series “Designing and building a search system for log data”. Part 2 is here, and part 3 is here.
For the past few years, I’ve been building indexing and search systems, for various types of data, and often at scale. It’s fascinating work — only at scale does O(n) really come alive. Developing embedded systems teaches you how computers really work, but working on search systems and databases teaches you that algorithms really do matter.
When you’d like to contribute to an open-source project it can be difficult to know where to start. Check out my latest post for the InfluxDB blog, explaining how we on the Core team have curated a set of issues, hopefully making it easy for potential contributors to start.