I’m a conviction engineer.
To me it’s the only way to be an effective engineer, software developer, and technical leader. You’ve simply got to truly believe in the value of what you do, and you’ve got to believe in doing it the right way.
Tag Archives: programming
rqlite v3.3.0 released
rqlite is an open-source distributed relational database, with SQLite as its storage engine. v3.3.0 has been released and includes new functionality to control the Raft consensus subsystem, along with improvements to the CLI.
You can download the release from GitHub.
rqlite v3.2.1 released
rqlite is an open-source distributed relational database, with SQLite as its storage engine. v3.2.1 has been released and includes new functionality for cluster management, along with new documentation on running clusters. v3.2.0 also includes some bug fixes.
You can download the release from GitHub.
How to organise a Go project
Some fellow developers, using Go for the first time, recently asked me how to organise a Go project and for some high-level guidance on programming using the language.
I thought the most effective way to answer this question was to build a simple Go HTTP service, that provides a key-value store. It also includes a README, outlining my most important guidelines for Go programming. You can check it out here.
rqlite v3: Globally replicating SQLite
rqlite is an open-source distributed relational database, which uses SQLite as its storage engine. rqlite is written in Go and uses Raft to achieve consensus across a set of SQLite databases. It gracefully handles leader election, and can tolerate machine failure.
With the v3 release series, rqlite can now replicate SQLite databases on a global scale, with very little effort. Let’s see it in action using the AWS EC2 cloud.
rqlite v3.0.1 released with leader redirection
rqlite is an open-source distributed relational database, with SQLite as its storage engine. v3.0.1 has been released and it is a significant upgrade relative to the 2.0 series. The 3.0 series allows more sophisticated clusters to be built and simplifies rqlite client coding requirements.
Continue reading rqlite v3.0.1 released with leader redirection
New Python clients for rqlite
Thanks to Zac Medico, there are new Python clients available for rqlite — a pure Python client, and a SQLAlchemy mapper.
Check them out on GitHub.
rqlite at the San Francisco Go Meetup
I made a presentation on rqlite tonight at the San Francisco Go Meetup. It was an enjoyable evening, and I had a chance to discuss why I built rqlite, how it works, and where it might go in the future.
What I learned from programming databases
Programming a database is fascinating work. I’ve been deeply involved with developing open source databases for the past two years and programming a database is possibly the most instructive project one can ever complete as a software developer.
What’s really striking however, is how much my attitude towards databases has changed over the past 6 years. From a state of disinterest, I’ve come to think of these systems as a pinnacle of software engineering.
rqlite and Hashicorp Raft Consensus
I’ve started replacing go-raft within rqlite with the implementation from Hashicorp. go-raft is no longer maintained, and I’ve good experience with the Hashicorp code, due to my work with InfluxDB and hraftd. I’m also going to change the API, so it’s more useful. The existing implementation and API has been tagged as v1.0, so it’s still available.
You can follow the work on this branch, and I hope to merge it to master in the near future.
New browser interface for Ekanite
Ekanite is an open-source Syslog server with built in log search. Thanks to some nice work by Fabian Zaremba, Ekanite now supports searching your logs via a browser.
If you’d like to understand more about the design and development of Ekanite, check out this series of posts.
Revisiting syslog-gollector
It’s been 18 months since the first commit to my first significant Go project — syslog-gollector. After an initial burst of activity to create a functional Syslog Collector that streamed to Apache Kafka, the source code hadn’t been updated much since. But today I received a report that it no longer built, so I spent some time porting the code to the latest Shopify Sarama framework.
It was amusing to see how naive much of my early Go code was.
Designing a search system for log data — part 3
This is the last part of a 3-part series “Designing and building a search system for log data”. Be sure to check out part 1 and part 2.
In the last post we examined the design and implementation of Ekanite, a system for indexing log data, and making that data available for search in near-real-time. Is this final post let’s see Ekanite in action.
Continue reading Designing a search system for log data — part 3
Designing a search system for log data — part 2
This is the second part of a 3-part series “Designing and building a search system for log data”. Be sure to check out part 1. Part 3 follows this post.
In the previous post I outlined some of the high-level requirements for a system that indexed log data, and makes that data available for search, all in near-real-time. Satisfying these requirements involves making trade-offs, and sometimes there are no easy answers.
Continue reading Designing a search system for log data — part 2
Designing a search system for log data — part 1
This is the first part of a 3-part series “Designing and building a search system for log data”. Part 2 is here, and part 3 is here.
For the past few years, I’ve been building indexing and search systems, for various types of data, and often at scale. It’s fascinating work — only at scale does O(n) really come alive. Developing embedded systems teaches you how computers really work, but working on search systems and databases teaches you that algorithms really do matter.
Continue reading Designing a search system for log data — part 1
Contributing to InfluxDB
When you’d like to contribute to an open-source project it can be difficult to know where to start. Check out my latest post for the InfluxDB blog, explaining how we on the Core team have curated a set of issues, hopefully making it easy for potential contributors to start.
Testing InfluxDB Storage Engines
Another post for the InfluxDB blog — on testing the storage engines within InfluxDB.
You can check it out here.
Building a distributed key-value store using Raft
Hashicorp provide a nice implementation of the Raft consensus protocol, and it’s at the heart of InfluxDB (amongst other systems). I wanted to experiment with a simple system built using this particular Raft implementation, so was inspired by raftd to built hraftd.
Continue reading Building a distributed key-value store using Raft
Coding like it’s 1999
“Run into an obstacle in what you’re working on? Hmm, I wonder what’s new online. Better check.”
If you haven’t already, you should start reading Paul Graham’s essays. In one on philosophy, Graham believes that many of the answers provided by philosophy are useless because “…of how little effect they have”. By that standard another of his essays is of high utility because it has affected the way I program. John Stuart Mill would be pleased.
