Ekanite is an open-source Syslog server with built in log search. Thanks to some nice work by Fabian Zaremba, Ekanite now supports searching your logs via a browser.
If you’d like to understand more about the design and development of Ekanite, check out this series of posts.
“Bad money drives out good.”
When is the last time you spoke with your fellow developer? I mean actually spoke? Or was it just over Slack?
I like really Slack, Flowdock, Hipchat and the like. I couldn’t do my job without them. But as with Gresham’s law, bad communication is driving out good.
It’s been 18 months since the first commit to my first significant Go project — syslog-gollector. After an initial burst of activity to create a functional Syslog Collector that streamed to Apache Kafka, the source code hadn’t been updated much since. But today I received a report that it no longer built, so I spent some time porting the code to the latest Shopify Sarama framework.
It was amusing to see how naive much of my early Go code was.
While running services is hard, it’s also fun. It confronts you with a whole set of engineering challenges because you’re building — and running — large scale systems.
So after 18 great months working on the InfluxDB source, I’m returning to Services and have started a new role with Percolate, as Director of Engineering on their Data Platform team.
I recently presented at the InfluxDB San Francisco Meetup, on InfluxDB and the Raft consensus protocol. My talk was about the fundamental problems of distributed systems, and how InfluxDB uses Raft to solve these issues.
This is the last part of a 3-part series “Designing and building a search system for log data”. Be sure to check out part 1 and part 2.
In the last post we examined the design and implementation of Ekanite, a system for indexing log data, and making that data available for search in near-real-time. Is this final post let’s see Ekanite in action.
This is the second part of a 3-part series “Designing and building a search system for log data”. Be sure to check out part 1. Part 3 follows this post.
In the previous post I outlined some of the high-level requirements for a system that indexed log data, and makes that data available for search, all in near-real-time. Satisfying these requirements involves making trade-offs, and sometimes there are no easy answers.
To better understand the design of these systems, this post examines in detail a program I wrote to receive logs over the network, index the data, and allow the logs to be searched. The program is called Ekanite.
This is the first part of a 3-part series “Designing and building a search system for log data”. Part 2 is here, and part 3 is here.
For the past few years, I’ve been building indexing and search systems, for various types of data, and often at scale. It’s fascinating work — only at scale does O(n) really come alive. Developing embedded systems teaches you how computers really work, but working on search systems and databases teaches you that algorithms really do matter.
When you’d like to contribute to an open-source project it can be difficult to know where to start. Check out my latest post for the InfluxDB blog, explaining how we on the Core team have curated a set of issues, hopefully making it easy for potential contributors to start.
Another post for the InfluxDB blog — on testing the storage engines within InfluxDB.
You can check it out here.
Hashicorp provide a nice implementation of the Raft consensus protocol, and it’s at the heart of InfluxDB (amongst other systems). I wanted to experiment with a simple system built using this particular Raft implementation, so was inspired by raftd to built hraftd.
“Run into an obstacle in what you’re working on? Hmm, I wonder what’s new online. Better check.”
If you haven’t already, you should start reading Paul Graham’s essays. In one on philosophy, Graham believes that many of the answers provided by philosophy are useless because “…of how little effect they have”. By that standard another of his essays is of high utility because it has affected the way I program. John Stuart Mill would be pleased.
I always use the names of economists for my machines’ hostnames. keynes, friedman, marx, fisher, ricardo. So every so often the strange economics of open-source software hits me.
I’ve written my first post for the InfluxDB blog. In it I discuss the new statistics and monitoring system built into InfluxDB, starting with the 0.9.4 release. Functionality like this is critical when it comes to running a distributed database like InfluxDB.
You can check it out here.
It’s been 418 days since my first Github commit of Go code. In that time I’ve written a Syslog-to-Kafka producer, a Raft-based distributed SQLite database, a near real-time log search system, and become a core developer of InfluxDB.
I came across another great video about software engineering management, this time by Bryan Cantrill. It’s a really great talk, and discussed in-depth — with plenty of humour thrown in — the importance of Mission to high-performing software developers.
I’ve recently been thinking about why running Services is particularly hard. By Services I mean Software-as-a-Service platforms. During the years, I’ve written software for many different systems — embedded software, web services, databases, and distributed systems, but being involved with designing and running a SaaS platform was difficult in a whole new way: running Services is hard work.
This past week I attended Gophercon 2015, in Denver, CO. It was also a chance to get together with the rest of the InfluxDB team. And because the Go community is still relatively young and small, it was a great chance to meet, in person, some of the best people working with Go today.
The first version of the 0.9.0 series of InfluxDB has been released. It’s alpha-quality software but all of us on the InfluxDB team are very excited to see the software reach this stage.
You can read more about the release on this blog post.
Well, almost nothing.
Obviously it’s got something to do with computers since developers spend so much of their time in front of one. But software development is actually all about people. And successful software development even more so.