Designing a search system for log data — part 1

This is the first part of the series “Designing a search system for log data”.

For the past few years, I’ve been building indexing and search systems, for various types of data, and often at scale. It’s fascinating work — only at scale does O(n) really come alive. Developing embedded systems teaches you how computers really work,  but working on search systems and databases teaches you that algorithms really do matter.

Continue reading

Coding like it’s 1999

“Run into an obstacle in what you’re working on? Hmm, I wonder what’s new online. Better check.”

If you haven’t already, you should start reading Paul Graham’s essays. In one on philosophy, Graham believes that many of the answers provided by philosophy are useless because “…of how little effect they have”. By that standard another of his essays is of high utility because it has affected the way I program. John Stuart Mill would be pleased.

Continue reading

Running services is hard

I’ve recently been thinking about why running Services is particularly hard. By Services I mean Software-as-a-Service platforms. During the years, I’ve written software for many different systems — embedded software, web services, databases, and distributed systems, but being involved with designing and running a SaaS platform was difficult in a whole new way: running Services is hard work.

Continue reading

Book Review: Cassandra High Availability

Packt recently asked me to review their new publication Cassandra High Availability, written by Robbie Strickland. I’ve worked with Cassandra in the past — early designs of Loggly‘s 2nd generation Log analytics platform used Cassandra as its authoritative store for log data, but we ended up pulling it and using elasticsearch as both the store and search engine.

Continue reading

Software Development for Infrastructure

Bjarne Stroustrup has another very interesting paper on his website.  Titled Software Development for Infrastructure, it discusses some key ideas for building software that has “…more stringent correctness, reliability, efficiency, and maintainability requirements than non-essential applications.”  It is not a long paper, but offers useful observations and guidelines for building such software systems.

Continue reading

Philip O'Toole