Tag Archives: syslog

Designing a search system for log data — part 1

November 22, 2015 Philip O'Toole Leave a comment

This is the first part of a 3-part series “Designing and building a search system for log data”. Part 2 is here, and part 3 is here.

For the past few years, I’ve been building indexing and search systems, for various types of data, and often at scale. It’s fascinating work — only at scale does O(n) really come alive. Developing embedded systems teaches you how computers really work, but working on search systems and databases teaches you that algorithms really do matter.

Continue reading Designing a search system for log data — part 1 →

Writing a Syslog Collector in Go

July 24, 2014 Philip O'Toole 7 Comments

I’ve started coding in Go (golang), and I received some advice recently from Robert Griesemer, whom I was fortunate enough to sit beside at a recent Go Meetup. To learn Go, Robert suggested that I code a solution in Go for a problem I had previously solved in a different language.

Continue reading Writing a Syslog Collector in Go →

Infrastructure at Scale: Apache Kafka, Twitter Storm and elasticsearch

December 25, 2013 Philip O'Toole Leave a comment

AWS have posted the video online of Jim Nisbet’s and my talk at AWS:reinvent 2013. In it, Jim and I describe the system we built at Loggly, which uses Apache Kafka, Twitter Storm, and elasticseach, to build a high-performance log aggregation and analytics SaaS solution, running on AWS EC2.

Continue reading Infrastructure at Scale: Apache Kafka, Twitter Storm and elasticsearch →

Speaking at AWS re:Invent 2013

November 16, 2013 Philip O'Toole Leave a comment

This past week I had the opportunity to speak, with my colleague Jim Nisbet, at AWS re:Invent 2013. Titled “Unmeltable Infrastructure at Scale: Using Apache Kafka, Twitter Storm, and Elastic Search on AWS“, Jim and I described the architecture of Loggly’s next-generation log aggregation and analytics Infrastructure, which went live 3 months ago, and runs on AWS EC2.

Continue reading Speaking at AWS re:Invent 2013 →

Loggly Generation 2 Released!

September 7, 2013 Philip O'Toole

After 14 months of hard work, the next generation of Loggly has been released. It’s been a great time to be part of the Software Infrastructure team at Loggly and we have put together a superb log aggregation & real-time analytics platform.

We used a combination of custom log Collectors, Apache Kafka, Twitter Storm, ElasticSearch, and lots of secret sauce. You can find more details about the technology stack from my Loggly blog post.