Today sees the launch of Analytics 2.0 on the Percolate platform. After 12 months of hard work by my team, I am very proud of the new platform.
1 year ago the San Francisco team was tasked with rebuilding the Analytics system at Percolate. In place of our legacy MySQL-based system, we now have a brand new architecture, based on Apache Kafka and Elasticsearch. It’s more responsive, more flexible, and offers much richer functionality.
You can learn all about the new system on the Percolate blog.
Last night Percolate hosted the San Francisco Elasticsearch Meetup.
I acted as host, and it was a great night, with excellent speakers and presentations.
Continue reading Elasticsearch Meetup at Percolate
The creator of the network monitoring system Riemann, Kyle Kingsbury, has put together a comprehensive series of blog posts, on the fault-tolerance, high-availability, and general correctness of number of database and storage technologies. Of the technologies discussed I am most familiar with — elasticsearch and Apache Kafka — I found the posts to be a great read.
If you haven’t read them yet, you should check them out on his site.
Packt recently asked me to review their new publication Mastering ElasticSearch by Rafał Kuć and Marek Rogoziński. Since most of my experience with elasticsearch has been from a systems points of view — index management, cluster maintenance, indexing performance — I paid most attention to the chapters about those parts of elasticsearch.
Continue reading Book Review: Mastering ElasticSearch
AWS have posted the video online of Jim Nisbet’s and my talk at AWS:reinvent 2013. In it, Jim and I describe the system we built at Loggly, which uses Apache Kafka, Twitter Storm, and elasticseach, to build a high-performance log aggregation and analytics SaaS solution, running on AWS EC2.
Continue reading Infrastructure at Scale: Apache Kafka, Twitter Storm and elasticsearch
Loggly recently held an elasticsearch meetup, which was a great success. One question that was repeatedly asked was how to ensure elasticsearch does not suffer a partition — known as a split-brain. This can be a particular problem in AWS EC2, where the network is subject to interruptions. It can also happen if the elasticsearch master node performs long garbage collection cycles.
One configuration that is very effective at preventing this problem is described in this post.
Continue reading Avoiding elasticsearch split-brain