Writing a Syslog Collector in Go

gopherI’ve started coding in Go (golang), and I received some advice recently from Robert Griesemer, whom I was fortunate enough to sit beside at a recent Go Meetup. To learn Go, Robert suggested that I code a solution in Go for a problem I had previously solved in a different language.

So I decided to code an RFC5424-compliant Syslog Collector, a problem I had to solve while at Loggly — and had to solve well. At that time I solved it using C++ and the Boost ASIO library. At a high-level the goals of the Collector are to receive log messages over TCP or UDP, perform some basic parsing, and then stream those messages to Apache Kafka.

Coding the Collector in Go was a very interesting experience. I am impressed with Go — I find that the code I write behaves in the manner I expect. The Go net package also had some nice features that helped enormously to deal with some common corner cases.

For example, you can easily trigger timeouts if data has not been received on a socket after a certain period of time. Go also comes with basic packages for collecting statistics within the program itself. Features like this show me that the Go creators really did have system software in mind when creating the language.

I also really enjoy that I don’t have to write code using a heavyweight IDE, which is almost required when coding in Java. I coded the Collector using vim, though I took the time to install the vim-go plugin. I do plan a more detailed post in the future describing my Go development setup.

I hope to develop this project further. You can find the code on github.

7 thoughts on “Writing a Syslog Collector in Go”

  1. Hi, I get this error trying to compile the syslog-gollector tool.
    user@box:/home/syslog-gollector# go install github.com/otoolep/syslog-gollector
    src/github.com/otoolep/sarama/snappy.go:5:2: cannot find package “code.google.com/p/snappy-go/snappy” in any of:
    /usr/lib/go/src/pkg/code.google.com/p/snappy-go/snappy (from $GOROOT)
    /home/syslog-gollector/src/code.google.com/p/snappy-go/snappy (from $GOPATH)

  2. Hi Philip:
    Thanks for sharing the code of syslog-collector. I am new in learing programing using Golang. It helped me a lot. However, I have a question about it. If the syslog-collector receives too much syslog message or the Kafka responds slow, the collector can’t process all syslog promptly? How do we handle this? the excessive syslog message will be dropped? What library in go will help with this? Any suggestion is much appreciated?

        1. Thanks Phillip for all the suggestions. It is much apprecaited. Sorry for getting back to you. I thought that I will get an email notification when you replied to the post.

Leave a Reply

Your email address will not be published. Required fields are marked *