I’ve started coding in Go (golang), and I received some advice recently from Robert Griesemer, whom I was fortunate enough to sit beside at a recent Go Meetup. To learn Go, Robert suggested that I code a solution in Go for a problem I had previously solved in a different language.
So I decided to code an RFC5424-compliant Syslog Collector, a problem I had to solve while at Loggly — and had to solve well. At that time I solved it using C++ and the Boost ASIO library. At a high-level the goals of the Collector are to receive log messages over TCP or UDP, perform some basic parsing, and then stream those messages to Apache Kafka.
Coding the Collector in Go was a very interesting experience. I am impressed with Go — I find that the code I write behaves in the manner I expect. The Go net package also had some nice features that helped enormously to deal with some common corner cases.
For example, you can easily trigger timeouts if data has not been received on a socket after a certain period of time. Go also comes with basic packages for collecting statistics within the program itself. Features like this show me that the Go creators really did have system software in mind when creating the language.
I also really enjoy that I don’t have to write code using a heavyweight IDE, which is almost required when coding in Java. I coded the Collector using vim, though I took the time to install the vim-go plugin. I do plan a more detailed post in the future describing my Go development setup.
I hope to develop this project further. You can find the code on github.
Hi, I get this error trying to compile the syslog-gollector tool.
user@box:/home/syslog-gollector# go install github.com/otoolep/syslog-gollector
src/github.com/otoolep/sarama/snappy.go:5:2: cannot find package “code.google.com/p/snappy-go/snappy” in any of:
/usr/lib/go/src/pkg/code.google.com/p/snappy-go/snappy (from $GOROOT)
/home/syslog-gollector/src/code.google.com/p/snappy-go/snappy (from $GOPATH)
Hi John — the best place for feedback like this is via an issue on Github. I’ll take a look at the build break and see what is going on.
https://github.com/otoolep/syslog-gollector/issues
We do have a real issue:
https://github.com/otoolep/syslog-gollector/issues/20
I’ll fix it up shortly.
Hi Philip:
Thanks for sharing the code of syslog-collector. I am new in learing programing using Golang. It helped me a lot. However, I have a question about it. If the syslog-collector receives too much syslog message or the Kafka responds slow, the collector can’t process all syslog promptly? How do we handle this? the excessive syslog message will be dropped? What library in go will help with this? Any suggestion is much appreciated?
Cheers
Hi Gavin — syslog-collector, in the default mode, will buffer that data in RAM. Lines, as they are received, are written to a channel until they can be drained by the other side. If they are not drained, they will simply pile up. So eventually the system could run out of RAM, the program crash, and the data lost.
See https://github.com/otoolep/syslog-gollector/blob/master/input/input.go#L101 to see the lines getting written to the channel.
You may be interested in reading my other post, about approaches to dealing with too much data: https://www.philipotoole.com/drop-throttle-buffer/
Thanks Phillip for all the suggestions. It is much apprecaited. Sorry for getting back to you. I thought that I will get an email notification when you replied to the post.