400 days of Go

gopherIt’s been 418 days since my first Github commit of Go code. In that time I’ve written a Syslog-to-Kafka producer, a Raft-based distributed SQLite database, a near real-time log search system, and become a core developer of InfluxDB.

So if there is one word to summarize Go, it’s productive.

I haven’t had the same burst in productivity since I first started building web services in Python, so what have been the key reasons?

No IDE required

I’ve written much of my Go code using nothing but vim. Sometimes Sublime, but often just vim. I once tried to write Java code using vim, and the runaway proliferation of source files, of AbstractSingletonProxyFactoryBeans, quickly made an IDE required. Not so with Go, and this is important.

Fast build times

Compile and link a C++ program. Do it 10s of times a day. It adds up to minutes, sometimes hours. But not with Go. The entire InfluxDB system compiles, with Go 1.4, in 3 seconds on my machine. An entire distributed time-series database compiled to native code in 3 seconds. This has done wonders for the productivity of the InfluxDB development team.

Standard library

The standard library is a joy to work with. Many parts of it show me that the Go team really did have systems software in mind when designing the language. Little things like making it easy to set read-and-write timeouts on sockets, or gather server statistics, have made it so much faster to produce robust and reliable code.

Key data structures built-in

Maps are a ubiquitous data structure in programming — I’d guess that 10-20% of all Go code I work with involves maps. That the Go team decided to make them a built-in for a statically-typed language was an incredibly important decision, that has made the language so much more powerful.

Dynamic arrays — Slices in Go — are also critical to Go’s productivity. Along with the functions append and range, coding with these key data structures is a breeze.

By building both in, it has resulted in a standard and systematic way for Go programmers to deal with these data structures. This has become a force-multiplier within teams of Go developers and the wider Go community.

go fmt

Sometimes you don’t realise how much time you are spending doing something until you are not doing it any longer. Source code layout conventions are one of those things. Debating bracket placement, character spacing, noisy diffs due to whitespace changes — all banished to a time before Go. go fmt is an inspired tool, and it is a tribute to the Go team that the choices made by the tool have been so readily accepted by the Go community.

Single binary

Even though this is widely acknowledged when discussing Go, I believe it remains an under-appreciated attribute of Go, something that will become even more important as time passes. Network and storage are cheap, but the deployment, configuration, and management of modern computer systems is not. Anyone who has spent time in Technical Operations — as I have — knows the pain of dependency hell, JDK upgrades, or installing Perl-bindings. The single-binary approach is, and will remain, a significant productivity boost.

Test framework

The test framework is excellent. And perhaps as important, using just the standard library’s test framework has proved to me that using only the standard library (when humanly possible) is the right approach to writing Go code. There are many, many test frameworks for Go, and the temptation to use a third party library is strong, but sticking with one framework keeps the code consistent, which helps productivity enormously.

Performance analysis

The performance analysis tools — CPU profiling and memory profiling — have been key to development at InfluxDB. After all, it’s only with databases that O(n) comes alive.ย  It’s been fascinating analysing — and fixing — the database Go code I’ve written using the CPU profiler.

Tell me how you really feel

So is there anything not so bright-and-shiny?

Garbage collection

Strangely, some key parts of Go don’t feel like they have made a huge difference. Sure garbage collection is nice, but I never had too much trouble writing leak-free C and C++, not with technologies like Valgrind and Boost Smart Pointers. Now I may underestimate the work GC is doing for me — quite possibly — but if I had to track the memory myself, and still write Go code, I don’t think I’d really mind.

Boilerplate

I certainly find myself writing chunks of boilerplate during Go development. This can make code maintenance more difficult. With a language like C++ there is a drive to abstract everything to avoid this issue, and to make heavy use of features such as templates (generics). But anyone who has written a reasonably-sized C++ program knows that this drive is relentless and that’s the problem — it’s very easy for abstraction to go too far.

Within our industry there is now a drive to return to simplicity. Go’s easy acceptance of writing a little boilerplate to make source code navigation less cumbersome , and its comprehension easier, is actually refreshing.

GOPATH

GOPATH is weird. Once you get the hang of it, it’s fine, but early in my Go programming days it caused plenty of confusion. It feels like a bolt-on, and I’ve learned to use a separate GOPATH for every project. It doesn’t feel right, and often I wonder how it made it into the Go system.

The Go Way

Go feels like a pedantic language, but that suits most programmers. It certainly suits me.

Syslog clients, distributed databases, and search systems — and all without a JVM or V8 engine in sight. After 300 days with node.js, the succeeding 400 days with Go have been so much better.

32 thoughts on “400 days of Go”

      1. Using one GOPATH does not hinder separation.

        GOPATH
        ../bin
        ../pkg
        ../src
        …./project1
        …./project2
        …./project3

        and if you plan on distributing…
        ../src
        …./github.com
        ……/yourusername
        ……../yourpackagename(project4)

        One GOPATH. All the seperation you’ll ever need. Adding a new GOPATH for every single project seems ridiculous to me.

        1. Regularly backing up one’s own program sources is difficult when many “go gotten” sources are mixed into the src directory with them, resulting in bloated backup files. A separate src directory for each project is easier for doing zips of sources.

          1. Then create a namespace for yourself.
            $GOPATH
            ./src
            ./myprojects
            ./project1
            ./project2
            ./project3
            ./go-get1
            ./go-get2

          2. The site obliterated my whitespace. Here’s the comment again for clarity.

            Then create a namespace for yourself.
            $GOPATH
            ………/src
            ……………../myprojects
            ……………………./project1
            ……………………./project2
            ……………………./project3
            ……………../go-get1
            ……………../go-get2

      2. With how Go manages packages, workspaces can be elegantly resolved by separating responsibilities. This also assists with the article’s “abstraction” concerns. Google search “go source code”, click the github link, and peek at the src/net directory. Net is a low level package that gives a baseline interface for net related packages to extend upon. Net/Http is a package that extends Net for HTTP related concerns. This is an example of how to design a package, with repositories responsible for specific use cases, and packages below expanding the abstract packages to more detailed use cases.

        Go was designed to utilize version control software, like git, a strategy I use for local-only packages (like prototypes or initial proof of concepts) is I place it in the src directory, with a custom domain, and extend out the “workspaces” within the domain. This allows an easy way reference other packages I locally create, and keep it versioned, and when I want to extend to a remote package, I add a remote repository to my workspace.

        Any packages that need to be compiled should have their own directory within the domain, leveraging package main. If you prefer having your workspaces grouped outside the GOPATH, you can set up the directories for your workspace however you like, and symlink to them in your gopath/src.

        While you can compile go code outside of the GOPATH’s src, the code being written is likely going to be awkward and have growing pains as it extends in complexity since it has no easy way to create packages to improve reusability and abstraction.

    1. I do this as well. Despite the fact that you don’t need an IDE, using one can certainly increase productivity. Using a separate GOPATH per project reduces the number of files my IDE needs to index as well as the number of choices for me to wade through when it suggests packages to import.

  1. Thanks for all the feedback.

    I know a distinct GOPATH is not required, but I find it useful to have such a distinction between workspaces. When it comes to my InfluxDB development — a well-known open-source project — I find it particularly important to maintain a workspace that is clean of any other dependencies, and this is the way I do it. The last thing I want to do is commit or rely on code that is not an explicit dependency specified by InfluxDB (obviously this is unlikely to happen in practise). It suits me to keep my work siloed in this manner.

    1. If you have your git repo directly in $GOROOT or $GOROOT/src you’re “doing it wrong” (or at least against what’s prescribed by tools like ‘go get’ and general best practice).

      You should be namespacing your code.

      Having your git repo and source for example in ‘$GOROOT/src/github.com/githubuser/githubproject’ instead of directly in src will make your code much more reusable by other projects that shouldn’t have to re-vendor your code just to use it.

      1. Thanks Jesse.

        Yes, I follow the practise as you outlined it for my recent personal projects, and as we do at InfluxDB . See our CONTRIBUTING file, to see what we (and I) do.

        And I think you mean GOPATH, no?

  2. I also have the impression to be productive in Go but like you said there’s a lot of boilerplate to write, more than the other languages I use.

    About gofmt one thing that makes me mad is that you can’t import a package you’re not using, seriously I want to keep log imported in case I’d like to print something for testing stuff without having to delete the line and rewrite it every damn time. There should be an option for that.

    I like Go and all, but seriously, I don’t understand that Java rant about AbstractSingletonProxyFactoryBeans, you’re talking (and linking to) about a class that no one uses directly on a daily basis, in a version of Spring that is 8 years old. It has recently become a breeze to write apps and services with Java and Spring, there’s no need to bash them in here ๐Ÿ˜‰

    1. If you change your import statement to from โ€œlogโ€ to _ โ€œlogโ€ it wonโ€™t complain about the unused imports.

    2. I completely agree with Alex. Every article that bashes Java links to the same Spring glass. With Java8, Spring 4.x, and many of today’s modern Java tooling, every other language seems underwhelming.

  3. Interesting read, though not getting the IDE part, still not getting it after reading the linked code smell blog post.

    My main benefit of an IDE (for Scala) is:

    – Syntax check, unused variable check etc. to prevent compiles for ‘syntax checking’/ not getting out of the zone

    – Jump to a class, IDE is clever enough to find the class based on fragements

    – Jump to a method from the code

    – Auto import

    – Debugger

    I wonder why with Go this is not needed. Faster compilation makes syntax checks/unused vars check in IDE not necessary? Less files? Does Go import everything? Larger import scopes? How do you find the implementation of a method? How is debugging working in Go?

    Is the compiler integrated into vim? The debugger? Thx.

    1. I do use the plugin vim-go, which is very nice. But one of the stated goals of Go was that the entire language could fit inside the hit of programmer. I find this to be true, because all I need is vim.

      I tend to use IDEs when first getting up to speed on a code base, but I found it close to difficult to write much Java without an IDE. It just wasn’t a pleasant experience.

  4. Have been using GO for about the same amount of time and I agree “productive” is the optimal word. I have found an editor (vim / atom / sublime text … ) with integrated lint and fmt to be a huge time saver and nice addition when programming with GO. My team has moved away from python in favor of GO for a huge part of our software with little or no productivity loss, and significant performance gain. Good Summary

  5. For all the folks who love a separate GOPATH per project, check out Dave Cheney’s excellent “gb” tool: http://getgb.io/ – it has quirks still, and may not be easy to use for a huge project you’ve already done “the go way”, but it makes project-based source code management amazing IMO.

    I’m the kind of person who loves having a Makefile on all my go projects so my sources can be how *I* want them, not necessarily how Go wants them. gb has eliminated most of my special case Makefile crap.

  6. Thanks to all for the feedback on this article. It’s certainly an effusive piece about Go, but I truly find it to be a great language that allows me to get stuff done — and still write decently-engineered software.

  7. Hi Philip,

    Thanks for your work in the community!

    Would you comment on how you handle 3rd party dependencies? Do you vendor with godep or the experimental plugin? I’m not sure the best approach to versioning between projects (esp w/ something like the AWS Go auto-generated packages).

    1. I take the easy way out — I don’t do anything, almost. ๐Ÿ˜‰ This is one reason to minimize using anything but the standard library.

      Of course, it’s not always possible. But I have not found it to an issue in practise. In one or two instances I have forked repos and use my own fork. Those projects didn’t change that much, so cherry-picking patches if needed has not been too much work.

  8. Hi Phillip,

    Thank you for your blog post. Although I’m new to Go, I also find it very productive for one of my current projects. I wonder what you use to debug Go code though? I currently use Jetbrains IntelliJ IDE with Goplugin but I fond it impossible to debug. Can you shed some light please?

    Thanks a lot,
    Vu

    1. Without a doubt, Go is weak when it comes to a debugger. I have used https://github.com/mailgun/godebug from time to time, and it works reasonably well.

      To be honest, I have found following Go philosophy of never ignoring errors (so what is happening in the program is always clear), and liberal use of println during actual debug, gets my code to the quality I need.

Leave a Reply

Your email address will not be published. Required fields are marked *