Tag Archives: linux

Drop, Throttle, or Buffer

Real-time — or near real-time — data pipelines are all the rage these days.  I’ve built one myself, and they are becoming key components of many SaaS platforms. SaaS Analytics, Operations, and Business Intelligence systems often involve moving large amounts of data, received over the public Internet, into complex backend systems. And managing the incoming flow of data to these pipelines is key.

Continue reading Drop, Throttle, or Buffer

Monitoring Storm Kafka Spouts using Python

kafka-logo-tallWhen running a large real-time processing system, monitoring is critical. But it does more than allow you to keep an eye on your system. During development it allows you test hypotheses about how it works, how it performs when certain parameters are changed, and takes the guessing out of working with dynamic systems.

Storm, a real-time computational framework open-sourced by Twitter, is such a system and comes with a Spout, allowing messages to be streamed from a Kafka Broker.

Continue reading Monitoring Storm Kafka Spouts using Python

My guidelines for reusable Django applications

djangoI came to Django development from much lower-level development — embedded software, device drivers, and system software. What has impressed me most about Django (and python in general) is the manner in which it guides you to do the right thing in terms of code construction. The framework and language naturally make you think about better ways to express your designs.

Continue reading My guidelines for reusable Django applications

Installing Fedora Core 12 on Chembook 2370VA

Fedora_verticalAfter a good experience with Fedora Core 8, and a reasonable experience with Fedora Core 11, I decided to install Fedora Core 12 on my Chembook laptop.

In summary, while FC8 and FC11 worked out of the box, FC12 failed to provide me with sound. I discovered later that the KDE mixer had set the center volume to mute. Once I unset that, I had sound.

Other than that, it worked pretty well.

Continue reading Installing Fedora Core 12 on Chembook 2370VA

Valgrind, blessed be its name

Valgrind comprises a bunch of very useful tools for detecting problems with your programs. I first came across it a couple of years back and find it to be excellent. In particular I use its memory profiler, which helps you catch errors such as memory leaks and invalid accesses. In my experience these types of errors sometimes indicate logic errors, not just areas where you’ve forgotten to free some previously allocated memory — which is another reason why it is such a great tool.

Continue reading Valgrind, blessed be its name

Yellow Dog Linux 6.1 on the PS3

Yellow Dog Linux on the PS3
Yellow Dog Linux on the PS3

I got around to installing Yellow Dog 6.1 using a DVD of the full distro. The installation went OK, and the installer fired up in graphical mode. However it proceeded to create the swap partition almost immediately because of low-memory concerns.

When it completed YDL was quite zippy – a much, much better experience than I got from FC12. I even had audio.

I may actually use this – it depends if I can get particular media players running on it.