Real-time — or near real-time — data pipelines are all the rage these days. I’ve built one myself, and they are becoming key components of many SaaS platforms. SaaS Analytics, Operations, and Business Intelligence systems often involve moving large amounts of data, received over the public Internet, into complex backend systems. And managing the incoming flow of data to these pipelines is key.
Tag Archives: design
How you should write software design documents
In my last blog post I explained why writing design documents is such a powerful approach to building well-engineered systems. But what should one document?
Continue reading How you should write software design documents
Why you should write software design documents
Many software engineers never write design documents. Design documentation takes time, and implementations often proceed so far without any documentation that if it happens, it’s an act of recording what has been done — a tedious task at the best times.
Many software engineers argue “the code exists, it’s running, it’s working, let’s move on and build the next thing.”
Continue reading Why you should write software design documents