Preamble

“Well, can’t you throw a bunch of additional servers at the problem, make the process run in parallel and make it faster?” asks an exasperated Tom, the Chief Technology Officer at Acme Widgets.

Acme’s data management team has developed multiple data pipelines to transport, transform and deliver data to various systems for consumption. The pipelines read data from various sources — such as kafka streams, queues, disks and even a network socket — and execute important computations on the data before sending them to the destination systems. While the timing for computation is rather predictable, the timing of the arrival…

Preamble

“Hard to believe it’s not possible,” implored a confused Ted. “This is 2020; surely, there must be another way”.

It was a teaching moment for Acme Widgets. A database in their technology stack failed and brought down a lot of their services. Ted, the Chief Technology Officer, called an emergency meeting of the senior technology leaders to chart out a course of action for tactical and long term solutions. One of the solutions he thought of was to convert all databases to active-active with no shared assets amongst them, spread geographically to reduce risks; but with no application changes. All…

Preamble

“Hard to believe it’s not possible,” implored a confused Ted. “This is 2020; surely, there must be another way”.

It was a teaching moment for Acme Widgets. A database in their technology stack failed and brought down a lot of their services. Ted, the Chief Technology Officer, called an emergency meeting of the senior technology leaders to chart out a course of action for tactical and long term solutions. One of the solutions he thought of was to convert all databases to active-active with no shared assets amongst them, spread geographically to reduce risks; but with no application changes. All…

Defining clearly what smartness is when hiring for smart people.

I have been hiring people in a profession that demands intellectual capital; not just bodies. I am often asked how high (or, low, depending on how you see it) bar I set for the “smartness” in people. To set a bar, I first need to define what it is and I usually get rolling eyes when I tell them the answer.

So, what exactly is meant by being “smart”?

As it turns out it’s hard to put a precise definition to the term. …

How do you enable parallel processing in python? Threading is one way, but building a fully resilient multi-threaded process is hard. Incorporating asynchronous processing in python is a much simpler and practical way to introduce parallel processing in your applications. In this very short blog post, you will learn how to enable simple async functionalities.

The “Normal” Way

Let’s examine the standard way a program is written. Consider a process defined in a function called myproc() that does something that takes 5 seconds. We will simulate it by putting a sleep of 5 seconds. When the function call is made, we print a…

Learn how to use the API approach to enable full duplex data transfer between client and server asynchronously using WebSockets which is an upgrade from HTTP, with a working Python code.

Let’s Catch Up on APIs

The typical data integration pipeline is generally one way, i.e. a client such as a mobile application makes a call to the RESTful API running on a server somewhere. The API does all the work and sends the results back to the client. If the client needs the same data again, it has to call the API once again. If it needs the data continuously, e.g. …