Easily control asynchronous execution of individual functions inside a program instead of parallelizing the processing

clip art of man and woman in suits walking forward along teal wall with white arrows
clip art of man and woman in suits walking forward along teal wall with white arrows

Preamble

“Well, can’t you throw a bunch of additional servers at the problem, make the process run in parallel and make it faster?” asks an exasperated Tom, the Chief Technology Officer at Acme Widgets.

Acme’s data management team has developed multiple data pipelines to transport, transform and deliver data to various systems for consumption. The pipelines read data from various sources — such as kafka streams, queues, disks and even a network socket — and execute important computations on the data before sending them to the destination systems. While the timing for computation is rather predictable, the timing of the arrival…

Choosing the right pattern for the right case

image in shade of blue of 2 men on opposite cliffs with ladders looking at each other
image in shade of blue of 2 men on opposite cliffs with ladders looking at each other

Preamble

“Hard to believe it’s not possible,” implored a confused Ted. “This is 2020; surely, there must be another way”.

It was a teaching moment for Acme Widgets. A database in their technology stack failed and brought down a lot of their services. Ted, the Chief Technology Officer, called an emergency meeting of the senior technology leaders to chart out a course of action for tactical and long term solutions. One of the solutions he thought of was to convert all databases to active-active with no shared assets amongst them, spread geographically to reduce risks; but with no application changes. All…

Image for post
Image for post

Preamble

“Hard to believe it’s not possible,” implored a confused Ted. “This is 2020; surely, there must be another way”.

It was a teaching moment for Acme Widgets. A database in their technology stack failed and brought down a lot of their services. Ted, the Chief Technology Officer, called an emergency meeting of the senior technology leaders to chart out a course of action for tactical and long term solutions. One of the solutions he thought of was to convert all databases to active-active with no shared assets amongst them, spread geographically to reduce risks; but with no application changes. All…

Defining clearly what smartness is when hiring for smart people.

Image for post
Image for post
<a href=’https://www.freepik.com/vectors/background'>Background vector created by iconicbestiary — www.freepik.com</a>

I have been hiring people in a profession that demands intellectual capital; not just bodies. I am often asked how high (or, low, depending on how you see it) bar I set for the “smartness” in people. To set a bar, I first need to define what it is and I usually get rolling eyes when I tell them the answer.

So, what exactly is meant by being “smart”?

As it turns out it’s hard to put a precise definition to the term. …

Image for post
Image for post
<a href=’https://www.freepik.com/photos/background'>Background photo created by kjpargeter — www.freepik.com</a>

How do you enable parallel processing in python? Threading is one way, but building a fully resilient multi-threaded process is hard. Incorporating asynchronous processing in python is a much simpler and practical way to introduce parallel processing in your applications. In this very short blog post, you will learn how to enable simple async functionalities.

The “Normal” Way

Let’s examine the standard way a program is written. Consider a process defined in a function called myproc() that does something that takes 5 seconds. We will simulate it by putting a sleep of 5 seconds. When the function call is made, we print a…

Learn how to use the API approach to enable full duplex data transfer between client and server asynchronously using WebSockets which is an upgrade from HTTP, with a working Python code.

<a href=”http://www.freepik.com">Designed by Freepik</a>
<a href=”http://www.freepik.com">Designed by Freepik</a>
Image Courtesy: <a href=”http://www.freepik.com">Freepik</a>

Let’s Catch Up on APIs

The typical data integration pipeline is generally one way, i.e. a client such as a mobile application makes a call to the RESTful API running on a server somewhere. The API does all the work and sends the results back to the client. If the client needs the same data again, it has to call the API once again. If it needs the data continuously, e.g. …

Arup Nanda

Award winning data management and engineering leader, big data and processing enthusiast, raspberry pi junkie, dad and husband — not necessarily in that order.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store