Data Catalogs aid the users to identify appropriate data from multiple data stores.

Photo by Aleksi Tappura on Unsplash

TL;DR

In the previous article, you understood the concept of a Data Mesh. In this article you will learn that one of the two keys to a successful Data Mesh architecture is an accurate data catalog. Creating, maintaining and validating the catalog is essential to the users finding out where they…


<a href=’https://www.freepik.com/vectors/technology'>Technology vector created by macrovector — www.freepik.com</a>

TLDR;

While the term is relatively new, Data Mesh as a concept existed for decades. In this article, narrated in a fable format, you will learn how the data systems evolved, how distributed data systems gave rise to Data Mesh and why we need them.

Background

It was an early in the…


Easily control asynchronous execution of individual functions inside a program instead of parallelizing the processing

clip art of man and woman in suits walking forward along teal wall with white arrows

Preamble

“Well, can’t you throw a bunch of additional servers at the problem, make the process run in parallel and make it faster?” asks an exasperated Tom, the Chief Technology Officer at Acme Widgets.

Acme’s data management team has developed multiple data pipelines to transport, transform and deliver data to various…


Choosing the right pattern for the right case

image in shade of blue of 2 men on opposite cliffs with ladders looking at each other

Preamble

“Hard to believe it’s not possible,” implored a confused Ted. “This is 2020; surely, there must be another way”.

It was a teaching moment for Acme Widgets. A database in their technology stack failed and brought down a lot of their services. Ted, the Chief Technology Officer, called an emergency…


Preamble

“Hard to believe it’s not possible,” implored a confused Ted. “This is 2020; surely, there must be another way”.

It was a teaching moment for Acme Widgets. A database in their technology stack failed and brought down a lot of their services. Ted, the Chief Technology Officer, called an emergency…


Defining clearly what smartness is when hiring for smart people.

<a href=’https://www.freepik.com/vectors/background'>Background vector created by iconicbestiary — www.freepik.com</a>

I have been hiring people in a profession that demands intellectual capital; not just bodies. I am often asked how high (or, low, depending on how you see it) bar I set for the “smartness” in people. …


<a href=’https://www.freepik.com/photos/background'>Background photo created by kjpargeter — www.freepik.com</a>

How do you enable parallel processing in python? Threading is one way, but building a fully resilient multi-threaded process is hard. Incorporating asynchronous processing in python is a much simpler and practical way to introduce parallel processing in your applications. …


Learn how to use the API approach to enable full duplex data transfer between client and server asynchronously using WebSockets which is an upgrade from HTTP, with a working Python code.

<a href=”http://www.freepik.com">Designed by Freepik</a>
Image Courtesy: <a href=”http://www.freepik.com">Freepik</a>

Let’s Catch Up on APIs

The typical data integration pipeline is generally one way, i.e. a client such as a mobile application makes a…

Arup Nanda

Award winning data management and engineering leader, Chief Architect of Cloud Data at JPMorganChase, raspberry pi junkie, dad and husband.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store