Data Catalogs aid the users to identify appropriate data from multiple data stores.
TL;DR
In the previous article, you understood the concept of a Data Mesh. In this article you will learn that one of the two keys to a successful Data Mesh architecture is an accurate data catalog. Creating, maintaining and validating the catalog is essential to the users finding out where they can find that data. The Catalog can be maintained when the data is acquired by the system, or land in a raw zone and then the updated in the catalog later. …
TLDR;
While the term is relatively new, Data Mesh as a concept existed for decades. In this article, narrated in a fable format, you will learn how the data systems evolved, how distributed data systems gave rise to Data Mesh and why we need them.
Background
It was an early in the morning late winter morning. Debby, the chief data systems architect at Acme Widgets was studying half-mindedly the swirling steam coming up her cup of joe and contemplating how to attack the day looking at her calendar. She has been around the block for almost two and half decades, has seen…
Easily control asynchronous execution of individual functions inside a program instead of parallelizing the processing
Preamble
“Well, can’t you throw a bunch of additional servers at the problem, make the process run in parallel and make it faster?” asks an exasperated Tom, the Chief Technology Officer at Acme Widgets.
Acme’s data management team has developed multiple data pipelines to transport, transform and deliver data to various systems for consumption. The pipelines read data from various sources — such as kafka streams, queues, disks and even a network socket — and execute important computations on the data before sending them to the destination systems. While the timing for computation is rather predictable, the timing of the arrival…
Choosing the right pattern for the right case
Preamble
“Hard to believe it’s not possible,” implored a confused Ted. “This is 2020; surely, there must be another way”.
It was a teaching moment for Acme Widgets. A database in their technology stack failed and brought down a lot of their services. Ted, the Chief Technology Officer, called an emergency meeting of the senior technology leaders to chart out a course of action for tactical and long term solutions. One of the solutions he thought of was to convert all databases to active-active with no shared assets amongst them, spread geographically to reduce risks; but with no application changes. All…
Preamble
“Hard to believe it’s not possible,” implored a confused Ted. “This is 2020; surely, there must be another way”.
It was a teaching moment for Acme Widgets. A database in their technology stack failed and brought down a lot of their services. Ted, the Chief Technology Officer, called an emergency meeting of the senior technology leaders to chart out a course of action for tactical and long term solutions. One of the solutions he thought of was to convert all databases to active-active with no shared assets amongst them, spread geographically to reduce risks; but with no application changes. All…
Defining clearly what smartness is when hiring for smart people.
I have been hiring people in a profession that demands intellectual capital; not just bodies. I am often asked how high (or, low, depending on how you see it) bar I set for the “smartness” in people. To set a bar, I first need to define what it is and I usually get rolling eyes when I tell them the answer.
So, what exactly is meant by being “smart”?
As it turns out it’s hard to put a precise definition to the term. …
How do you enable parallel processing in python? Threading is one way, but building a fully resilient multi-threaded process is hard. Incorporating asynchronous processing in python is a much simpler and practical way to introduce parallel processing in your applications. In this very short blog post, you will learn how to enable simple async functionalities.
The “Normal” Way
Let’s examine the standard way a program is written. Consider a process defined in a function called myproc() that does something that takes 5 seconds. We will simulate it by putting a sleep of 5 seconds. When the function call is made, we print a…
Learn how to use the API approach to enable full duplex data transfer between client and server asynchronously using WebSockets which is an upgrade from HTTP, with a working Python code.
Let’s Catch Up on APIs
The typical data integration pipeline is generally one way, i.e. a client such as a mobile application makes a call to the RESTful API running on a server somewhere. The API does all the work and sends the results back to the client. If the client needs the same data again, it has to call the API once again. If it needs the data continuously, e.g. …
