A study shows that around 2.5 quintillion bytes of data is generated every day. The sheer volume of data we’re collecting today is one of the most transformative things the world is witnessing. All this data is critical to the success of a business and so is the way organizations store it.
While it all started with a database, today companies widely use data warehouses and are gradually adopting data lakes for bigger data storage.
In this article, we will find out what a data lake is, how it is different from a database or data warehouse, and evaluate if your business could benefit from a data lake.
First, let’s understand what database, data warehouse, and data lake mean, and how they differ.
What's a database, data warehouse, and data lake?
To better understand these three different data storage types, let’s take the example of a confectionery company called Andy’s Candies, that manufactures and sells different types of sweets and candies. Below, we will try to understand a little bit about what each of these data storage options mean, and how Andy’s Candies would be able to use them.
Database
A database is a group of structured data that can be stored, accessed, or retrieved. In a database, Andy’s Candies would maintain three key tables or data sets: candies, customers, and sales. So, each newly launched candy is recorded on the ‘candies’ table, new customers go into the ‘customers’ table, and every sale is logged in the ‘sales’ table.
A good example of a database is SQL.
Data warehouse
It is a specialized form of database designed mainly for analysis and reporting purposes. For instance, a data warehouse could include a table showing all of Andy’s Candies’ buyers' regions or age groups in a given month. And they could use this data to analyze certain trends and use those insights to build up their marketing or sales strategies.
Why do we have a separate database and data warehouse?
The main purpose of a database is to keep track of the business. So, let’s say Andy’s Candies decides to eliminate the mint-flavored candy, then all data related to this candy is deleted from their database. And at a later point, if they would want to look at the volume of mint candies their company sold, it would become difficult to get that data since it no longer exists.
This is why companies have a separate data warehouse, where they can regularly archive it. So, data from the database is taken, redesigned, and moved into the data warehouse, which acts as a repository of each product's historical data. The companies can then access or retrieve any information related to a certain product at any given time.
Data lake
The data in a database or a data warehouse is stored in a table, which is made up of rows and columns. Each column consists of just one type of data and cannot help much if Andy’s Candies needed to store multiple types of data.
That’s where a data lake can help with storing different types of data together, such as tables, files, images, videos, etc. Simply put, a data lake is a centralized storage area for unstructured, raw, semi-structured, and structured data collected from multiple sources. According to James Dixon, founder and former CTO of Pentaho, who coined the term ‘data lake’ in October 2010,
“If you think of a Data Mart as a store of bottled water—cleansed and packaged and structured for easy consumption—the Data Lake is a large body of water in a more natural state. The contents of the Data Lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.”
How do I evaluate if my organization needs a data lake?
If your company stores big data, you may already be using a data warehouse. As a data engineer or a data administrator, you would be aware that data warehouses can store limited data and do not support every data type or high-speed data. However, these problems are less relevant if your company has smaller datasets, stores only one type of data, or runs low-speed data.
Data size, types, completeness, and speed are the four key factors to consider when evaluating if a data lake storage could help meet your business goals. While data lakes are popular today, sometimes an organization is better off with a database or a data warehouse. They are beneficial for certain purposes, but you also need to be aware of challenges in the data lake to get a complete picture.
Below are a few key factors that will help you evaluate when a data lake could be a good fit for your business and when you’re better off without it:
A data lake can benefit your organization if:
-
Your company produces copious amounts of data and rapidly generates new data
-
Your data configuration is frequently modified
-
Your total cost of hardware, software, and maintenance is eating up your IT budget
-
You are looking to eliminate data silos caused by disconnected datasets
-
You want to enable users with complete and flexible access to organizational data
-
Your users need to explore data in a flexible manner
-
Your analysts need to work with semi-structured and raw data
-
Real-time data is used in a variety of formats
-
Your company consumes fast data
-
You are looking at implementing different types of analytics workload. Ex. Machine learning
A data lake may not be able to help your organization if:
-
You have lower volumes of data
-
The pace of data production and modification in your organization is slow
-
You need strong control over data access by users
-
Your organization has no storage requirement for semi-structured or raw data
-
Most of your data analysis is performed by hired business experts
-
Your data streams at a low speed and only in one format. Ex. JSON files
-
Your data warehouse can manage the format in which your real-time data exists
How to leverage data lake for faster and superior analytics
A data lake is essentially a source of information in its rawest form. Businesses can filter through all of the data they have collected over the years and find helpful insights they didn't even realize were there! A data lake can help with, for instance, optimizing sales strategies or even streamlining a supply chain. But to tap into its true potential, you need to make sure you have the right cataloging systems in place so that you can be on top of things and keep everything in order.
Besides, the value a data lake can offer depends on how clean or pollution-free you keep it. Without proper care and management, all of your company’s data in the data lake could become susceptible to adulteration and other security and compliance threats.
Whether you use a database, data warehouse, or data lake, data governance is crucial in maintaining your data’s overall hygiene. You need access to quality data that has been prepared and modeled to be able to be used for various business purposes in the long run.
Looking to make the most of your organization’s data and achieve speed to business analytics? Explore our Data Analytics Solution for Dynamics 365 F&SCM factsheet to learn how it can help you derive data-driven insights to gain maximum business value.
Sumana Samuk
