Data fabrics and data meshes help businesses manage and analyze data more quickly and efficiently. They are two different approaches to data management that work to make more accurate data available to business users faster. Data fabrics are designed to break down information silos, while data meshes are structured to reduce bottlenecks in business data analysis procedures. Although both can be implemented in a single organization, they have different goals.
This guide discusses the benefits of data fabrics and data meshes, as well as potential pitfalls and barriers to implementing them.
What is a data fabric?
A data structure is a data management architecture that uses automated and intelligent systems to connect data stored in various places and in various formats. By pulling data from multiple storage sources and centralizing it, the data fabric enables teams to study collected data holistically, providing better insights.
A data fabric is designed to be flexible, standardize data management, analyze information and help teams make smarter business decisions. For example, a single company may store data in a database, a customer relationship management (CRM) system, and a network attached storage (NAS) array. Implementing a data fabric would allow teams to better understand all this data and avoid silos between the three systems.
Typically, a data fabric will include multiple solutions that work together. A full-featured fabric will integrate storage or data management solutions from more than one vendor; for example, you can use Talend’s Data Fabric platform to integrate data from your MongoDB database and the NetSuite platform. It could also extract information from data lakes, data stores and applications.
Data fabric platforms unify information from such disparate sources using application programming interfaces (APIs) and other integration technologies to pull data from applications like Google Drive, databases like Microsoft SQL Server, and data warehouses such as Amazon Redshift.
How does a Data Fabric work?
A full-featured data fabric includes the following features:
Support for multiple data storage formats. Ideally, this should include both structured and unstructured data, as most companies have both.
Data ingestion and integration capabilities. Raw data must be collected from all relevant applications and transported to a single location, and then all data must be integrated for analysis.
Multiple network paths. With data moving in so many different directions, it’s important to have more than one path for information to travel or it could bog down the network, creating more latencies.
Data fabrics use ingestion and integration to collect and analyze both structured and unstructured data. Ingestion technologies include extract, transform, and load (ETL) processes and SQL commands using ingestion tools such as Apache Kafka, Databricks, and Amazon Kinesis. Automatic ingestion is preferable to manual because it reduces potential human errors.
Data fabrics must also integrate the data or clean and analyze it all together once it has been ingested into a core location, such as a warehouse or a single lake. One of the key components of a data fabric is to eliminate silos. If your company has some customer data stored in SAP but other data residing in Salesforce, you may not have an accurate picture of customer demographics until all of this data is combined. Potential problems include duplicate data and inaccurate or outdated information.
Learn more about using data fabrics to drive data management.
What is a Data Fabric for?
Data fabrics are ideal for companies that store data in many different locations, especially large companies with multiple databases and other storage systems. Data fabrics can also benefit big data operations because they centralize large volumes of information. For data fabrics, data flexibility and agility is critical: to quickly analyze information from multiple sources, a data fabric must move data between storage systems efficiently.
A disadvantage of data fabrics is simply the effort required to set them up. It can take months to integrate all these storage solutions and establish data governance best practices so that the data being analyzed is high quality and accurate. This could be especially difficult for small businesses or organizations with small business intelligence or data teams.
What is a data grid?
A data mesh is a data management architecture that decentralizes data analysis from a single source so that it is readily available to multiple departments. A data mesh:
It focuses on data as a first-class product. Data must be well managed, protected and valued.
Classify data based on relevant business sector. This does not automatically mean that all HR data is separate from other business data, but that HR information is together.
Give the closest business user access to the data. CRM data should be readily available to sales teams, for example, while accounting data should be readily available to the finance department.
A company implementing a data mesh may have a single data lake for all structured and unstructured data, but categorize the metadata in a way that facilitates category searches. Data should also be periodically examined for accuracy and cleanliness, eg deduplication. Each team would have its own account within the company’s data management software, which it could use to search for relevant data.
Access to data includes analytics that help users understand information relevant to their work. For example, in a traditional enterprise data architecture, a marketing team would send a request for a dashboard to the business intelligence team, and the BI team would create the dashboard when the request appeared in the queue, but what if the marketing team needs the data immediately for a critical campaign?
Data meshes make data directly available to the right team so they can make decisions faster. Eliminating the bottleneck caused by having a single analytics team improves overall efficiency, eliminating some of the manual work, simplifying data analysis, and even increasing revenue. The ability to immediately act on data is critical to many sales, web and technology teams.
Data meshes also focus on data as a product. This just means that data is treated as a product, rather than a broad or vague concept. Data must be well managed, protected and valued, and easily accessible and usable.
A data mesh needs:
Clear government expectations. All teams need to know who is in charge of specific data in their department and how that person will manage it.
Access controls. For security reasons, only those who really need to view or edit the data should have access to it.
Quality guarantee. Data must be cleaned and organized so that it is useful and accurate for the teams that need it.
More information about the better data governance tools to manage large data sets.
How does a data grid work?
Unlike most storage technology, a data mesh is a general approach to business data availability rather than a specific hardware and software implementation. They will look different depending on the approaches of individual organizations.
First, all teams should have domain knowledge and data ownership. This takes time to teach and cultivate, but key team members should learn to read charts and graphs, understand what data is important, and know how to keep it clean and organized.
Teams should also have secure methods of accessing data. Examples include single sign-on (SSO) and multi-factor authentication (MFA). Companies must establish strict access controls so that only users who explicitly need data to do their jobs can see or edit it.
How can business users easily find data? In a data warehouse or database, where data is structured, querying should be easy and logical. In object stores and other unstructured data environments, metadata should be meaningful and easily searchable.
Learn more about security practices for stored data.
What is a data grid for?
Data meshes are convenient for all business departments and teams because they remove existing bottlenecks to important information. Teams that can benefit from a data mesh architecture include:
Sales Marketing Publishing Search Engine Optimization Paid Media Engineering Product Social Media Information Technology
Although teams perform different functions within a company, most need accurate and organized data to make decisions. Because data meshes approach information as a first-class product, they recognize the importance of data to business operations. Data is no longer an afterthought in the business world, but a priority.
A potential drawback of data mesh architectures is data security. If multiple teams have access to company data, this can be dangerous for security protocols and compliance. The more people who can handle sensitive information, the greater the risk of a security breach. While methods such as strict identity and access management can protect data, it still presents a downside for businesses, which can be mitigated but will take time to navigate.
Data meshes are also challenging because they require each team to have members who can manage the data and its associated technology. For example, each team may need someone who can create dashboards and who knows how to use data cleansing tools. For this reason, data meshes can be challenging for most small and medium-sized businesses to implement successfully, simply because they don’t have enough employees.
Conclusion: What do you need to know?
Both data fabrics and data meshes are useful data architectures for businesses. Organizations may use both, but should determine when to centralize data (a fabric) and when to distribute it across different teams (a mesh). Each approach is beneficial, but requires careful planning and data protection methods. Is your business considering a data management solution? Read more about the best data management platforms below.
[ad_2]
Source link
