We know big data refers to the massive amounts of structured, semi-structured, and unstructured data being generated today. But in a landscape marked by huge and complex data sets, it’s time to dig deeper into what big data actually is and how to manage it.
Below, we cover the defining characteristics of big data, or the 5 V’s.
The challenges of big data
There are several challenges associated with managing, analyzing, and leveraging big data, but the most common roadblocks include:
The need for large-scale, elastic infrastructure (e.g., cloud computing, distributed architecture, parallel processing).
The need to integrate data from various sources and in various formats (e.g., structured and unstructured).
A crowded and interconnected tech stack that creates data silos.
The preservation of data integrity, including keeping it up-to-date, clean, complete, and without duplication.
Ensuring privacy compliance and data security.
Big data characteristics – The 5 V’s
Big data is often defined by the 5 V’s: volume, velocity, variety, veracity, and value. Each characteristic will play a part in how data is processed and managed, which we explore in more detail below.
Volume
Volume refers to the amount of data being generated (at a minimum, many terabytes but also as much as petabytes).
Because of the staggering amount of data available today, it can create a significant resource burden on organizations. Storing, cleaning, processing, and transforming data requires time, bandwidth, and money.
For data engineers, this increased volume will have them thinking about scalable data architectures and appropriate storage solutions, along with how to handle temporary data spikes (like what an e-commerce company might experience during holiday sales).
Velocity
The word velocity means “speed,” and in this context, the speed at which data is being generated and processed. Real-time data processing plays an important role in this regard, as it processes data as it’s generated for instantaneous (or near instantaneous) insight. Weather alerts, GPS tracking, sensors, and stock prices are all examples of real-time data at work. Of course, when working with huge datasets, not everything should be processed in real time. This is one of the considerations an organization would have to think through, what should be processed in real time vs. batch processing?
Distributed computing frameworks and streaming processing frameworks like Apache Kafka or Apache Flink have become useful in managing data velocity.
Variety
Data diversity is another attribute of big data, encompassing structured, unstructured, and semi-structured data (e.g., social media feeds, images, audio, shipping addresses). Organizations will need to map out:
How they plan to integrate these various different data types (e.g., ETL or ELT pipelines).
Schema flexibility (e.g., NoSQL databases).
Data lineage and metadata management.
How data will be made accessible to the larger organization via business reports, data visualizations, etc.
Veracity
For all the effort that goes into data collection, processing, and storage, if there are any inconsistencies or errors (like data duplicates, missing data, or high latencies) then data’s usefulness quickly erodes.
Veracity refers to the accuracy, reliability, and cleanliness of these large data sets. Ensuring data veracity comes down to good data governance, and implementing best practices like:
Automating QA checks and flagging data violations in real time
Adhering to a single tracking plan
Standardizing naming conventions
Data Tracking Plan Template
A data tracking plan helps businesses clarify what events they’re tracking, how they’re tracking them, and why. Use this template to help create your own tracking plan.
Value
True to its name, Value refers to the actionable insight that can be derived from big data sets. While it might seem like huge amounts of data should automatically lead to greater insight, without the proper processing, validation, and analytics frameworks in place, it will be extremely difficult to derive value. (Hence the need for the four previous V’s.)
This is where artificial intelligence and machine learning can come in, to help extract learnings and action items at a rapid rate (e.g., predictive analytics or prescriptive analytics).
Another key aspect to making data valuable is to make it accessible across teams, like with self-service analytics.
The right tools for harnessing big data will depend on your business, but might include the following:
The ability to collect various types of data from different sources using batch processing, event-streaming architecture, ETL or ELT pipelines, and more. A couple popular tools include Amazon Kinesis or Apache Kafka.
Scalable storage destinations(e.g., cloud-based data lakes or data warehouses).
Data transformation and validation tools
Analytics tools like Looker or Power BI to visualize and report on data.
AI and ML tools to build and train machine learning algorithms.
Data security tools to ensure encryption, privacy compliance, and access controls.
The role of customer data platforms in managing big data
Segment helps manage big data by providing a scalable infrastructure. It processes 400,000 events per second, is able to deduplicate data, and its Go servers have “six nines” of availability.
It offers over 450 pre-built integrations with various different sources and destinations (including storage systems like Amazon S3, Redshift, Snowflake, Postgres, and more).
Segment is also able to validate customer data at scale, by automatically running QA checks, and flagging any data that doesn’t fit a predefined naming convention or tracking plan. This allows teams to proactively block bad data and understand the root cause of an issue before it impacts reporting.
Segment can then unify this data into real-time customer profiles, and sync these profiles to the data warehouse so they’re enriched with historical data.
On top of that, Segment’s Privacy Portal helps ensure compliance with fast-changing regulations (offering encryption at rest and in transit, automatic risk-based data classification, and data masking).
