Every technological revolution brings with it a wave of skepticism, and real-time analytics is no exception. This blog addresses those who doubt the necessity and value of real-time analytics.

UNDERSTANDING REAL-TIME ANALYTICS

What is Real-Time Analytics?

While this question was relevant a decade ago, but asking it today is similar to asking any tech or industry leader, “Is GenAI making an impact?” Real-time analytics is more prevalent than most can image. So let’s give concrete examples for the skeptics to showcase its ubiquity.

• Ridesharing: When you check the estimated time of arrival for your Uber, that’s real-time analytics at work.
• Social Media: The likes and impressions on your posts are updated in real-time.
• Stock Market: Monitoring market trends and volume changes in real-time.
• Sports: Viewing live stats during a game involves real-time data processing.

These examples illustrate that real-time analytics is an integral part of our daily lives.

Defining Real-Time Analytics

Real-time analytics involves translating events or data into insights within milliseconds to few seconds. The time it takes to derive insights from events determines its classification as real-time. If the process takes longer, it’s considered batch processing. The value of an event peaks when it occurs and diminishes rapidly over time, making immediate analysis crucial.

THE NECESSITY OF REAL-TIME ANALYTICS

Is Real-Time Analytics a Must-Have?

You might wonder whether real-time analytics is essential or merely a nice-to-have feature. This reminds me to the data center days early in my career, where we had to generate reports by 5:00 AM pacific before the stock marketing open in the east coast. The process involved starting data processing after 6PM pacific, requiring no change in the system and complete utilization of resources. If we contrast that to today’s world of real-time processing. It allows for instant report generation, saving not only time and resources, but also brings in the element of freshness.

The rise of real-time analytics is comparable to the adoption of streaming technologies like Kafka. Although, initially it was met with skepticism, streaming has now become ubiquitous with thousands of companies leveraging it for various applications. The percentage of data generated in real-time has grown exponentially, driven by the demand for instant information in our on-demand economy, such as Linkedin, Uber, Doordash etc.

Why the Shift?

The primary drivers of this shift are evolving consumer expectations and technological advancements. Consumers now expect immediate access to information, compelling businesses to adopt real-time processing. Businesses must embrace event-driven architecture and real-time OLAP to meet these demands and stay competitive.

REAL-WORLD IMPACT AND COST EFFICIENCY

Business Benefits

Adopting real-time analytics offers tangible business benefits. For instance, Uber Freight’s real-time scoring system for drivers reduced cancellations by 4% and improved on-time drop-offs by 1%, resulting in significant cost savings.

Numerous companies across various industries have already embraced real-time analytics, demonstrating its widespread applicability and value.

Cost Considerations

A common concern among skeptics is the perceived high cost of implementing real-time analytics. While it was expensive 15 years ago, technological advancements have drastically reduced these costs. Technologies like Kafka and Apache Pinot have made logging and querying data more affordable and efficient.

For example, LinkedIn’s move from traditional technologies to Apache Pinot reduced the number of nodes needed from 1,000 to 75, while traffic grew fivefold. This reduction in infrastructure costs, combined with improved query performance, highlights the economic viability of real-time analytics.

OVERCOMING COMPLEXITY

Is Real-Time Analytics Complicated?

Transitioning to real-time analytics can be challenging due to the learning curve associated with new technologies and concepts. However, advancements in tools and platforms are simplifying the process, making it more accessible to businesses of all sizes.

In the batch processing world, traditional architectures involved capturing daily snapshots and processing them in data warehouses. The shift to real-time processing introduced complexities like offsets, checkpoints, and transactions, which were unfamiliar to many developers. Despite these challenges, the benefits of real-time analytics outweigh the initial hurdles.

CONCLUSION

Real-time analytics is no longer a luxury but a necessity in today’s fast-paced, data-driven world. The skepticism surrounding it is natural but increasingly unfounded as more businesses demonstrate its value. By embracing real-time analytics, companies can gain a competitive edge, improve operational efficiency, and meet the ever-growing expectations of consumers. As we move forward, the question is not whether to adopt real-time analytics but how quickly we can do so to stay ahead of the curve.

NOTE: This writeup was inspired by the Keynote given Kishore Gopalakrishna, CEO & Cofounder of StarTree, “Mission Critical Data Can’t Wait” at RTA Summit ‘24.