Real-time data processing is an essential capability for nearly every business and organization. It underlies services such as identity management, fraud prevention, financial transactions, recommendation engines, customer relationship management, and social media monitoring. It is also the foundation of predictive analysis, artificial intelligence (AI), and machine learning (ML).
Real-time Data Scaling Challenges
The challenge for many organizations is to scale real-time resources in a manner that reduces costs while increasing revenue. Several factors make such scaling difficult:
On-Premises Requirements for Sensitive Data
One approach to consider is to migrate data to the public cloud. The cloud is appealing because it reduces capital spend in exchange for operating spend that is flexible based on a company’s dynamic requirements. The cloud also supports fast scaling.
However, data transfer fees can add up fast, and not all data is appropriate for the cloud. To comply with government regulations and/or internal security policies, organizations may find it necessary to secure sensitive data on-premises. Similarly, a company may decide to keep its most critical data – everything from financial records to engineering files – local where it can protect this data best.
Thus, teams need to able to store, process, and manage real-time data in their own data centers. They need a solution that reduces costs, simplifies management and scales quickly. And they need to be able to transform this data into revenue faster than the competition.
Building a Scalable, Cost-Effective Environment
These four tips can help create a scalable, cost-effective environment for processing data on-premises or at the edge.
- Integrate a NoSQL database with Kafka and Spark: For organizations with a database more than 5TB and the need to process a high volume of data in real-time, consider deploying a NoSQL database alongside other real-time tools like Kafka and Spark.
- Match your server components to your use case: For the software supporting your database to achieve the best real-time performance at scale, you need the right server hardware as well. At scale, server memory (DRAM) is expensive and consumes increasing power. It also requires hard drives to provide reliable long-term storage. New server persistent memory (PMem) options are available that match the speed of DRAM but are less expensive and retain data during a power interruption.
- Scale up and scale out: Typically, systems are designed to either scale up (e.g., add more resources to an existing server or node) or scale out (e.g., increase the number of servers or nodes). Ideally, real-time data processing requires a database, hardware and software solution that can both scale up and scale out.
- Use smart data distribution to reduce latency while increasing resiliency: As processing clusters grow, it’s important to avoid “hot spots.” Hot spots arise when a portion of a cluster is required/used more frequently than other resources. This leads to bottlenecks and overall cluster performance degradation. Technology such as load-balancing ensures that all resources in a cluster are doing approximately the same amount of work. Spreading the load in this manner reduces latency and eliminates bottlenecks. Smart distribution also enables the creation of clusters that span multiple data centers, increasing resiliency.
Real-World Results for Real-time Data
Dell Technologies has worked with Aerospike to accelerate processing of real-time data. Aerospike provides solutions that eliminate tradeoffs between high performance, scale, consistency, and low total cost of operations.
For example, Aerospike enables the use of flash storage in parallel to perform reads with sub-millisecond latency. This supports the very high throughput (100K to 1M) necessary for heavy-write loads during real-time processing. Using a hybrid memory architecture with a purely in-memory index, Aerospike can achieve vertical scaleup at 5X lower total cost of ownership compared to a pure server random access memory (RAM) implementation. Thus, the storage architecture can be optimized for performance and scale.
In addition, Aerospike’s “shared nothing” architecture supports algorithmic cluster management combined with global cross-data center replication to support complex filtering, dynamic routing, and self-healing capabilities. This enables systems to quickly recover from adverse events while maintaining performance, making it ideal for mission-critical real-time data processing.
Large-scale organizations deploying efficient real-time data processing to deliver tremendous results include:
- PayPal: Real-time digital payment fraud prevention – 30x reduction in false positives
- Charles Schwab: Reduced intraday trading risk at hyperscale – down from 150 servers to 12
- LexisNexis: Securing global digital identities at scale – latency reduced from 100 milliseconds to 30 milliseconds
- Wayfair: Hyper-personalized recommendations – 1/8 server footprint.
Real-time data processing is only going to become more essential for businesses over time. With the right technology, businesses can overcome today’s real-time data challenges to improve their overall agility, efficiency, and profitability. And by investing in hardware and software solutions that work together to provide optimal performance, real-time data processing environments will continue to scale up and scale out for years to come.
For a detailed look at how the right technology can help turn your organization’s real-time data into revenue, check out the 4 Tips for Processing Real-Time Data paper and watch the webinar.
***
Intel® Technologies Move Analytics Forward
Data analytics is the key to unlocking the most value you can extract from data across your organization. To create a productive, cost-effective analytics strategy that gets results, you need high performance hardware that’s optimized to work with the software you use.
Modern data analytics spans a range of technologies, from dedicated analytics platforms and databases to deep learning and artificial intelligence (AI). Just starting out with analytics? Ready to evolve your analytics strategy or improve your data quality? There’s always room to grow, and Intel is ready to help. With a deep ecosystem of analytics technologies and partners, Intel accelerates the efforts of data scientists, analysts, and developers in every industry. Find out more about Intel advanced analytics.
