Recent Posts by Michael Chen

Apache Spark is the dominant force in big data, seamlessly combining scalable data processing, reporting, and machine learning in a convenient package. How widespread is Apache Spark usage? Consider this: since launching a decade ago as an open-source project at UC Berkeley, Apache Spark has powered AI and big data for some of the world’s largest tech companies, and in doing so, has changed the way the world consumes data. From driving user recommendations to...

It’s no secret that big data is getting much bigger with each passing year—in fact, the world is seeing exponential growth in the amount of data generated, as plenty of research shows. That creates the issue of storage. If all those bits and bytes are being transmitted and you need access to them in order to  analyze and derive insights via business intelligence, then the next logical step is a data lake. But what happens when all of that data is sitting in the data lake?...

It's common knowledge in the modern business world that big data —that is, the large volumes of data collected for analysis by organizations—is a significant element of any strategy—operations, sales, marketing, finance, HR, and every other department rely on big data solutions to stay ahead of the competition. But how that big data is handled remains another story. Enter the world of data lakes. Data lakes are repositories that can take in data from multiple sources. Rather...

For most businesses, getting the most out of data means assembling the right tools needed for the development of deep insights. But what happens when your entire business is about doing that for others? Suddenly, your data isn’t just your own internal dataset, but every client’s dataset too. Data management suddenly becomes much more complicated. The team at DX Marketing (DXM), an award-winning insights company with offices in Greenville, South Carolina, and Savannah,...

From a technical perspective, the term "graph analytics" means using a graph format to perform analysis of relationships between data based on strength and direction. That might be a bit hard to understand for the uninitiated, particularly when the traditional idea of data analysis brings up images of poring over spreadsheets—very big spreadsheets, when you’re looking at big data with petabytes or exabytes of data. So let's break down that definition piece by piece to offer...

Big data is rapidly becoming the key driver in the financial services industry. Big data covers a lot of areas: transactions, customer accounts, vendors, and more. All include individual fields of data, from time stamps to payment amounts to unstructured text fields of additional data (such as call center notes). Consider these numbers: the volume of digital banking users has increased from 20% in 2010 to 61% in 2018—more than tripling in a number of years. At the same time,...

Machine learning (ML) represents one of the keys to thriving in the modern age of big data. Consider “just how big” big data is for most organizations. Even small and medium-sized businesses (SMBs) deal with multiple data sources such as CRM tools, social media feeds, and transaction platforms. That's not even including internal data from HR platforms, submission forms, emails and media, and cloud-based tools that transmit metrics. All of that data means that unprecedented...

What is the difference between structured and unstructured data—and should you care? For many businesses and organizations, such distinctions may feel like they belong solely to the IT department dealing with big data. And while there is some truth to that, it’s worthwhile for everyone to understand the difference, because once you grasp the definition of structured data and unstructured data (along with where that data lives and how to process it), it’s possible to see how...

What converts a lead into a sale? Or in the case of a subscription service, what converts an existing customer into a renewing customer? That was the dilemma that San Francisco-based Riverbed Technology was facing not too long ago. Riverbed’s primary service is delivering performance management and analysis for networks and applications. Hardware monitoring comes via embedded solutions that track key metrics as defined by customers. From a customer side, that volume of big...