Digital Edition

SYS-CON.TV
Big Data – How We Got Here and Is It For Everyone? | @BigDataExpo
An organization needs to be data driven in order to understand the data’s value and lament its absence

Big Data is an umbrella term for a multitude of new capabilities that are being used for storage and computing operations at scale. These capabilities allow organizations to store massive amounts of data, in disparate formats, and perform both batch and real-time analyses upon them.

The forces driving Big Data into the mainstream are the ever-decreasing cost of storage and processing, coupled with the open source enhancements of distributed systems techniques and software. Companies have realized that data storage is on the verge of being limitless, and they no longer need to be as judicious about what kinds of data they store. This realization has led to the storage of all manner of data, in addition to the traditional structured data found in relational databases.

Sometimes unstructured or semi-structured, this type of data encompasses emails, social media feeds, clickstreams, sensor data, videos and more. Further, the questions companies can ask of their data to realize value have become more complex. But the time window for analysis completion has remained the same or shrunk due to the massively parallel computation Big Data systems provide.

To organize all this limitless data - structured and unstructured - new tools have emerged. We no longer have just one hammer in our toolkit - the relational database - with which to fashion data. There are now a myriad of systems, thanks to big organizations, many of them Internet giants (Google, Yahoo!, Amazon, Facebook, and LinkedIn to name a few). Out of a need to scale storage and compute tasks confronting them, they made new data systems to satisfy these specific use cases. The most common characteristics of these systems are that they are not row based or relational databases. The systems have horizontal linear scalability (just add more nodes), and they expect components and nodes to fail so they are fault tolerant.

Most of these tools have been open sourced and, years later, are being adopted by other businesses that recognize their value. Examples of these non-relational databases are key-value pair databases (e.g., Riak), document databases (e.g., CouchDB, MongoDB), columnar databases (e.g., HBase, Cassandra), graph databases (e.g., Titan, Neo4J), distributed queues (e.g., Kafka, Kestrel) and spatial databases. There are also new computing tools such as Hadoop and Spark that allow immense amounts of data to be processed, and Storm, Samza and Spark Streaming which analyze data in near real-time, something that was previously only possible with supercomputers.

The ability to store unlimited amounts of disparate data in order to perform endless analysis in batch and real time is the allure of Big Data. But is it for everyone?

Are you ready for the Big Data bandwagon?
The promise of Big Data is very real - more data + new ways to analyze data = better business intelligence. So every business should be rushing out to see what Big Data can do for them, right? Not so fast.

Although Big Data has the capability to provide unprecedented insight into business operations, it is not for everyone. At least not yet. Big Data tools are still immature enough, even eight years after we've started down this road, that companies really have to be highly motivated to take on the task.

In my experience, this motivation typically comes from pain. For example, it is the pain of dropping valuable data on the floor because long-term storage isn't feasible within the current infrastructure. Or possibly it's the pain of not being able to monetize collected data due to technical hurdles with existing data systems. This type of pain - and the knowledge that the only way to escape it is to embark on Big Data - provides the fortitude necessary to push through on what can be quite a challenging project. In fact, it was this kind of pain, as experienced by Internet giants who needed to find a way to store, analyze and, ultimately, monetize the massive amounts of customer data to which they had access, that birthed the Big Data movement in the first place.

In addition to being highly motivated to take on Big Data, an organization must also be data driven. Organizations that treat data as currency are better equipped to tackle the challenges of Big Data, because they inherently understand its value and will nurture the initiative, no matter how unruly it becomes. Without this level of buy-in from top-level management, the obstacles of Big Data may prove to be too large to overcome.

The combination of the pain of unexploited data with a data-driven culture goes hand in hand. You typically won't find one without the other. It is when these two factors come together that organizations can then successfully tackle the challenge of Big Data. Without the right motivation or executive support, Big Data endeavors often die before they even begin. Ultimately, an organization needs to be data driven in order to understand the data's value and lament its absence.

About Brad Anderson
As vice president of Big Data informatics, Brad Anderson is responsible for Liaison's Big Data solution implementation, leveraging the company's world-class cloud infrastructure. He is a 20-year data management veteran with expertise in enterprise data warehouses and building and using non-relational Big Data tools.

In order to post a comment you need to be registered and logged in.

Register | Sign-in

Reader Feedback: Page 1 of 1



ADS BY GOOGLE
Subscribe to the World's Most Powerful Newsletters

ADS BY GOOGLE

With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO an...
JETRO showcased Japan Digital Transformation Pavilion at SYS-CON's 21st International Cloud Expo® at...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple ...
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As au...
"We began as LinuxAcademy.com about five years ago as a very small outfit. Since then we've transiti...
More and more companies are looking to microservices as an architectural pattern for breaking apart ...
CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018,...
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news an...
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22...
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held Novemb...
Organizations planning enterprise data center consolidation and modernization projects are faced wit...
A traditional way of software development efforts reimbursing is pay by the hour, which in case of r...
Adding public cloud resources to an existing application can be a daunting process. The tools that y...
CI/CD is conceptually straightforward, yet often technically intricate to implement since it require...
Fact: storage performance problems have only gotten more complicated, as applications not only have ...
"We do one of the best file systems in the world. We learned how to deal with Big Data many years ag...
Traditional IT, great for stable systems of record, is struggling to cope with newer, agile systems ...
Containers, microservices and DevOps are all the rage lately. You can read about how great they are ...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can ...