Enabling Disruptive Thinkers in Your Organization
September 19, 2013, SNIA Analytics and Big Data Summit, Santa Clara, CA—Molly Rector from SpectraLogic discussed the changing nature of knowledge workers and how they are affecting technology choices. Data is not just historical content, but is now helping to predict future activities.
One technology trend is the growing overlap of high-performance computing and enterprise datacenters. A disruptive technology creates disruptive users, and the users are becoming more important to business performance and technology change.
An example of a disruptive technology change was the Model T automobile. It wasn’t just that it displaced the horse and buggy, but affected the whole horse and buggy supply chain. Instead of custom building buggies by hand, the Model T was assembled on an assembly line from baskets of identical parts. This was not a new technology, as Eli Whitney did for rifles and cotton gins in the Civil War, just a new application of existing technology.
The technology area is being flooded by data from the mobile and social spaces. The volume of data changes how to use that data, and leads how to store and access that data. Only when all of the data is accessible can one unlock the value of big data.
The value is not only in the mobile and social spaces. In healthcare, DNA sequencing is becoming a highly accurate predictive mechanism. This is coupled with the costs associated with a DNA analysis dropping to the $200-300 range for high-risk issues. Continuing cost decreases and larger DNA databases will enable highly personalized medication.
In similar areas, the CDC is using Google searches for cough medicine and cold remedies to track flu outbreaks in hours and days, rather than the weeks from their formal medical practitioners’ inputs. The math models that Google uses that localize the searches for Nyquil and other over-the-shelf remedies to show where people are starting to experience cold and flu symptoms. These analytical models are combined with other search capabilities to expand the information base for the users.
The definition of big data is still going through changes. It encompasses not only large data sets, but also more data from different sources. By 2020, the data will be over 40ZB a year. Last year, over two thirds of all data was created by individuals, resulting in more information about people than they created, by combining various sources through analytics.
Historically, big companies have used old data as static entities with small value to predict the future. Now they are extracting value from the large volumes and high velocity of current data. The change to a big data perspective facilitates a mix of old and new data to become predictive. The use of correlative searches changes the questions from what to why.
As a result, companies are starting to implement no-delete policies for their data in case the historic data may be useful in the future. The current data snapshots are not that useable or useful on their own. Analytics that merge old and new data sets allows for better data interpretation and understanding. This process only works when the data sets are fully accessible.
The use of correlative functions answers some of the questions that companies don’t even know how to ask. For example, Wal-Mart has found that Pop Tarts sell more near the time for a hurricane, so they are now putting high-margin items near the Pop Tarts when there is a hurricane warning in an area. This example shows how changing why to what will happen next can benefit a user.
Changes in the datacenter due to the large increases in data across all sectors lead to deep storage—low power, low cost, high density, and high latency. This data is accessed through a REST protocol and web browser. The attributes of deep storage can be fairly open: public or private cloud, object store or file system, with protection built in. The data protection is from multiple copies in diverse locations, but not part of any backup processes.
The data in the cloud is the next step in virtualization. Access is through http or REST protocols. Nested file systems can be configured as flat or hierarchical, and the trees can be expanded, limited by the scalability of the metadata stores. A file system probably has a 5-10PB NAS scaling limit.
Structuring the deep store as an object store with a flat file system overcomes the physical limits of file systems and can contain both the data and increases in metadata. This type of system can keep the data and enriched metadata to contain more content information. Most experts and contractors are already familiar with the web or REST interfaces as well as the data protection concepts.
The data attributes are key to the future of big data use. Users must put the data into formats that are most accessible and make changes to security provisions to store today and use tomorrow. The advent of real-time analytics is now available on the Web and some are used by retailers to predict buyer habits, in oil and gas exploration R&D, and in security for surveillance with the data being objects with rich media.
The move to this type of big data is still slow. Hadoop is a partial solution and also a barrier, due to its language structure. The points of contact are everywhere, so do-it-yourself solutions are dominant. Despite the talk, big data is still 2-3 years out, due to the various issues like systems, storage, etc.
