Big Data: Part1 - New Data Management Perspectives

With each new trend there is always the hype cycle and the technology readiness level.  The new technology or new capability provides a unique way to solve or analyze a business problem.  The hype cycle is the sales and marketing by magazine crowd.  The technology readiness level evaluation provides more a tested and realistic point of view used by NASA, Air Force, and other life-or-death situation organizations.
Most IT departments have a mix of the hype cycle and the technology readiness level considerations.  The Big Data trend is starting to end its hype cycle since everything lately is Big Data capable or ready for Big Data.  The hype cycle phase is beginning to reach its “Peak of Inflated Expectations” point. Based on your situation expectations can go higher or Big Data can begin its transition into the next phase—the “Trough of Disillusionment.”

Given that the data management departments are always left to maintain all hype-cycle trend implementations such as client/server, distributed, and unique hybrid systems, I’m detailing five new data management perspectives I think will make it through the technology readiness level evaluation criteria.  These concepts should be developed within your data management standards and practices.

• New data types.  Unstructured, audio, video, and other new data types have been around for a while. Now tools, processes and application have been developed that can evaluate, combine and leverage them to establish more critical insights. More robust solutions can be drawn and built from these new unique data types.  Data management departments need to adopt new standards, practices, and tools to model and incorporate these unstructured data types into new and existing application solutions.
• New data sources.  Mobile, social, and machine monitoring metrics are producing enormous amounts of data through their ability to capture many new aspects of the situation.  Using location, social attitudes, and machine operating conditions, business is gathering new insights and using it to improve customer care, understand people’s feeling about a product, and build more reliable machines.  These tremendous amounts of data and sometimes utilizing new data types, require special considerations because of their size, storage, and performance requirements.
• New data temporal requirements.  These new mobile, social, and monitoring data capabilities have brought the need for temporal value of data and timeliness of data to the evaluation equation for data management.  Business has mostly used its systems to keep the business records.  New to the data management evaluation is that a given location or social attitude or operating condition may only be important for an instant of time or that it is only relevant when large numbers of the same condition exist.  Cell phone tower capacity, “Likes” of a new artist’s music, and the temperature/RPMs of a Jet engine are only few examples of how the value of the data is different as it progress through the temporal spectrum.
• New data search capabilities.  Along with the new data types that I mentioned earlier there are new data storage databases such as graph structures, triple stores, and network databases that some say optimize their these unstructured data types’ storage and search capabilities.   Additionally, there are new evaluation capabilities in the object oriented development application technologies such as Java, C++ and others, that provide new graph store search and other evaluation capabilities.  Information can now be evaluated in a number of new different and complex ways through increasingly more business user friendly tools.
• New data analytics.  Through these new increasingly friendly business tools there are new analytics capabilities.  Personalization, data stream evaluation, complex calculations, and unique business industry specialized formulas can be programmed by the business user.  Tablets, phones, and bring-your-own-device displays provide new ways to display, visualize, and share information to everyone who might be interested.

All of these new data management perspectives have been attributed to Big Data.  Your business users are desperate for all these capabilities now and are willing to spend big money on new personnel, hardware, and software.  Use Big Data in your conversations and budget proposals to get the funding and support to bring these new capabilities and perspectives into your data management existing applications before the “Peak of Inflated Expectations” turns into the “Trough of Disillusionment,” and you are stuck supporting another unique application that doesn’t integrate into your existing systems.
________________________________________________________

Dave Beulke is a system strategist, application architect, and performance expert specializing in Big Data, data warehouses, and high performance internet business solutions.  He is an IBM Gold Consultant, Information Champion, President of DAMA-NCR, former President of International DB2 User Group, and frequent speaker at national and international conferences. His architectures, designs, and performance tuning techniques help organization better leverage their information assets, saving millions in processing costs.