Three Ways to Avoid Big Data Chaos

Companies are always trying everything to gain a competitive edge, but with dysfunctional data procedures, management/CIO turnover, and lean business profits, new data projects face unprecedented difficulties.  With the plethora of IT trends, directions and technologies, cloud platforms, big data, and new anemic exotic scripting languages, IT personnel are getting conflicting directions for their project efforts, leading to data chaos within many companies.

The following three items will help you cut through the confusing big data chaos situations and focus your company’s data management projects for success.

1. Have clearly defined and fully understood results related data points.  Big data is being boiled and dissected so much that micro marketing to individuals is now commonplace.  While this might be good for some, it may not improve the bottom line.  As companies file their taxes and write down all the big data project failures as noted in this Register article, and this Wall Street Journal article companies may realize they need more structure around their big data projects. Your project needs an ultimate bottom line results relate goal data point to succeed.  Better data mining, even to the micro market level, only marginally improves customer relations and the bottom line.  Determine your bottom line success data point before you begin.
   
   
2. Use science to measure analytics scientifically.  The Google Flu Trends big data project that was previously used as a success story that now has been discredited in Harvard Business Review and  here in Live Science  showing that the proper analytics need to be done against the correct data.  If analytics are going to be used against a situation, understand the context and circumstances of the big data input and measure the results scientifically.  There are lies, damn lies, and statistics so prove your analytics scientifically to insure enough business profit.
   
   
3. Big data hype is over, now it’s time to work IT work scientifically. Big Data hype is over, now it’s time to work IT work scientifically. While your company may have benefited from new sensor, social or web data, big data can be a dangerous area to bet on.  Some of these big data project cancellations and failures are starting to emerge because it took the team a while to discover that NoSQL sometimes really does mean no data integrity and it also means no SQL interface.  Whether it’s a big data, Agile or any other project, at its core data processing needs to be run scientifically. Scientifically because total cost of ownership, performance, and data integrity are the most important IT and business components of all projects.  The trench of disillusionment for big data is upon us and the next buzzword bingo is just around the corner.

This resurgence of scientific analysis of technology is coming back just in time as Moore’s law is running out of gas and there won’t be new powerful CPU chip improvements to mask poor designs or application processes much longer.

Use these three ideas and prepare for the next hype cycle because this big data hype has reached its apex and coming back to earth.  Now maybe we can all get back to the computer processing CPU and I/O fundamentals. Or is the hype cycle started up already using the cloud as its new buzzword?
____________________________________________________

Dave Beulke is a system strategist, application architect, and performance expert specializing in Big Data, data warehouses, and high performance internet business solutions. He is an IBM Gold Consultant, Information Champion, and President of DAMA-NCR, former President of International DB2 User Group, and frequent speaker at national and international conferences. His architectures, designs, and performance tuning techniques help organization better leverage their information assets, saving millions in processing costs.

_______________________________________________________

Sign up for the IDUG DB2 Technical Conference in Phoenix, Arizona this May 12-16th.

• Sign up for dinner with me during the IDUG Dine Around Dinner Thursday night at IDUG.
• Also plan on attending my presentation 1221-F04 “Big Data Disaster Recovery Performance,” Tuesday 4:30 at the IDUG conference in Phoenix.
• The IDUG European conference is still accepting presentation abstracts to be potentially picked for the IDUG EU conference in Prague, Czech Republic November 9-14th.

For more details on any of these items go to www.idug.org.