3 Big Data Performance: Building Analytical Consensus

Working with big data performance everyday takes the “big” out of it.  The big data is just a bigger data beast that your procedures and processes need to plan for and deal with.  Every big data decision is just another decision, but a bit more interesting and intensified because the outcome is more impactful and bigger across your customers, product, and profit base for the company.

Your big data performance is always in the back of your mind as the smaller test and prototype environments provide quicker answers but can lie with a smile and return no data row answers or acceptance of your questions because of their limited range and domain of data.  I have written many blogs over the years about big data performance and big data considerations. Check them out from the list below.  When driving for big data performance, build your procedures and use these three big data performance ideas to build consensus for your analytical outcomes.

1. Which big data business element(s) are critical to your analytics?
   Within your big data there can be a variety of information, but all your analytics and solutions need to be based off your core business attributes.  The core business of your big data is usually either your customers, products, transactions, social interactions, or some of other concrete data point element(s) that everyone trusts and understands.
   
   These big data elements are the critical business analytics points that successful decisions have been used as a basis and which drive many business questions and answers.  Marketing campaigns, new product rollouts, and general business are done every day with these elements.  These big data element examples are easy to understand, but within your type of business or processing there are also the associated elements to these major key elements. Customers, products, transactions, and social interactions are at the first layer and easy to identify.  Now peel another layer of the big data onion to expose the next level of elements.
   
   
2. What associated attributes of your critical business elements can be analyzed for deeper insights?
   So you have identified your top most popular big data keys and use them consistently.  So what are the next level of elements make up your big data analytics element(s) associated keys?  What are customers’ relationships to other, deeper pivot points within your big data lake?  This goes back to your original data modeling of your business at the basic level down to your many attributes.  What other associated elements have shown relationships to your major big data elements?
   
   These other next level elements are pivot points to your main big data elements.  These next level elements have relationships but probably not one to one relationships with your main big data keys.  A basic example is your customers, with the next level being customer country or location attributes, with next level male or female, next level type of product and the associations continue with your answer set size continuously getting smaller.  Research your indicators, codes, status and other elements within your data to uncover them.  Document these other obscure, often forgotten elements, and use their values to analyze and link new data into your analytics procedures.
   
   
3. What is significant enough to drill into further?
   Within your big data statistical analytics procedures, the results of the different attributes always need to be quantified and documented.  The documentation of the different analytics is vital to understanding the result quantities, ratios, and percentages to other big data attributes.  The documentation of the quantities can lead you to understand the statistical significance or tipping points of the different attributes.
   
   These different tipping points can be vital because they can help everyone understand when the combination of complex elements is significant enough to result in an investment in or not.  For example, if your marketing campaign data analytics shows a 35% chance of success with 20% of your customers, that may or may not be a profitable campaign.  Depending further on your product price point, discount offered the customer, profit margin, and other factors, it could be wildly successful or not. So knowing the significances and margin of success needed within your big data analytics is critical for overall success.  As new big data analytics comes into your data lake, develop the success criteria along with your element investigation strategy to know when the results justify the analytics work.

These are three more ideas that need to be researched, understood, and defined when developing your next new big data analytics project.  The quicker your sub-project can dig deep into the data and understand its deeper relationship and success criteria, the better everyone can leverage the results.

List of related articles:

3 DB2 Critical Design Factors for Big Data Analytics Scalability 2-10-16

3 Critical Factors for Big Data Analytics Performance 1-13-16

3 Important Big Data Design Points 6-19-14

Three Ways to Avoid Big Data Chaos 4-17-14

Five More SQL Performance Tips for your Big Data 4-3-14

Big Data: Four Factors to Tune your Meaningful Analytics 3-20-14

Big Data: Three More Ways to Choose Meaningful Analytics 3-13-14

Big Data: Three Ways to Choose Meaningful Analytics 3-6-14

Dave Beulke is a system strategist, application architect, and performance expert specializing in Big Data, data warehouses, and high performance internet business solutions. He is an IBM Gold Consultant, Information Champion, President of DAMA-NCR, former President of International DB2 User Group, and frequent speaker at national and international conferences. His architectures, designs, and performance tuning techniques help organization better leverage their information assets, saving millions in processing costs.