3 Important Big Data Design Points

Mobile phone growth, sensor data recording, and social media updates feed big data into our systems.  Whether your company likes it or not, startups and the competition are integrating big data and analytics into every aspect of their marketing, products, and operations.  Big data system performance system and high-speed database designs have never been as critical as they are today.
The following are 3 things that your existing and future system designs should incorporate to integrate these new big data management requirements.

  1. Leverage the latest hardware capabilities.  As I mentioned during the prior weeks (here, here, and here) during the series that I wrote about I/O considerations, the latest storage hardware has sometimes as much as 10x the I/O performance improvement over older devices.  Also, most of the latest storage hardware from the various storage vendors have Snapshot or Flash Copy capabilities.  These Flash Copy capabilities provide instantaneous sync point hardware solutions to address backup and recovery scenarios for systems and databases that are continually used for any type of big data or systems that are always on.

  2. Design your data along time based boundaries.  Your data management structures and practices have new governance, security, and context challenges as the big data is captured, ages and becomes obsolete.  Designing your system along time based boundaries provides a way to build in solutions to these governance, security and context challenges so they can be addressed automatically and data governance can evolve as time passes. 

    DB2 Database partitioning, temporal tables, and historical archiving are only a few DB2 database table design options that provide this type of flexibility to change these controls as the data and the database matures.  For example, the governance and storage rules for health care data will change over the lifetime of the patient or insured. Data structures, database design, and archiving strategies need to reflect these considerations and business flexibility requirements.

  3. Evaluate the declining/ascending value of your data.  Sensor, health care, and other types of big data have different and varying degrees of value.  For example, health care data detailing a stroke, heart attack, or other health care incident can be critical for current and future patient care decisions.  Just like the requirements to design your database structures need to reflect the changing governance models over time, your big data design must reflect the varying importance of the big data occurrences.  By identifying and having data importance valuation metrics within your big data system, you will assist your business operations. On-going analytics can be leveraged to quickly identify significant business factors.  Realizing the value of the data and its significance to the patient and/or business operations is critical for correctly evaluating, scoring, and making the best decisions possible for  big data usage and return on investment.

All data, especially big data, is not created equal.  The governance rules and security profiles will change over time, so design your big data system to be flexible and adjustable for these types of data management considerations or else you will have to change it later. That is never ever easy.

Dave Beulke is a system strategist, application architect, and performance expert specializing in Big Data, data warehouses, and high performance internet business solutions. He is an IBM Gold Consultant, Information Champion, and President of DAMA-NCR, former President of International DB2 User Group, and frequent speaker at national and international conferences. His architectures, designs, and performance tuning techniques help organization better leverage their information assets, saving millions in processing costs.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>