I’ll never forget my first experience analyzing a data set. Now, I know that does not sound particularly exciting, but stay with me. It was shortly after I graduated from college, and I was working in the area of international economic development. I was in Togo (West Africa), and my team had just completed an extensive survey of 5 or 6 villages in a rural area. We had a lot of good, raw data. The project manager asked me to produce a preliminary report of the data before I left the country. He gave me 3 weeks, a team of surveyors, and an Apple computer loaded with Excel. It was intimidating.
However, there is a reason I won’t forget this experience (outside of the unusual conditions!) It is because the principles applied to that early, small project, are fundamentally the same principles needed today. There may be much more technology, and more sophisticated tech available, but these principles have not changed.
Principles of Data Analysis
Let’s explore these principles and how they apply to modern situations.
1. Establish The Foundations of a Data-driven Organization
First things first.
How did my first project apply this principle?
The organization knew that to prove to their donors the economic development project was successful several years later, they had to establish a baseline of data before starting. So, they conducted a set of surveys of local households in 5 or 6 villages in the project area. They collected information on land farmed, produce brought to market each year (if any), cash on hand (if any), micro-businesses run by members of the household, equipment owned, and so on. They also collected a pile of economic data in a way that was appropriate to the economics of the area prior to starting anything.
How does this principle apply to our companies?
The leadership team in any organization is responsible for establishing the foundations for a data-driven organization. How are decisions currently made? Are there presentations of data that can appropriately inform the decisions? Even if the answer is no, by turning to data, the leadership team creates the demand for the appropriate reporting. This, in turn, drives the organization to create the necessary data collection and analysis program.
Start with the existing and immediate goals of the business. What are the metrics or KPIs associated with these goals? You have to be able to measure it easily and accurately in order to achieve it. This is a key point – the leadership of an organization must consistently establish metrics to measure the achievement of the business goals. Of course, there are some goals that are more qualitative in nature, rather than quantitative. However, often even these lend themselves to some sort of measurement, coupled with qualitative evaluation. Additionally, the leadership team must allocate money and resources to an effective data program. It requires technology, training, and specific skill sets to produce effective analytics.
2. Prioritize Data Integrity
This is the first, fundamental rule in applying data insights to decision making. It can’t work unless you have put in the time and effort to prioritize the integrity, accuracy, and cleanliness of your data.
How did my first project apply this principle?
It was very manual and laborious, but we did it. I still remember sitting at a table with a team of 2 surveyors, staring at an enormous pile of paper surveys, filled out by hand.
We started by reviewing the surveys. Were there any that should be discarded due to questionable circumstances of the interview? Then, we sat down in front of 2 old Apple Macintosh computers and started typing data into Excel (more on technology later!). One person entered and the next person double-checked the entries for accuracy.
How does this principle apply to our companies?
We have far more sophisticated and automated ways to collect data these days. But, the same principle applies. How was the data collected? If financial data, are there ways that data is audited or compared to other numbers? If survey data, were the questions phrased accurately and properly, without any leading? This is the sort of evaluation that must go on continually and consistently.
The data is then collected into a data storage system. This can be a database, data lake, or unstructured data. Then, it must be very clear to those analyzing the data precisely what the data represents. A definition of the data points is critical to the correct application of analysis to the data points.
Don’t shortcut this principle. It’s tempting to do so because the fun is in the analysis and insights, which come after this. Don’t do it. Once you have a compelling presentation of the data, it will cause the audience to draw conclusions. If those conclusions are based on questionable data, the wrong decisions will be made. Data integrity and quality can’t be verified in a presentation. They can only be assured early in the data collection process.
3. Get the Appropriate Data Technology and Tools
One size does not fit all. You must evaluate the volume of data you have, the format you prefer, the number of data sources, the type of data, and the kind of insights desired. These are just a few of the criteria. There are several layers of technology and processes in this stack, made even more complex by AI.
How did my first project apply this principle?
Perhaps Apple Macintoshes and Excel don’t strike you as technology even worth discussing today. But, at that time, it was a big deal.
Typically, the technology “platform” used for surveys like this was to send the raw data (that would be the paper surveys) to a data service provider like SAS or SPSS. They would enter the data (with no actual understanding of the data), and run it through their statistical software to provide a printout of correlations and analytics. The resulting report was dense and required a thoughtful analyst to carefully evaluate it and put it into a more approachable presentation. The time you sent the raw data in to the time you received the dense report typically took at least 3 months.
The project manager wanted to experiment with a new approach. He tried this 2-week sprint of using me and my survey team to enter and analyze the data on a smaller, more accessible system. He had a good presentation in 3 weeks. Three months later, the sophisticated SPSS report showed up. One month later, he had that presentation. He told me that the information and insights in the SPSS report were effectively the same as mine.
How does this principle apply to our companies?
- Ensure modern data platforms and analytics tools are in place.
Data technology stacks are rapidly evolving and changing. How we store the data, manage access, engage analytics tools, and so on, must adapt. We like what we are familiar with, and we like what we know. However, as company leaders, we have to be open to new platforms and tools… within reason. Experiment carefully. Just because it is new and has “AI” in the name, does not mean you will get more accurate, insightful, or actionable information.
- Advocate for scalable and secure infrastructure. Leverage cloud-based data warehouses and lakes for flexible and scalable data storage.
It’s a given these days that your data will grow. The best way to accommodate this growth is to have scalable, yet secure infrastructure. Cloud-based infrastructure is easily scalable. Security is a critical component of your infrastructure. Your data is extremely valuable, so treat it that way. Put resources behind your cybersecurity programs.
Typically, your data comes from different data sources. Finding correlations and relationships between data points from different systems is far easier when the data is collected in a data warehouse or lake. This leads us to the next point.
- Implement integrated data technology platforms that streamline data collection, transformation, and analysis.
Having a data technology platform that easily supports collection, transformation, and analysis ensures that as you change your KPIs or business goals, your new needs can be easily addressed. The reports and dashboards you look at today may not be what you need next year. If your data platform is set up properly, you can adapt easily to the new views required.
Data transformation is a critical aspect of your data program. It:
- Takes diverse and unstructured data and converts it into consistent, structured formats that are easier to analyze
- Cleans, standardizes, and enriches data
- Facilitates the merging of data from multiple sources for a unified view
- Optimizes data for analysis
- Can support regulatory compliance by applying consistent data handling practices that align with data privacy and security regulations
Remember point 2 – prioritizing data integrity? Data transformation is a critical part of data integrity assurance. It ensures data integrity and usability.
- Incorporate advanced AI and machine learning tools to automate data processing and uncover deeper insights.
There are many examples of these kinds of tools, which can uncover data relationships and insights much more efficiently. Executives should be exploring options to incorporate into their data programs.
4. Leverage Accurate Analytics for Actionable Insights
Finally, we come to the analytics. Just like the need for integrity in the data, there is also a critical need for integrity in the analytics. It is always possible to cherry-pick data elements to present a report to prove a preconceived conclusion. This happens often in organizations.
The company culture and leadership should work diligently to promote honesty first. What we all want is the truth. It enables us to make good decisions. And so, we want the truth in the analysis of the data. Even if it doesn’t support our current thinking.
Create a data organization that is free of politics and independent. The data team should have access to the executive team, and vice versa.
Here are some principles for data analysis to keep in mind.
- Focus on key metrics that can drive decisions. Don’t get distracted by interesting metrics that are fun to know, but that can’t drive decisions.
- Regularly review and revise the metrics. Some will stay the test of time, but many will change as your organization and the market changes.
- Invest in skilled analysts.
- Make data accessible and actionable. This cannot be stated enough. If data analysis is available easily, on demand, and self-served, it will be used. If it isn’t, it won’t. If the data analysis is presented in a way that is actionable, it will be used in the decision-making process.
- Train as many in your company as possible on data analysis. While executive team members may not be actually doing the analysis, the more they understand how to read and interpret the reporting, the better off the organization.
- Encourage cross-functional collaboration. No silos.
Modern Data Analysis, Integrity, and Tools
The principles of data analysis I first encountered during that small project in Togo remain as relevant as ever. Establishing a strong data foundation, prioritizing data integrity, and leveraging the right tools are universal necessities for success in any organization. While the technologies and methods have evolved, the core ideas endure. As we navigate an increasingly data-driven world, embracing these principles with curiosity and adaptability will empower organizations to unlock their full potential and thrive in a competitive landscape. If you want to learn more about how to drive better decision-making, foster collaboration, and enable innovation within your organization, contact our experts today.
