Seven guiding principles for analyzing data

I was talking to an old friend the other day who is involved in using the results of research to help grow a business. He told me some interesting stories that made me revisit some basic tenets of good analysis. Yes, you may think that some of these are obvious, but they still bear repeating. Here are seven interrelated principles to start with:

• Process is a way of thinking, not a substitute for thinking. You’d be surprised at how many people fall into this trap. For example, in behavioral research certain metrics might be the norm for capture. These might include the number of times that eye contact was made, or the quality of the interaction with the examiner. However, simply because others have used these “tried and true” measures doesn’t mean that they necessarily fit the situation you’re currently examining. Think about it.
• Data needs to be thought about and reported in context. This is a pet peeve for me. If someone tells me that 1.5 million Americans were out of work at some point during the Great Depression I may think that is terrible, but I don’t really understand what that means because that fact was not put into context. I don’t know what percent of the working population this represents, or for that matter if it includes women or other groups. When a vendor tells me that Company X saved $20M by utilizing its product, that’s great but what does it really mean? What percent of its overall costs (whether by department or company) does this represent? How is another company, looking at this information, supposed to respond unless it understands what the data mean in context.
• Look before you leap. Before you start applying statistical techniques or cranking out charts and reports, take a good hard look at the data you’ve collected. Be thoughtful. Ask yourself some basic questions such as, “Do the data seem reasonable, complete, and accurate?” “What are the data suggesting?” “Is there some sort of hypothesis I can propose to test based on the data?” Often times people simply jump into running every sort of analysis on their data, simply because they can.
• Question everything. If you are using the results from someone else’s analysis to build upon, you need to question how they got their results. Did this analysis make sense? How big was the sample? This is (I hope) a basic principle in scientific research but I haven’t seen it necessarily carried over into business. If your sales figures have jumped by 50%, you need to ask yourself, “Why?” Perhaps new products were added or new markets were tapped. Whatever the reason, make sure the data makes sense. Data quality is obviously important here.
• Do a gut check. this is an extension of the question everything principle. Again, once you’ve done some analysis, you need to ask yourself whether it makes sense to you or not. Remember the old saying, if something is too good to be true it probably is. If your sales figures have jumped by 150%, you need to ask yourself if this is possible and then go and figure it out.
• Coincidence is not the same as causality. Just because it may appear that two variables are somehow related it doesn’t mean that they are. Remember to question everything and do a gut check.
• Just because the data exist doesn’t mean the data are relevant. Here, you need to ask yourself what you are trying to figure out. Just because you have the data doesn’t mean that the data are necessarily useful to your analysis.

I’m sure you can think of more and I know I will certainly come up with others. But, that is all for now.