"The first principle [of science] is that you must not fool yourself and you are the easiest person to fool." – Richard P. Feynman

People are very good at hitting targets. We can’t really help it—we are made that way. When we think we know the result of an experiment before we have actually performed it and analyzed the data, we have given ourselves a target, and that can easily impair our objectivity. We need a way to prevent this from happening. One way is Blind Analysis.

What is Blind Analysis?

The basic idea of blind analysis is to hide the result from the analyst. This result can be hidden in different ways. However the result is hidden, you don’t want the analyst to know what the result is while performing the analysis. While this is done to prevent the analyst from inadvertently changing the result with his own personal bias, it can also be used to protect proprietary information.

The Difficulty of Bias

Every model will have some bias. Bias is an average difference between what you measure in the lab and what your model predicts. A useful model will have minimal bias.

Bias can come from many sources, among them are:

1. Mathematical artifacts of data analysis.
2. Mistakes
3. Skewing of the data by the people participating in the experiments (participants).
4. Skewing of the data by the people running the experiment (the experimenters).
5. Skewing of the result by the people analyzing the data (the analysts).

We’ll look at all of these sources of bias, but our main focus will be on preventing number 5.

Noise and Mathematical Artifacts

Mathematical artifacts come from the nature of data. What makes a piece of datum different from a number is that the datum is uncertain. Eight is always 8, but a measured 8 inches may be 8.1 inches one time and 7.9 inches another time. Data comes from measurements. All measurements are uncertain. Everything that comes from data, including predictions, is uncertain because the measurements upon which all calculations from data are based are uncertain.

The uncertainty in measurements is often referred to as noise. Noise tends to be random and symmetrical – it just as often makes a measurement look larger than its average as it makes it look smaller, and it is impossible to predict the direction or the magnitude. If we could repeat a measurement an infinite number of times the average would be the true measurement free of noise. All of the high noise would exactly cancel the low noise.

Unfortunately we cannot repeat a measurement an infinite number of times even if we want to! We can only repeat a measurement a finite number of times. When we do this, it is almost certain that the high noise and the low noise will not exactly cancel each other. This leads to bias – either the average will be a little too high or a little too low. Predictions from models are predictions of averages, so the predictions will tend to be either a little too high or a little too low.

As long as this bias isn't too large it will not prevent our models from being useful.

Mistakes

Good experimenters take every precaution they can to avoid mistakes. Unfortunately, mistakes sometimes happen anyway. When they do, they lead to bias. The Peer Review process is one way to catch these mistakes.

Single Blind, Double Blind, and Triple Blind Studies

In any experiment the participants, the experimenters, and the analysts could introduce bias. This happens when the data is skewed in some way, either intentionally or unintentionally. Three approaches have been developed to address these sources of bias: Single Blind, Double Blind, and Triple Blind studies are the result.

In a Single Blind study, information that could bias the participants is withheld from them. The experimenters and the analysts have all the information. The "Pepsi challenge" is an example of a Single Blind study. The experimenter pours Pepsi into one cup and Coke into another cup. The participant does not know which cup is which. The participant tastes each drink and states which he prefers.

In a Double Blind study, information that might lead to bias is withheld from both the participants and the experimenters. The "Pepsi challenge" could be modified to a Double Blind study if the experimenter did not know which drink was which. Two bottles labeled "A" and "B" could be provided to the experimenter. He would have no way to know which contained Pepsi and which Coke, so he could not subtly influence the decision of the participant. Another experimenter, who would not be present, would hold the key to which was which.

In a Triple Blind study, information that might lead to bias is withheld from the participants, the experimenters, and the analysts. The analyst would be provided data without knowing the identities of the participants and only knowing if a participant preferred A or B. When his analysis was completed, he would know if either A or B was Statistically Significantly preferred, and if one were preferred whether it was A or B. Once again, another experimenter who did not interact with the others would hold the key to the true identities.

The Triple Blind study is the only study that incorporated Blind Analysis – hiding information that could bias the analysts. Blind Analysis can be performed without blinding the participants and/or the experimenter. This is convenient because not all experiments lend themselves to the full Triple Blind protocol.

Performing a Blind Analysis

When performing a Blind Analysis the following steps are used:

Disguising the Data

In order for an analysis to be blind, the data must be disguised in some way. It is not necessary that every aspect of the experiment be disguised – only those features that could lead to bias. For example, in the "Pepsi challenge" it would not be necessary to prevent the analysts from knowing they were analyzing taste test data.

Revealing the Results

When the analysis is complete and the analysts have agreed that all anomalies have been accounted for and they are confident that the analysis is correct, the experimenter holding the key can now reveal the actual results.

Further Analysis of Revealed Data

It is not necessary that analysis should stop once the results are revealed. If the result is clearly wrong, it would be foolish to accept it just because it was produced by a blind analysis. For instance, you might run the experiment predicted by your model. If it doesn’t predict well, the model isn’t useful.

If the analysts are still blind to the results (it is best not to let them know everything until you are satisfied with the result) you could ask them to review the analysis in light of new information – the experiment that did not agree with the prediction. Of course the new information must also be disguised.

In some situations, it would make sense to perform a limited additional analysis without blinding. For example, calculating the Statistical Tolerance Limits on results generally needs no blinding.

No Guarantees

It is important to realize that blind studies do not guarantee that no bias has been introduced. They always rely on the integrity of everyone involved. The purpose of blind studies, and Blind Analysis specifically, is to prevent accidental bias.

Blind Analysis can help you to avoid the danger of inadvertently trying to hit a target. Sometimes it is best to put on your blinders!