Controlled Experiments and Discoveries

Knowledge advances in two ways: intentionally and by accident (and these are, by definition, mutually exclusive: that which is not intentional is an accident, and vice versa).

It's worthwhile to understand what it is that constitutes knowledge and research, because having that fundamental understanding gives us the greatest opportunity to learn from the data we have.

I had considered titling this post "Found Art" because some of research is largely "found art": that which was discovered serendipitously, but was, perhaps, thought refuse.

But the motivation for this post was talking with a writer who had attempted to run an experiment, and the experiment failed. "I should try something else," he suggested. But I'm wondering what could be found in the wealth of data generated by what he did do. It may not be that the failed experiment will provide a gem of information, but it might provide valuable insights that will guide future research.

Controlled experiments are one of the paradigms of research--it is intentional research in its most extreme form: possible outcomes are limited as much as possible to that which can be accurately measured.

In the laboratory a great deal of control can be exerted to limit different kinds of variability. That kind of control cannot be exerted in the field. And so controlled experiments may break down for various reasons beyond the control of the researcher, eliminating the possibility of getting the results that had been desired and intended.

In the field, however, if you are documenting the process extensively, even a failed experiment will generate masses of data that can be processed and analyzed for insights that were missing when the experiment was set up.

The first place to look is the failures. Your experiment failed because of record-keeping lapses by the participants? What does this teach you about setting up an experiment that will work in similar conditions? Does this suggest a failure to engage in the experimental activities? Why? What can the failures of the experiment teach about setting up an experiment to study the issue that motivated the original study? The failures, in some cases, may tell you about the very thing that you're testing, too. Do they indicate any results that would indicate that there are problems with the general premises under which you are operating?

Careful examination of a "failed" study can be quite valuable because of all the data generated--it simply requires one to look at the data in a different way--to see it through different eyes--to see the urinal as art.