Unresolvable questions and the search for understanding

Researchers want answers, and ideally, they want good, solid answers that will stand up to examination and challenge. Unfortunately, some questions do not have any fixed answer. The absence of clear answers, however, does not preclude important learning, and researchers can benefit from being able to delve into such uncertainties in search of many interesting ideas, even if no specific answer can be found. 

In the late 1950s, a philosopher named W.B. Gallie demonstrated that there were some things could never be fully defined—he called them “essentially contested concepts.” In the late 1960s, design theorist Horst Rittel argued for a class of “wicked” problems (which included but was not limited to design problems) whose members had no definitive formulation, among other characteristics.  For both Gallie and Rittel, a crucial factor was the social element: different people view things differently. Gallie relied on a sporting example derived from cricket (I believe—I’m working from memory) to demonstrate how different views about the sport made it impossible to define the “best.”.  Gallie’s example is particularly salient for me because I enjoy the sporting fan’s common questions regarding which players are best and what teams should do—one of my favorite authors is Bill James, the baseball analytics guru—and yet, as a philosopher and researcher, the more I look at such questions, the more complexity there is to see. And ultimately, given that I accept the ideas of Gallie and Rittel, I see these questions as unanswerable.  Despite believing these questions are unanswerable, I still see the debates that they produce as interesting and often informative.

Not to mention that I am somewhat interested in basketball and the NBA playoffs, and a lot of current discussion revolves around an unanswerable question that is, nonetheless interesting, and perhaps even informative, to explore. That question is the question of LeBron James. How great is he? Is he the greatest ever, the GOAT (Greatest Of All Time)? Or is he “just” top five? You don’t have to read much to find people discussing LeBron’s “legacy.”

Before LeBron, Michael Jordan was generally regarded as the greatest of all time (GOAT). Some argued for Bill Russell with his 11 championship rings and multiple MVP awards, or, perhaps, for a few others, but Jordan was the most common choice as GOAT. LeBron, however, has been doing amazing stuff that no one else in basketball can do, and his accomplishments are piling up. He is currently having another spectacular postseason, at least in terms of individual performances, although his team is facing a 1-2 deficit to Boston in the conference finals.  It is these performances that spark the debate: “look at that performance,” says one side, “he’s the best ever.” The other side says “well, it doesn’t mean much if he loses. Jordan won all six times he went to the finals!” 

This is the simplistic version of the argument, of course, because on closer examination this simple argument will reveal complexity that cannot be eliminated—complexity of the sort that contributed to the claims of Gallie and Rittel that some things cannot be completely defined. The simplicity of saying “Jordan won all six times in the finals; LeBron only won three and lost five,” might be fine for chatting at the bar during a game, but it certainly isn’t enough for serious research.

Firstly, we note that we can’t just reduce the argument to who has more rings, because by the “rings” standard, neither Jordan nor LeBron is all that close to the top of the list. By the rings standard, Bill Russell is the greatest, followed by a bunch of his teammates and Robert Horry. And, with all due respect to Robert Horry and Russell’s Celtics teammates, they are not all-time greats—no one is suggesting that Sam Jones, with 10 rings, is the second-greatest player ever. Trying to reduce the debate to a single dimension distorts the question: a player’s performance is much more complex than that single dimension.  This problem of multidimensionality makes it difficult to evaluate many things—what is intelligence? what is creativity? what is a “good employee”? how do we evaluate students’ learning?

Once an issue is understood to be multidimensional, it becomes increasingly difficult to make any certain decision. Beyond championship rings, there are many statistics that allow a comparison between two basketball players—points scored, rebounds, assists, etc., etc.  But the more dimensions added to the evaluation, the greater the likelihood that there will be contradictory indications. If one player had better stats in every possible category than any other player, there wouldn’t be difficulties. But that’s not the case, and that leads to complexity and uncertainty: how do you choose to weight different dimensional in an overall evaluation? If Jordan has more points but LeBron has more rebounds, who is greater? What’s more important for evaluating greatness?

Focusing on won-loss records can give an example of this problem of evaluation. MJ won more rings than LeBron, and that matters. But LeBron advanced to the finals more times, and that’s worth something, too: after all, if LeBron is criticized for losing in the Finals, shouldn’t MJ be criticized for losing in the Conference Finals? How do we compare those different achievements? Or, at the other end of the playoffs, we can see that MJ lost in the first round three times, while LeBron has never lost in the first round. If MJ were clearly greater, shouldn’t he have a better record in the first round? There are lots of stats and sometimes Jordan’s are more impressive (30.1 pts/gm vs. “only” 27.2 for LeBron), sometimes LeBron’s are (7.4 rebounds/game, 7.2 assists/game vs. 6.9 and 5.3 for Jordan).

The search for detail in comparing the two may not lead to any conclusive answer about which is better, but it can help us see the question more richly, and this can inform us about basketball and about processes of evaluation.  And, in a way, what we get out of the examination is potentially more valuable than an answer: it doesn’t really matter who is “the greatest”—whether we say that Jordan is the greatest or LeBron is the greatest or Russell, Kareem, Bird, Magic, Wilt or whoever. It doesn’t really matter who gets called the greatest, or who really is the greatest (if it makes sense to reduce such complexity to such a simple question). But, although the question itself cannot be answered, what is learned in the process of trying to answer that unanswerable question can give us insight into the more general process of player evaluation, and that has practical value to basketball organizations or to fantasy players.

Some questions that have no answer are still worth asking and examining.

Controlled Experiments and Discoveries

Knowledge advances in two ways: intentionally and by accident (and these are, by definition, mutually exclusive: that which is not intentional is an accident, and vice versa).

It's worthwhile to understand what it is that constitutes knowledge and research, because having that fundamental understanding gives us the greatest opportunity to learn from the data we have.

I had considered titling this post "Found Art" because some of research is largely "found art": that which was discovered serendipitously, but was, perhaps, thought refuse.

But the motivation for this post was talking with a writer who had attempted to run an experiment, and the experiment failed. "I should try something else," he suggested. But I'm wondering what could be found in the wealth of data generated by what he did do. It may not be that the failed experiment will provide a gem of information, but it might provide valuable insights that will guide future research.

Controlled experiments are one of the paradigms of research--it is intentional research in its most extreme form: possible outcomes are limited as much as possible to that which can be accurately measured.

In the laboratory a great deal of control can be exerted to limit different kinds of variability. That kind of control cannot be exerted in the field. And so controlled experiments may break down for various reasons beyond the control of the researcher, eliminating the possibility of getting the results that had been desired and intended.

In the field, however, if you are documenting the process extensively, even a failed experiment will generate masses of data that can be processed and analyzed for insights that were missing when the experiment was set up.

The first place to look is the failures. Your experiment failed because of record-keeping lapses by the participants? What does this teach you about setting up an experiment that will work in similar conditions? Does this suggest a failure to engage in the experimental activities? Why? What can the failures of the experiment teach about setting up an experiment to study the issue that motivated the original study? The failures, in some cases, may tell you about the very thing that you're testing, too. Do they indicate any results that would indicate that there are problems with the general premises under which you are operating?

Careful examination of a "failed" study can be quite valuable because of all the data generated--it simply requires one to look at the data in a different way--to see it through different eyes--to see the urinal as art.

Boiling water and the complexity of our actions

Once I was a TA in a basic computer course (back in 1995); one exam question (or quiz or homework) asked students to write "pseudo-code" for boiling water ("pseudo-code" being a sort of plain-English description of an algorithm).

The expected answer was something like
Get a Pot
Fill it With Water
Put it on the Stove
Turn On the Stove
Wait

And that is a basic level description of what we do. But there's far more complexity there. The reason I write this is as a follow up to the previous post about research questions: we have basic ideas about the world, but when we examine them, we find that they open up into great complexity.

This sort of thing happens with computer programming, too, and that was the problem a lot of the students had with moving from five lines of pseudo-code to a working program--they just didn't recognize all the little details that are worthy of attention. Similarly, when we're looking at an assertion, we need to recognize all the details that are worthy of attention and examination.

Let's take a brief look at the pseudo-code. The first step: "Get a Pot."
Easy? Yes, it's easy if you have a pot, or know where to get one. Let's say you have a pot that you intend to use and it's in your kitchen. Then "get a pot" requires first going to your kitchen, which itself may not be trivial, if, for example, you're out running errands. You have to go home, then once home you have to go to the kitchen. Once you have gotten to the kitchen, you have to find the pot. This may be easy if your kitchen is well-kept. But maybe the pot is already in use--then you have to find an alternate pot. Or maybe the pot is dirty, and you have to wash it. Or maybe the pot isn't where you would expect it because the friend you had over for dinner the previous evening put away the dishes. Once you've gotten the pot and it's clean and ready for use, then you have to fill it with water. Again, we can find complexity here if we look for it. We need to have running water, we need to keep the pot oriented in the right direction, we need to keep it still, we need to support its weight, etc.

We want to look at our assertions at this level of detail, and then the research questions will start popping out at us.

Research and Research Questions

Some research--or at least some important discoveries--are not made as the result of a specific research question. We might imagine Newton under the apple tree: the discovery of an idea that appears through the data. Similarly we might imagine that Darwin had no interest in evolution, but was only cataloguing the creatures he observed in his travels.

Such research need not be driven by a question; it comes about serendipitously. And that is somewhat problematic when there is an expectation to publish, to complete research projects and write them up. Can we just wait for a discovery to come to use as we peruse ever more data? That depends on what you want your life to be.

However, if your goal is to finish a research project so as to get a degree or to get published, then it helps to have a research question.

Research questions shape a work and guide it. They provide the focus. Any question could be a research question, but some are better suited to study than others.

A research question comes from a way of looking at the world. It starts with a basic perspective. We each have a fundamental, mostly unconscious set of ideas about how the world works. This then shapes the way we interact with the world and the questions we will ask of it.

We might, for example, believe the Christian creation myth, and set off on a journey to discover and document the many different species that survived the flood on Noah's ark. This would all be consistent with a desire to know all of God's creation that it might be celebrated. Our research question in such a case might be "are there any creatures that have not been documented yet?" or "how many undocumented creature can I find and document to the greater glory of the Lord?" These research questions are unlike the questions that are asked in most universities in America, but they are consistent within a certain world view. In much the same way that the questions asked at a secular university are consistent with their own world view.

Whatever your world view, it is the place from where you start: "I believe the world operates this way," you assert. Usually we have a number of assertions about how things work: "the sun will rise tomorrow; water will run when I turn the tap; e=mc^2; the sperm fertilizes the egg; drinking too much alcohol will make me sick; etc." We have a whole world of assertions; each of us has slightly different ones. Some of them we accept without question, and some of them we're curious about.

We might believe "Process X will improve the quality of my work." It's of obvious value if it's true. As a researcher, it is appropriate to be skeptical. Is this assertion true? And that becomes the research question that you test. The first place to look for answers, of course, is in the literature. Has anyone else asked this question? If so, what was their answer? If not, has anyone asked similar questions? Perhaps no one has asked "will Process X improve work in my field (let's call it 'field A')?" but someone has asked "will process X work in field B (which is related to field A)?"

By starting with what you believe, and testing what you believe, you can move into the logic of your area of interest in search of a question for which you want an answer but which there is no answer to be found.

Maybe you found someone, Dr.Q, who said "Process X will improve work in field A," but their argument was only theoretical, and they had never tested it. This then becomes an assertion that is in need of an empirical test, so you can set up a study to see if it will work in your field.

Maybe you also found someone who said "Process X works in field B if you make adjustments 1, 2 and 3." One thing you could do is to say, "I want to test process X in field A, as Dr. Q suggests, but I want to make adjustments 1 and 2 because of the similarity of fields A and B." Or you could say "I wonder whether the conditions that require adjustment 1 in field B also hold in field A, and if so will adjustment 1 suffice in field A?"

The examples should be viewed as examples of ways of thinking and asking questions. The premises and assertions used as examples could be replaced by any assertion or premise. Once we have a premise, we can start to look at whether it is true, and what reasons we have to believe it, and we can then go from there.

What is Research?

Research can be construed in many different ways, but one way to look at it is the exploration of hypotheses: we believe the world works in a certain way, but we don't know for sure, and we want to test that hypothesis.

So we might, to choose a culinary analogy that may not hold up, for example, have a hypothesis that tofu and pomegranate would go well together in a raw salad.

We may have reasons to believe this--we may, for example, have read a review of restaurant that served such a dish, or we may have read a cookbook that suggested that the flavors would work well together, or we may have read some chemistry/biology textbook that leads us to believe that some chemicals in the two would combine well. We have reasons that we believe the hypothesis. To the extent that such reasons are supported in published literature, and that our idea came from reading the literature, we can add such elements to our literature review. But presumably there is not such a preponderance of evidence to suggest that the exact thing that we're studying is certain (e.g., there are no reports of a tofu-pomegranate salad craze in major metropolitan restaurants, or other indication that our hypothesis has been extensively tested).

In order for us to be doing research, we have to be testing something that is at least in question. As far as empirical science is concerned (whether social science or hard science), a hypothesis, no matter its logical antecedents, is worthy of empirical testing if the given empirical test (or one substantially similar to it) has not been executed. The fact that theory suggests that something will happen is no guarantee that it will.

Things are complex. It's simple to say something about preparing tofu and pomegranate, but that hides a great deal of potential complexity, and potential difficulties. To make up an example, it might be the case that tofu and pomegranate go well only with a third ingredient, but that ingredient is rare, or expensive, or hard to work with in some way, making practical execution of a dish infeasible, even if the theory suggests that it should work.

The scientist looks for this complexity within the simpler statement.
"Tofu and pomegranate will go well together" is a simple hypothesis but it suggests more detailed hypotheses and issues: will they go well together when fried? when raw? when boiled? when mixed with vinegar? Will any problems crop up? What will be done to find out? The scientist looks at the simple hypothesis and asks detailed questions about how that can be true. And the starting place for that exploration is intellectual: what do you know about the situation? what are the important ideas that define the situation? what kinds of theories shape your understanding of the situation? what kinds of questions can you ask about the situation? What details are pertinent? Where do theory and practice diverge? And what is the impact of that divergence? If you're attempting to import a theory or practice from one type of endeavor to another, what differences are there going to be? For example, maybe one heard that pomegranate and chicken was really good, but was vegetarian, so thought of pomegranate and tofu instead. What reasons do you have to believe that the translation will work? What reasons do you have to believe that the translation won't work?

Looking at hypothesis with a critical eye looking for detail, many different questions and ideas and possibilities arise. Practically speaking, each needs to be tested individually. So if you start with a general question, you're looking to find a specific aspect of that question that you can test. If you think pomegranate and tofu will work well together, but that they'll need to have some sort of seasoning, then you'll try preparing some with one set of spices/flavors, and you'll see if that works, and then you'll test with a different set of spices/flavors, and see if that works. You won't throw all the spices in at once, because then all you get is confusion. So with a question that can be fragmented and broken down, you want to seek the different questions that could contribute to answering the main question, and look to answer one of those more detailed questions.

But whatever you're going to do, it starts with your looking at the world and putting forth a hypothesis: "I believe the world works this way," you say to yourself. For example "I believe pomegranate and tofu would taste good together," or "I believe that method X, used in field A, will also be useful in field B, despite some differences between those fields."

You start with having an understanding of how the world works, and an idea that it will work in a certain way. Then you look to see what evidence you have to support that view. If you think the evidence is overwhelming that the view is true, then it's not an interesting research question--but if there is doubt--perhaps there are people who believe that it isn't true, perhaps you doubt yourself; it doesn't matter where the doubt arises--then there is a viable research question: you believe the world will work in a certain way, and then you want to test that idea, and you try to find a way to test that idea. It all starts with how you understand the world and your exploration of the places where you are uncertain and curious.

Quantitative and Qualitative (2)

Quantitative studies are studies that rely on counting and measurement...they rely on numbers. So if you want to do a quantitative study, you have to have some idea of what you are going to count or what you're going to measure.

Qualitative studies could be seen as any study that is not about counting, and not about applying numbers to things. It makes sense to look at this as an examination of the qualities of things, but qualitative studies can also operate in the same logical context as quantitative studies, if the qualitative study is viewed as an opportunity to examine possibilities (see my previous post on this).

The questions asked by quantitative studies and by qualitative studies are different.

Here are some quantitative questions.
"How many are there?"
"How often does this occur?"
"Are these measurements correlated?"
"Which occurs more frequently?"
"Is the rate of occurrence stable or changing?"
"How do these groups differ for a given measurement?"
"What are the likely results of an action, and what are the likelihoods/probabilities?"

Qualitative questions:
"What is it like to do X?"
"What are the experiences of group Y?"
"What are the perceptions of group Y?"
"What are the characteristics of group Y?"
"What has occurred?"
"What are possible consequences of that action?"
"What are different ways that Z can be interpreted?"

What's the point of research?

Yesterday I was writing about the vast unconscious and how it is important to try to bring the premises we reason on into our conscious, reflective mind.

One point which is not often considered consciously is the basic point of research. What is research, and why are we doing it? What is the value in it?

The answer is not simple.
We often think in terms of a correspondence theory of truth--I had a friend tell me that science was to "discover facts"; the fact that he had (and still has) a Ph.D. from UC Berkeley reveals how this theory of truth is accepted by even those with a great deal of education and intelligence. As a natural scientist there is great value in seeing the research project as one depending on a correspondence theory of truth, but as science deals with the quantum world, and many different aspects of physics, correspondence theories become more and more problematic.

If you are working in the social sciences, in literature, history, the arts and many other fields, the idea of a correspondence theory of truth will serve you less well.
The very idea of a correspondence theory of truth--indeed, the very idea of truth--has been challenged by many philosophers of reknown over the past century. Michel Foucault and Jacques Derrida are two famed exemplars of this general debate. It would be considered naive, I think, at this point to attempt to work with such a vision in some fields, unless one were well-prepared to acknowledge and rebut the arguments of those opposed to correspondence theories. In the absence of a correspondence theory, one must seek some other theory on which to base research--undersanding other theories of research, e.g., hermenutics or phenomenology, can help bring focus to the purpose of research.

By bringing into conscious discussion the premises on which you base your research, and on which you write it up, ou being to develop a basis on which your whole work can be structured in a coherent fashion. By understanding what the purpose of your research is, and by understanding what you are trying to show, and how you hope to show it, you build a framework for presenting the work, and for working in the different pieces necessary to make the research proejct work as a whole.

I can't answer the question of what research is for you, because I think that different kinds of research will have different purposes. If you can give a clear purpose for the work, it can often help stay away from questions of epistemology--for example, if you assert that your research is intended to help clinicians work with a certain population, you don't need to worry about theories of truth. Or if you are going to develop an algorithm, or some technological device; or if you're doing any of a number of things, you can focus directly on your immediate purpose.

On the other hand some projects have a much more difficult time stating a purpose. For example a history. What is the purpose of a history? Especially we might ask what is the purpose of a history, when we know that histories inevitably reflect the historian's concerns? Having an understanding of what the purposes of other writers in history can help--how are these other writers grounding their work and giving it a sense of purpose? We can always present a history as a cautionary tale, or an educational study that helps us understand situations that we want to ameliorate. Or we could take a history as a story that educates us about a population: what are these people like and why? In that context, we might look at a piece of evidence as suggestive of ideas and attitudes that we have no direct evidence of--thus we might look at a bureaucracy and its policies as indicating or suggesting certain motives. We might not be able to substantiate the existence of the motive, but we can still infer it and indicate how the evidence is suggestive. Such techniques might please some and infuriate others. But my main point returns: as we make ourselves consciously address the question "what's the point of research?," we begin to have answers that can help us structure our research and writing efforts. To the extent that we develop a scholarly understanding that generates a theory of research, we can save ourselves from losing time wondering what it is that we're trying to do, ad we can also give ourselves an understanding that helps us explain our intentions to others--and sometimes a good verbal explanation can really change how your professor reads your work.