osewalrus (osewalrus) wrote,

One of many reasons I regard surveys as unreliable

This article by George Mason econ professor Daniel Klien does not just highlight the "my bias" effect as noted by Klein. It also points to a semantic problem. Survey questions -- especially on complicated issues -- are often not very reliable because of interpretation issues. Efforts to address these with gradation (e.g., range from "strongly agree" to "strongly disagree") have limited utility. This is why research firms that make money based on their success use so many different tools to gauge opinion.

Many of the problems using surveys are well known in research circles and serious researchers often take different approaches to try to compensate. There is also a great deal of literature now around the interpretation problem -- and not just in opinion polling research. This has long been a huge problem in education.

To get back to the specific issue, the survey used fairly general, one sentence questions to test people's "basic" economic knowledge. For each question, the respondent answered on a range from "Strongly agree" to "Strongly disagree" with "not sure" and "other" also available (as well as not answering the question). To attempt to address the complexity of the underlying problem, Klein and his co-author selected one answer as clearly wrong, and only counted the number of "clearly wrong" answers as indicators of ignorance/belief in something untrue.

The problem is that describing economic phenomena is rarely as simple as, say, describing physical phenomena. For example, while I can come up with ways to produce anomalous results, the general statement: "gasses expand when heated" would "true" whereas "heavy objects fall faster than lighter objects" is "false." Sure, I can play games and come up with scenarios where this won't happen. But it works as a general rule and most people would agree the statements are true. Indeed, you have to work to make them not true.

Now consider two questions from the surveys used here. One is "pro-Libertarian" and the other is "Pro-Progressive." The answer designated as "clearly wrong" and thus displaying willful ignorance, in the opinion of the authors, is given in quotes.

"rent control leads to housing shortages" (disagree)

"A dollar means more to a poor person than it does to a rich person" (disagree)

The problem with these questions is that they both encapsulate a large number of complex concepts and are subject to very different readings -- without ever testing one's knowledge of economics. Take question #1 on rent control. Directly? Indirectly? Over time? All other things being equal? The simplistic logic used by the author that all other things being equal, a control on revenue decreases supply is hard to apply in something as complex as a housing shortage. Heck, even if you agree with the premise, does it cause a "housing shortage" or a shift away from rental property to forms of purchase, such as condos? Is there a secondary market that effectively eliminates the disincentive? Is the business still so profitable that the disincentive is negligible? What does the question even mean?

The same semantic confusion applies to question #2. Based on the wrong answer, the question appears to be testing for the basic concept that when you have less income you become much more sensitive to price. But it is just as easy to read this as a question about absolute purchasing power (in which case a dollar is a dollar is a dollar).

What makes this so problematic is that public opinion and public policy get shaped by this crap. I run into it every day. It would be nice if people appreciated the limits of these things. But of course, that's just my opinion bias showing.

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your IP address will be recorded