Surveys, Studies and Experiments - How Far Should Our Trust Go?

[barr08]

This has been bugging me for a good part of my adult life, so I figured I would pose this question to the community. It seems like every day, the results of a new scientific experiment or study are posted on the Internet, or presented on the radio or TV. These studies range greatly in topic, but I included a few examples of what I am talking about (from the past day of Digg posts):

Being Bad is Good for Your Health
Teens With More Mature Brains Act Out More
How Men and Women Lie Differently
Tightwads and Spendthifts Attract

Other studies, off the top of my head, include the benefits of certain foods (coffee, chocolate, beer, wine), relationships between two random factors (ex: binge drinkers are happier and healthier), or studies of the human body (one can function on only 2 hous of sleep a night)

These are just examples of the hundreds of studies I have seen in my days of journeying the Internet for news and information. Personally, however, I am a skeptic. I have trouble deciding when something seems legit and practical, so I take all of these studies with a big grain of salt.

I guess my question is, besides the obvious (a medical journal is probably legitimate, 'pro-at-science.blogspot.com is probably not), how seriously should one take these studies. If something applies to my life, might it be worth keeping these "new facts" in the back of my head on an everyday basis.

I am really interested to see what you all have to say!

 

[Iscariot]

It is not a single study but the aggregate data of studies that should weigh one's opinions. This isn't a problem with studies so much as it is a problem with science reporting. Too many verdicts being rendered while the jury is still out.

 

[barr08]

It is not a single study but the aggregate data of studies that should weigh one's opinions. This isn't a problem with studies so much as it is a problem with science reporting. Too many verdicts being rendered while the jury is still out.
[/URL]

Another thing I see - flip flopping among the issues. One day coffee is the best thing you can drink, the next day it takes 10 years off your life. This makes it even harder to make an informed decision.

 

[mkrishnan]

I guess my question is, besides the obvious (a medical journal is probably legitimate, 'pro-at-science.blogspot.com is probably not), how seriously should one take these studies.

It is not a single study but the aggregate data of studies that should weigh one's opinions.

In addition to Iscariot's comment, which is certainly the well accepted idea among scientists, usually blog and newspaper postings like the ones you posted take peer-reviewed (you should really be looking for peer-review, and not the fact that the journal is "medical" ... for instance, some of the social psychology topics you mentioned are not generally published in medical journals) and try to make them accessible to the general audience. Usually there are a lot of inferences involved in the process and they're not always warranted. If you can, if you're really interested, try to find more direct literature like the actual journal articles, that really explains what was done so that you can understand what conclusions are drawn. Also look to multiple sources for confirmation... generally, it's rare for many major, high-quality publications like Forbes and the NYT to tell you to do something based on a research study, with none of them dissenting, unless the research itself has reached a level of fairly wide acceptance by scientists.

The other thing is that this whole question of how to change your actions based on research is a really troubling one.

Take the example of multivitamins. Many people take vitamin pills prophylactically -- that is, they take them when they are not sick or deficient in the belief that the vitamins will keep them in better general health or prevent illness. They don't do this against doctor recommendations -- on the contrary, many physicians encourage this.

There is almost no scientific evidence to support this.

That scientific evidence in fact, if anything, has on numerous occasions suggested that there are not only no benefits, but possibly dangers associated with this practice. Now that has not been generally accepted as of yet, and there are contrary studies with respect to the actual presence of harm associated with vitamins.

But the studies are pretty clear on the absence of benefit (in this case, that aggregation across many studies has been done... for instance).

Even at that level, the policy decision is hard to make. To date, across the scientific community, this evidence hasn't caused a systematic change in the way doctors suggest for or against vitamin consumption.

Another good example is circumcision to prevent HIV transmission... the data is now considered fairly conclusive that circumcision confers a substantial benefit in this regard. There are still many other issues to consider, for instance, if one were to want to circumcise themselves or their child for this reason, and there's a very active public policy debate over whether to start recommending this in the US, for instance, where circumcision rates have fallen off, albeit HIV infection rates are also pretty low here.

So that's just an example of how difficult it is to actually figure out what you should do -- even if you know all the scientific studies very well, you still make moral/ethical decisions on which the science does not inform you, and ultimately also act on your hunches and heuristics.

 

[barr08]

Excellent. Let's see if I am taking this in correctly...

The other thing is that this whole question of how to change your actions based on research is a really troubling one.

Take the example of multivitamins...

In my eyes, I have lived long enough (23 years) to know the proper and best way to handle everyday decisions and situations...let's say...10% of the time. I have a lot of learning ahead of me, and even though I may think I know everything, like most people my age, I definitely do not.

So when I hear these things, especially things that could apply to my life (choosing a cup of coffee instead of an energy drink in the morning), I take notice. While I base many of my day-to-day decisions off of life experiences, I just know there is more information out there than what I've picked up myself - and that information could make a difference in my life.

So I see a study like this, and I don't neccesarily change my life drastically. I do, however, keep these ideas in the back of my head. I feel like this is far more dangerous than it is helpful, especially given the average credibility of articles found online.

I would assume the average person may treat this data in the same way, perhaps even with less skepticism.

So that's just an example of how difficult it is to actually figure out what you should do -- even if you know all the scientific studies very well, you still make moral/ethical decisions on which the science does not inform you, and ultimately also act on your hunches and heuristics.

I agree. Even with full knowledge of sources and evidence, I still am more inclined to do things the way that I have taught myself to do them. The risk here is that I could be missing out on something that could make my life noticeably better.

 

[maflynn]

I don't trust any of those studies.

Just because one study says coffee is bad for you, there's a conflicting one that states the benefits of coffee.

I figure everything in moderation, taking too much of anyhting will not be overly healthy and I ignore all of those silly scientific studies.

 

[Mousse]

I take all these so called studies with a grain of salt, unless it's undergone extensive peer review. Remember the cloning break through from South Korea a few years back?
As for statistics...they're worthless IMO. I'd rather come up with my own conclusion based on the raw data.

 

[trule]

Think of it this way:

1/ if it were not for all these studies there would be lots of empty pages in newspapers etal.

2/ these studies are funded by people who generally want a certain outcome.

3/ scientists are not immune to fashion and trend.

4/ its much easier to control a population if they spend there time fearing trivial factoids rather than thinking and asking questions.

If you must worry, then worry about something useful.

 

[pooky]

I take all these so called studies with a grain of salt, unless it's undergone extensive peer review. Remember the cloning break through from South Korea a few years back?
As for statistics...they're worthless IMO. I'd rather come up with my own conclusion based on the raw data.

Using what? Magic? Or are you somehow capable of interpreting thousands of numbers in a way that has escaped the rest of us? Most of us can't even take an average (which is, by the way, a statistic, albiet a very simple one) on more than a few numbers. We need statistics to organize and summarize data. It is legit to criticize this or that person's application of statistics to a particular problem, but to throw it out as useless and claim that you can somehow deduce the answer directly from the raw data is beyond ridiculous.

 

[Peterkro]

I don't trust any of those studies.

Just because one study says coffee is bad for you, there's a conflicting one that states the benefits of coffee.

I figure everything in moderation, taking too much of anyhting will not be overly healthy and I ignore all of those silly scientific studies.

Everything in moderation especially moderation.

 

[mkrishnan]

Using what? Magic? Or are you somehow capable of interpreting thousands of numbers in a way that has escaped the rest of us? Most of us can't even take an average (which is, by the way, a statistic, albiet a very simple one) on more than a few numbers. We need statistics to organize and summarize data. It is legit to criticize this or that person's application of statistics to a particular problem, but to throw it out as useless and claim that you can somehow deduce the answer directly from the raw data is beyond ridiculous.

Statistics are only worthless when one doesn't understand how they're generated or what they mean. I'm certainly not implying that the thing to do is to throw one's hands up at scientific progress altogether and just wing it.

If one actually reads scientific findings and understands scientific method, the situation is not uniformly like the hyperbolic "coffee" example. Yes, there are fads. Most of the scientific community knew they would be fads to begin with. That's where the popular press is dangerous. No one in the scientific community has been coming out saying that coffee kills people or offers salvation. The reality is more complicated -- that there are good and bad things about coffee, depending on what aspects of its impact are being studied.

I think it's irresponsible to walk away from science altogether just because it doesn't provide black and white, oversimplified answers to one's liking.