What’s the First Rule about John Bargh’s Data?

Answer: You do not talk about John Bargh’s data.

I went on hiatus with back to school events and letter of recommendation writing.  However, I think this is a good story that raises lots of issues. I need to say upfront that these opinions are mine and do not necessarily reflect anyone else’s views. I might also be making a big enemy with this post, but I probably already have a few of those out there. To quote the Dark Knight: I’m not afraid, I’m angry.

Background: Bargh and Shalev (2012) published an article in Emotion where they predicted that trait loneliness would be “positively associated with the frequency, duration, and preferred water temperatures” of showers and baths (p. 156). The correlation between self-reported loneliness and self-reported “physical warmth extraction” from baths/showers was .57 in Study 1a (51 undergrads) and .37 in Study 1b (41 community members). This package received media attention and was discussed in a Psychology Today blog post with the title: “Feeling lonely? Take a warm bath.”

We failed to replicate this effect three times using three different kinds of samples. Our combined sample size was 925 and the overall estimate was – .02. We also used Bayesian estimation techniques and got similar results (the mean estimate was -.02 and 70% of the credible estimates were below zero). Again, the opinions expressed in this blog post are mine and only mine but the research was a collaborative effort with Rich Lucas and Joe Cesario.

[As an aside, John Kruschke gave a workshop at MSU this past weekend about Bayesian estimation. It was engaging and informative. This link will take you to his in press paper at JEP: General about the Bayesian t Test. It is well worth your time to read his paper.]

We just sent our paper off to get trashed in the undergo the peer review process.  However, the point that I want to raise is more important than our findings. Bargh let Joe Cesario look at his data but he forbids us from talking about what Joe observed. So a gag order is in place.

I think this is bull****. There is no reason why there should be a veil of secrecy around raw data. How can we have an open and transparent science if researchers are not allowed to make observations about the underlying data used to make published claims?

I doubt very much that there is even a moderate association between trait loneliness and showering habits. It might not be zero, but it is hard to believe the population value is anything around .50. Consider Figure 1 in Richard, Bond, and Stokes-Zoota (2003, p. 336). This is a summary of 474 meta-analytic effect sizes in the r-metric across social psychology. Richard et al. noted that 5.28% of the effect sizes they summarized were greater than .50. Viewed against this distribution, the .57 from Bargh and Shalev’s Study 1a is unusual. A .57 correlation is something I might expect to see when calculating the correlation between two measures of very similar constructs using self-report scales.

So before more data are collected on this topic, I would hold off on making any recommendations about taking warm baths/showers to lonely people. To quote Uli Schimmack: “In the real world, effect sizes matter.” I think replication and transparency matter as well.

Coverage of the Bargh and Shalev (2012) Study:

http://www.psychologytoday.com/blog/choke/201201/feeling-lonely-take-warm-bath

http://www.scientificamerican.com/podcast/episode.cfm?id=hot-baths-may-cure-loneliness-11-07-02

About these ads

29 thoughts on “What’s the First Rule about John Bargh’s Data?

  1. There are types of agreements for data sharing. I suspect this was the type of agreement they had: to only look at the data for internal use. I think it’s legitimate. One can’t get data and later change the agreement do what he wants with it- this is unreliable. People are sensitive and I think nobody will want to share data if agreements are not kept.
    To my view you present your “story” online in a way to create a negative propaganda. Why would anyone share data with your lab in future.
    I also refer you to SPSP formal document. Some people criticize it but this is the guidance we have.

    http://c.ymcdn.com/sites/spsp.siteym.com/resource/resmgr/files/task_force_on_responsible_co.pdf

    • Hadar, according to Wikipedia (“Scientific Method” entry)

      Another basic expectation is to document, archive and share all data and methodology so they are available for careful scrutiny by other scientists, giving them the opportunity to verify results by attempting to reproduce them.

      and

      To protect against bad science and fraudulent data, government research-granting agencies such as the National Science Foundation, and science journals including Nature and Science, have a policy that researchers must archive their data and methods so other researchers can test the data and methods and build on the research that has gone before. Scientific data archiving can be done at a number of national archives in the U.S. or in the World Data Center.

      Isn’t the whole point that data should be available to be shared and scrutinized? Isn’t that point frustrated if the person providing the data stipulates that the results of the scrutiny must remain secret?

      And what legitimate reason would there be for anyone to not WANT others to describe and discuss their data?!

  2. No negative propaganda was intended. I was just reflecting my opinions on my blog. To my mind, the point of data sharing is to facilitate transparency. I think my team should be allowed to talk about our impressions. But obviously people disagree. If/when our paper is accepted, I will post the de-identified data from our studies.

  3. Emotion is an APA publication, and APA’s code of conduct is pretty clear on data sharing:

    After research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose, provided that the confidentiality of the participants can be protected and unless legal rights concerning proprietary data preclude their release…

    The key phrase is “verify the substantive claims through reanalysis,” which is exactly what Brent is talking about in this blog post. Bargh could withhold the data if it was impossible to de-identify or if it were proprietary. But neither of those is plausible here, and anyway, he did provide it. Once that happens, Bargh has no grounds to prevent someone from saying whether they think the data support the published claims.

  4. The APA also says:
    (b) “Psychologists who request data from other psychologists to verify the substantive claims through reanalysis may use shared data only for the declared purpose. Requesting psychologists obtain prior written agreement for all other uses of the data”

    Using the data for publication should get agreement in advance and not retrospectively.
    This is also written in SPSP guidlines.

    Also Cesario was interviewed in a magazine trying to undermine Bargh, where he was he was implying grant agents should not fund his research (Bruce Bower’s paper). This blog uses sentences taken from that paper (the last sentence). The easiest thing is to get zero results…so the whole thing makes me feel that it’s nothing but a well directed effort and publicity to undermine Bargh. But of course, it’s my subjective opinion only.

    • Sheesh, Yohan, you think it is OK for someone whose work was funded by the public and published in a society-run journal to say that their raw data cannot be publicly described and written about?

      Why would it serve the interests of science to allow such behavior? To the degree funding agency and society policies speak of respecting limits on the “use” of shared data, I believe this was only to stop Party B from obtaining Party A’s raw data and writing their own research papers based on this work without sharing credit with Party A who after all did all work of collecting the data. I cannot believe any funding agency or scientific society wished to say it was OK for someone to say “you can look at my raw data but you cannot write about whatever you might find noteworthy about these data.”

      Also if you look at the quote from Cesario, he was _defending_ priming researchers and not attacking them–not that this matters in the slightest.

  5. I think the competing quotes from APA guidelines illustrate one big part of the problem: APA (and funding agencies) have provided no concrete guidance about these issues. All we get is the vague idea that “data sharing is good” with no real requirement that researchers–even those who get federal funding for their work–act in accordance with this principle. These organizations should take a stand about whether data sharing is essential to the science, or they should not pretend to support it.

    It is worth noting, however, that those who cite the specific wording of the guidelines in support of Bargh’s decision not to allow us to talk about the raw data rarely cite any compelling principles for WHY the data should remain veiled in secrecy. The data are not identifiable, so there are no ethical concerns. We were also not presenting new findings from the data, so we were not taking advantage of Bargh’s work for our own benefit. Rather, any information that would be revealed would be in the service of understanding an already published result. Why would someone not want that discussion to take place?

    Perhaps one could argue that those who wish to reanalyze the data will present their analysis in a misleading way. However, if everything is open and accessible, then the community of scientists can judge who is right. Secrecy prevents this discussion from taking place.

  6. This is directly from the Bower article. I don’t think the claim that I took a sentence from him has merit. We can also let others form their own impressions of the Cesario quote. People can judge for themselves:

    “Bargh has applied to several funding agencies for grants to study possible applications of priming to clinical work. One proposal would examine whether some alcoholics prefer drinking hard liquor because it provides a rush of physical warmth that temporarily lessens loneliness and fires up conviviality. If so, warm packs or other high-temperature cues could be tried as aids in alcoholism recovery programs, Bargh suggests.

    So far, his proposals have been turned down by funders. A “wave of negativity” about priming effects may partly account for grant reviewers’ coldness toward Bargh’s research proposals, remarks psychologist Joseph Cesario of Michigan State University in East Lansing. Cesario, who studies how responses to the same prime vary in different physical settings, says that priming critics are threatened by evidence that complex thinking doesn’t require conscious thought.

    Shanks admits having felt uncomfortable at first with claims that a few unobtrusive words or images can reshape behavior. Ensuing studies, rife with what he sees as flaws, have only solidified his misgivings. Included are the recent findings on unconscious links between social and physical warmth.

    Without independent confirmations, Shanks sees no future for clinical applications of warmth primes. “It’s puzzling that so little attention has been devoted to replicating priming results,” Shanks says. Until such replications are done, he recommends serving no prime before its time.”

  7. The title of this blog, written by a very young scholar: undergraduate or graduate student, the accusation of Bargh who is one of the most influential social psychologists of our generation, who was also helping this lab by giving them his results, and the expressed anger of the experimenter all teach us of a possible sever experimenter expectation bias that probably influenced the results. I know of many conceptual and direct replications for the warmth and cold studies showing us a very different picture , so I severely doubt these findings.

    • Yeah please do tell us more about these successful direct replications, Adi. Even if they’re not published, if someone stands behind them you should ask them to go ahead and post them on FigShare or PsychFileDrawer or their own blog if they have them.

      Also you didn’t offer any explanation for why the one of the most influential whatever it was would have any problem with people discussing his raw data… scientists are supposed to welcome scrutiny of their work, isn’t that right?

  8. Adi:

    Please provide these citations with effect size estimates. Also make the case about how we could have influenced the results. This is a survey study using very nearly the same items as the original Study 1a and 1b.

    Please refrain from insulting me. I am certainly not an undergraduate. I approved your comment to be transparent but I do not appreciate personal attacks. This is about data and about ideas.

  9. All professional guidelines and regulations for federally funded research, of which I am aware, including those posted above, MANDATE data sharing, and explicitly allow reanalysis of data (i.e., a repeat of the reported analysis and obvious minor variations thereon). The ambiguity, if there is any, is that none of the agencies make any mention of what to do if/when data sharing or use is refused. Surely any federal funders of Bargh’s work, as well as the original journal, should be pissed off if they hear that he has instituted a gag order regarding his original data. It is not a major ethical lapse, but it is a clear violation of their rules… and the rules are in place for good reason.

    Also, .57 is an insanely large correlation for such arbitrarily related variables. I don’t know exactly what the reliability of the proposed measures are, but if you correct for measurement error, this result must imply a true-score correlation near .80. Does anyone really believe that loneliness and shower temperature are correlated on par with height and weight?!?

    Now I have no bone to pick in the fight over whether or not the relationship is real, but this type of clearly anomalous result shouldn’t have gotten through peer review at a top journal. We seriously need to reanalyze how we judge “the best” papers in our field.

  10. I want to be clear about the source of my anger. I am angry about the gag order and my impressions of the data from Study 1a in Bargh and Shalev. I am also angry that the others do not share my belief in the importance of transparency and openness. I am not angry about the general idea of priming.

    The Bargh apologists make vague references to replication studies but neglect to cite specific examples of exact replications. Instead they try to accuse me of plagiarism or make ill-informed judgments about my CV and length of time in the field. These are not serious arguments. If you bother to look at my publications, you will see that I have some expertise in self-report methods and survey research. I am not an undergraduate student who just did an honors thesis.
    We asked to see the data to understand our initial failure to replicate the Bargh and Shalev findings from their Study 1a and 1b. We then detected something odd about the original data in Study 1a. We wanted to talk about the distributions of their variables in our original write-up and Bargh unloaded on Joe when we sent him a document for comments. We have tried to communicate with him but he has not been responsive and he has been unwilling to engage in a serious dialogue about his data.

    To be clear: I want to talk about means and SDs and whether or not variables deviate from normality in Study 1a. The fact this information was not in the original paper is a travesty to my mind. See my 2009 PSPB paper with Debby Kashy, Robert Ackerman, and Dan Russell for more about my perspective on data reporting. If researchers do not think this information is relevant, we are in bad shape as a field.

  11. Just to clarify for accuracy: I was *defending* Bargh and behavioral priming effects in my interview with Bower. Although I believe he made an error in attributing the comment about funding agencies to me (I don’t remember saying anything of the sort), the entire interview was me defending Bargh against claims that his (and others’) effects were due to experimenter expectancy. I also defended behavioral priming effects against this claim on Ed Yong’s blog, and I’ve argued that failures by other labs to replicate experimental effects are not as informative as we might like to think (e.g., on Daniel Simons’ g+ post here: https://plus.google.com/u/0/107191542129310486499/posts/VJH8wXxxc3f). Most of my own work falls under the general heading of “priming,” including work on priming social categories and automatic behavior specifically–exactly the same as some of Bargh’s work.

    I can only speak for myself (and not my collaborators), but there is no way you can write off my criticism of data sharing policies as antagonism against behavioral priming effects in general or Bargh’s work in particular. That just makes no sense at all.

  12. I also want to add one detail to the events described here that may clarify the situation (and may question the appropriateness of the phrase “gag order”). Bargh sent us the data under the conditions that (a) we would use the data only to understand our failure to replicate and (b) we would not share the raw data with anyone else. Whether one agrees with setting such conditions or not, this agreement was in place when the data were sent. So I don’t know that his actions are really violating any APA regulations.

    The real meat of the issue comes in defining what constitutes “raw data”, what is needed to understand a failure to replicate, and what is owned “privately” by the researcher vs. publicly once some aspect of a dataset has been published. In my view (and I believe this is shared by Brent D. and Rich L.), once an effect is published then anything that is relevant to understanding that effect (and failures to replicate) must be part of the published body of scientific knowledge. In the case of a correlation, this includes distributional information about the variables that went into that correlation. I don’t think it advances our science any to keep private the nature of the variables but insist that an effect dependent on those variables be accepted by all. (As Brent notes, if proper descriptive reporting was routinely done, then this would be less of an issue.)

    • Joseph,
      I think that in the past, behavior like you describe would fall between mundane and laudable. My intuition is that 80, 50, 30 and even 15 years ago, someone sharing raw data like this might well have been a sign of great generosity. In part, that was due to how inconvenient such sharing was, i.e., more than a quick email, and how impractical it would have been to make data “public”. However, my understanding of the current ethical guidelines, made explicit by professional societies and, more importantly, by journals, and even more importantly, by federal funding agencies, essentially state that raw data from published, federally funded projects should be available upon request and can be used for more or less any function. We are ominously close to mandates that all federally funded data be made publicly available.

      Again, I don’t think it is major ethical lapse, and at any rate I applaud you for sticking to the agreement under which you received the data. However, I suspect at some point the journals and the feds will get tired of hearing stories like this.

  13. This blog made me feel a bit less lonely, but I am still going to take long hot showers.

    It would be interesting to consult a lawyer and see whether Bargh has any legal standing to make his request (gag order). If not, you can just go ahead and use the data and say what ever you want. If this goes to trial, we could have lawywers figure out what APA guidelines really mean. Meanwhile, we should all request Bargh’s data and complain to APA if he violates their request to share data used for an APA publication.

  14. Hi Brent – I’m curious. What is the exact wording to which you agreed? If Bargh explicitly stated that you couldn’t share ‘raw’ data, then I don’t see what it would hurt to share – on this blog – some summary data, like frequencies. It could help us all to understand the failure to replicate. Providing understanding through pure objective means (like summary data) without any commentary (which could be construed as subjective or slanderous) seems to be one of the less objectionable things that any honest researcher could ever be subject to.

    • Hi Paul,
      Your question made me try to retrace all of the events in this story. Basically, I think different players in this drama have different interpretations of the original agreement and what counts as a summary statistic as opposed to raw data. Joe Cesario had the interactions with Bargh and got the data around June 4. I believe it was understood that Joe could run whatever analyses he wanted on their data for comparison purposes (i.e., to compare our Study 1 with his results). The files themselves were to be kept private so Joe did all of the analyses. Joe was not to post the raw data on any website.

      Joe showed Brent and Rich output from statistical packages. We (Rich and Brent) have not seen the raw data as in the underlying “spread sheet” but we have seen frequency distributions. In the process, we identified odd things about one or more of the focal variables in Study 1a. We had questions about those variables but the communication patterns with Bargh were fractured in July and early August.

      We finally wrote up a brief document that described our failure to replicate the effects with our college student sample (I took the lead writing this up). I suspected that any failure to replicate a Bargh effect would be subject to withering criticism. So I wanted to establish that there are solid grounds to be suspicious of the original .57 correlation.

      Thus, in an appendix to that document, we described the issues we had with Study 1a as a way to place our failure to replicate in a broader context. We basically said the data in their Study 1a are odd and this probably explains why we got different results. We included the Means, SDs, Min, Max, and Modes of the primary variables in this appendix. We also talked about some of the distributions and reported how the three bathing/showering items correlated with each other. I think this kind of distributional information is something that should have been in the original report. I do not consider this raw data.

      We sent this document to Bargh for comment around August 13. I then drafted a blog post and I raised the point that someone should try another replication using mTurk or something like that mechanism for getting non-college student data. I realized this was silly – we should do this kind of thing ourselves. I then submitted the IRB and we collected Studies 2 and 3.

      Joe got a strongly worded response back from Bargh on August 22 that indicated that we could not post anything at all about their studies in any document. Joe tried to engage in further dialogue around August 27 and again on September 13. He did not receive a response from either email.

      So this is a long-winded way of saying that there is an apparent disagreement about the original conditions under which Joe got the data and what he could do with the data. However, once Bargh told him that we could not post anything about their studies we dropped the appendix. I coined the “gag order” term at around this point in time. Joe himself does not like the term, I don’t think.

      • Well, then I guess the ball would be in Joe’s court, so to speak. If his understanding was that he could share summary data but not raw data then something like elaborating on some frequencies would be well within those guidelines.

        If the August 22nd communication is something that simply says you can’t post anything at all, including summary data, then that provides another out. He’s certainly not been smart enough to say that you can’t share his *correspondence*.

        This may be deeper than you want to dig in on this, but I would imagine that posting an email from him saying – essentially – that “John Bargh is above the spirit of academic honesty when it comes to data transparency” would be a pretty hard thing for all his supporters (or aliases) in this comment thread to rebuke.

  15. Paul: This is not something Joe will do. He is going to honor JB’s request. I have tremendous respect for Joe’s integrity. My hands are tied. I hope someone else gets the data under a different arrangement.

    • Well, for what it’s worth, I wonder how many of us in the same situation would be able to muster half the integrity that both of you have demonstrated. I’m a little happy that I *don’t* have Bargh’s data, because I don’t know I’d be able to hold my tongue quite as well.

      You’re like the Batman of psychological science – the hero we deserve but not the hero we need. Hopefully someday someone without a horse in the race will find a way to stand up to the Bargh’s and help put these continuing dark times behind us.

      • When I re-read my last comment I’m not sure it comes across – as much as I meant it – that I really support what you’re trying to do.

        Science without transparency is not science. It’s less than pseudoscience, nothing better than flim-flam, celebrity, and snake oil.

  16. Pingback: Transparency in Social Psychology | David J. Johnson

  17. Pingback: Evolutionary Psychology & Embodied Cognition | Evolutionary Psychology

  18. Pingback: Don’t blame Milgram | funderstorms

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s