Celebrated Same-Sex Marriage Acceptance Study Now Retracted for Fake Data

From Retraction Watch

5/20/15

Author retracts study of changing minds on same-sex marriage after colleague admits data were faked

In what can only be described as a remarkable and swift series of events, one of the authors of a much-ballyhooed Science paper claiming that short conversations could change people’s minds on same-sex marriage is retracting it following revelations that the data were faked by his co-author.

professordonaldpgreen

Donald Green, of Columbia, and Michael LaCour, a graduate student at UCLA, published the paper, “When contact changes minds: An experiment on transmission of support for gay equality,” in December 2014. The study received widespread media attention, including from This American LifeThe New York Times, The Wall Street JournalThe Washington Post,  The Los Angeles Times, Science FridayVox, and HuffingtonPost, as LaCour’s site notes.

michaeljlacourUCLA

David Broockman and Joshua Kalla, graduate students at University of California, Berkeley, were two of the people impressed with the work, so they planned an extension of it, as they explain in a timeline posted online yesterday:

As we examined the study’s data in planning our own studies, two features surprised us: voters’ survey responses exhibit much higher test-retest reliabilities than we have observed in any other panel survey data, and the response and reinterview rates of the panel survey were significantly higher than we expected. We set aside our doubts about the study and awaited the launch of our pilot extension to see if we could manage the same parameters. LaCour and Green were both responsive to requests for advice about design details when queried.

Earlier this month, they began a pilot of their extension. They soon realized that

The response rate of the pilot study was notably lower than what LaCour and Green (2014) reported.

When Broockman and Kalla contacted the firm they thought had performed the original study upon which the Science paper was based,

The survey firm claimed they had no familiarity with the project and that they had never had an employee with the name of the staffer we were asking for. The firm also denied having the capabilities to perform many aspects of the recruitment procedures described in LaCour and Green (2014).

After finding several other irregularities, the pair contacted Green, who was concerned, and they also asked Yale political science professor Peter Aronow to join their work. By May 16, the team had found other irregularities, and sent them to Green, who reviewed them and on May 17 agreed that

a retraction is in order unless LaCour provides countervailing evidence. Green also requests this report be made public concurrently with his retraction request, if this request is deemed appropriate.

Over the next two days, Green confronted LaCour and told the team that LaCour had

confessed to falsely describing at least some of the details of the data collection.

Green then added a note on May 19 to his website saying the paper was retracted, and submitted a retraction letter to Science:

I write to request a retraction of the above Science report. Last weekend, two UC Berkeley graduate students (David Broockman, and Josh Kalla) who had been working on a research project patterned after the studies reported in our article brought to my attention a series of irregularities that called into question the integrity of the data we present. They crafted a technical report with the assistance of Yale professor, Peter Aronow, and presented it to me last weekend. The report is attached. I brought their report to the attention of Lynn Vavreck, Professor of Political Science at UCLA and Michael LaCour’s graduate advisor, who confronted him with these allegations on Monday morning, whereupon it was discovered that he on-line survey data that Michael LaCour purported to collect could not be traced to any originating Qualtrics source files. He claimed that he deleted the source file accidentally, but a Qualtrics service representative who examined the account and spoke with UCLA Political Science Department Chair Jeffrey Lewis reported to him that she found no evidence of such a deletion. On Tuesday, Professor Vavreck and Michael LaCour for the contact information of survey respondents so that their participation in the survey could be verified, but he declined to furnish this information. With respect to the implementation of the surveys, Professor Vavreck was informed that, contrary to the description in the Supplemental Information, no cash incentives were offered or paid to respondents, and that, notwithstanding Michael LaCour’s funding acknowledgement in the published report, he told Professor Vavreck that he did not in fact accept or use grant money to conduct surveys for either study, which she independently confirmed with the UCLA Law School and the UCLA Grants Office. Michael LaCour’s failure to produce the raw data coupled with the other concerns noted above undermines the credibility of the findings.

I am deeply embarrassed by this turn of events and apologize to the editors, reviewers, and readers of Science.

Green tells Retraction Watch:

…Michael LaCour attended my summer workshop on experimental design in 2012 and proposed at that time a project that involved both canvassing and internet surveys.  It sounded to me too ambitious to be realistic for a graduate student but in principle worthwhile.  I later introduced him to Dave Fleischer, who heads up the LGBT canvassing operation in Los Angeles, and they struck up a collaboration.  Several weeks after the canvassing launched in June 2013, Michael LaCour showed me his survey results.  I thought they were so astonishing that the findings would only be credible if the study were replicated.  (I also had some technical concerns about the “thermometer” measures used in the surveys.)  Michael LaCour and Dave Fleischer therefore conducted a second experiment in August of 2013, and the results confirmed the initial findings.  Convinced that the results were robust, I helped Michael LaCour write up the findings, especially the parts that had to do with the statistical interpretation of the experimental design. Given that I did not have IRB approval for the study from my home institution, I took care not to analyze any primary data — the datafiles that I analyzed were the same replication datasets that Michael LaCour posted to his website.  Looking back, the failure to verify the original Qualtrics data was a serious mistake.

According to his website, LaCour will become an assistant professor at Princeton University in July. [Update: As of 8 a.m. Eastern on 5/20/15, that mention had been removed from his site, but it is still available on the Google cache version.] We’ve contacted him for comment, and will update with anything we learn.

Update, 10:30 a.m. Eastern, 5/20/15: Broockman tells Retraction Watch he agrees that this was remarkably swift and transparent:

I think there’s a couple reasons why. First, the study’s findings had huge implications for people who were trying to advance the cause of equality and have changed how advocates do their work. Every minute we knew the truth and did not disclose it really was a lie by omission to the advocates out there. There was some element of time sensitiveness. Second, the nature of the claims made in the article about the scope of the study — the survey’s scale, its funding, the number of bills that would have to have been incurred to pay for it, etc. — meant that we expected it would have been straightforward to produce at least one piece of evidence that something among these many claims occurred that would inject a shred of doubt into the suspicions. I would guess that Don realized that if not one such piece of evidence could be mustered after 48 hours it was very unlikely that anything satisfactory was going to ultimately materialize.