Estimates of the effects of door-to-door canvassing and phone call campaign experiments are compromised by low contact rates and treatment of control group subjects.

Green, Donald P. 2005. "New Voters Project Assessment." Unpublished Manuscript. Institution for Social and Policy Studies, Yale University.

Donald Green presents results from a phone and door-to-door canvassing campaign designed to increase turnout of youth voters in six selected states in the 2004 national election. Randomized field experiments were conducted in conjunction with the New Voters Project in order to determine whether, and to what extent, those individuals contacted by the New Voter Project during its get-out-the-vote campaign voted at higher rates than those in a control group who were not contacted. Low contact rates of the phone campaign led to high standard errors of the estimates prohibiting researchers from drawing causal inference with any degree of certainty. Failure to exclude the control group in the canvassing experiment meant that no experimental results are available.

Electoral Context: The experiment was carried out in the context of the November 2004 national election.

Subject Population: Because the New Voter Project was created to address the concern of the lack of political engagement among young people, the target population of this experiment was approximately 2 million people between the ages of 18 and 24 in Colorado, Iowa, Nevada, New Mexico, Oregon and Wisconsin.

Randomization Procedure: For the phone experiment, five percent of the names from target lists of individual registered voters were to be randomly assigned to a control group and the rest to a treatment group that would receive calls. For the canvassing experiment, the research plan was to confine the experimental portion of the campaign to high density areas in which ten percent of targeted voters would be randomly assigned to the control group and the rest to the treatment group. 90,000 voters were to be assigned to the treatment group and 10,000 to the control group.

Treatment: The New Voter Project used door-to-door canvassing methods and phone calls in its campaign. Phone calls were made using Public Interest Research Group phone banks . Calls were made between October 13 and November 1, and callbacks were made November 1 and 2 to people who, in prior conversations, and said they would or might vote. The project contacted 36,207 people in the callback phase of the campaign. Callers used scripts written by the campaign that varied according the voting rules of the state in which the subjects resided. The quality of the calls was monitored intermittently. Green and his research team found that the calls sounded like young volunteers, and while some stuck closely to the script, others assumed a more conversational tone. The calls used an auto-dialer which may have reduced contact rates due to the delay and recognizable click heard by call recipients prior to conversation with the caller. The contact rates for various phases of the phone campaign never exceeded nineteen percent.

The Door-to-Door canvassing campaign involved between 240 and 290 canvassers working out of 14 offices. Canvassers were able to reach 342,142 doors and had contact at 159,851 doors. The majority of canvassers were between 18 and 24 years old. The campaign targeted precincts with high densities of college-aged voters. Canvassing scripts were lengthy and contained both questions and encouragement.

Findings: The experiment faced complications in that lists of registered voters were sent to phone banks prior to random assignment to treatment and control. Within the "waves" of the campaign for which a treatment and a control group were randomly assigned as planned, the estimated treatment on treated effect is negative but not statistically significant. Due to a low contact rate, very large standard errors render confidence intervals large and make it impossible to ascertain the effects of the campaign on turnout with reasonable certainty. Similarly, in the canvassing experiment, the names of individuals assigned to the control group by the research team were not suppressed from walk lists, so there are no experimental results to report.