Results from four separate experiments in which varying degrees of partisan or non-partisan messages are presented. Only results from the nonpartisan phone campaign experiment yielded substantive and statistically significant estimated mobilization effects.

McNulty, John E. 2005. "Phone-Based GOTV - What's on the Line? Field Experiments with Varied Partisan Components, 2002-2003." The Science of Voter Mobilization. Special Editors Donald P. Green and Alan S. Gerber. The Annals of the American Academy of Political and Social Science. vol. 601: 41-65.

Partisan versus Nonpartisan GOTV Phone Drives

John McNulty presents the results of four field experiments embedded in GOTV efforts conducted in the San Francisco Bay Area in 2002 and 2003. The experiments were conducted within the context of four different GOTV phone drives including a quasi-partisan effort on a 2002 municipal ballot measure in San Francisco; a strictly non-partisan effort in Berkeley for the 2002 general election; a Democratic (partisan) effort in the East Bay also for the 2002 general election; and an East Bay Democratic (partisan) effort for the 2003 California gubernatorial recall. Only the 2002 non-partisan effort brought about substantively and statistically significant and positive effects on voter turnout. While none of the experiments directly compare partisan and nonpartisan messages in parallel treatment groups within the same design, McNulty argues that the results do not bode well for partisan phone drives aiming to turn out voters.

Experiment 1 (Quasi-Partisan):

San Francisco 2002 Municipal Measure, "No on D" Phone Drive

Electoral Context: Included on the San Francisco ballot in the 2002 general election, Proposition D was a municipal measure that would transfer control of power utilities from PG&E to the SF municipal government. Similar measures initiated by the activist community in San Francisco had failed in past elections.

Subject population: The "No on D" campaign targeted registrants it concluded from survey analysis to be likely supporters of the proposition - relatively conservative registrants with moderate attitudes about city issues that were skeptical about progressive reforms and less hostile toward PG&E than other groups. "No on D" targeted homeowners in particular because they tend to be more conservative on economic issues and home ownership serves as a proxy for other demographic variables including age, socioeconomic, and residential stability. The campaign focused its effort on Republicans with uncertain turnout prospects and other registrants (Democrats prioritized over Decline-to-state registrants) that were considered to be likely to oppose the measure. The subject population - the randomized portion of the campaign's list - was comprised almost entirely of Democratic and Decline-to-State registrants.

Randomization Procedure: For the experimental portion of the campaign (29,964 registrants out of a total list of 44,796 registrants), McNulty randomly assigned about 5% of the partial list to a control group (1,485 registrants) whose names were removed from the lists forwarded to a call center in the Midwest. The remaining 28,479 targeted registrants were assigned to the treatment group.

Treatment: Targeted registrants assigned to the treatment group received a call from a commercial phone bank on the Thursday or Friday preceding the November 5 election. Callers did not leave messages on answering machines. Data were collected on call completion; time and date of the call; and the registrant's response to a question about his/her voting intentions. The campaign carried out various extra-experimental activities such as door hangers, media activity and a mass mailing that was distributed to nearly everyone in both the treatment and control groups which McNulty argues may have weakened the effect of the treatment. The contact rate was 50.2 percent.

Findings: McNulty finds no statistical or substantive differences in turnout between the treatment and control groups. The difference of .23 percentage points with a standard error of 1.3 percentage points implies that the phone effort succeeded in turning out only 100 voters. The charge per call was 58.8¢ (total $22,000), so the estimated cost per vote was approximately $200. The contact rate was 50.2%, therefore the treatment effect on the treated was approximately .5% with a standard error of 2.6%.

Experiment 2 (Strictly Nonpartisan):

East Bay 2002 General Election Nonpartisan [National, Berkeley] Youth Vote Coalition Phone Drive

Electoral Context: The voter registration and voter mobilization drives were carried out prior to the November 5, 2002 general election in California.

Subject population: The subjects of this phone-based voter mobilization drive were newly registered voters gathered throughout a registration drive carried out by the UC Berkeley Youth Vote chapter (a coalition of nonpartisan student groups with the exception of the Cal Dems). The registrants' names were gathered around the UC Berkeley campus, the main objective of the drive being to register young voters. The resulting voter mobilization target list excluded registrants enlisted by members of the Cal Dems group which became subjects of Experiment 3 which is detailed below.

Randomization Procedure: Individual registrants were randomly assigned to one of two treatment groups or the control group.

Treatment: Out of a total of 2,089 registrants, 570 names were randomly assigned to the control group and received no phone calls. The remaining names were randomly assigned to receive one of two treatments. The first treatment group of 750 registrants received phone call (or voicemail message) from the National Youth Vote Coalition reminding them to vote. Messages appealed to the targeted voters' age status as young voters and implored them to encourage friends to vote as well. The second treatment group of 769 registrants received phone calls that were identical with the exception that the callers were speaking on behalf of the "local" Berkeley Youth Vote Coalition (as opposed to "national"). The randomized duality of the treatment group was an effort to explore the impact of branding the Coalition as a local or national group.

Subjects assigned to the "national" treatment group were contacted directly at a rate of 31.9% excluding messages (81.7% including messages left on voice mail.) The corresponding contact rates for the "local" treatment group were 37.3% directly contacted and 82.1% contacted when voice mail messages are counted.

Findings: The nonpartisan GOTV phone drive effort was successful. The treatment overall treatment coefficients were 3.6 percentage points with a standard error of 2.7 percentage points for the "national" group and 5.2 percentage points with a standard error of 2.7 percentage points for the "local" group. The strongest effect was on Democrats who were contacted by callers that identified themselves as members of the local Berkeley Youth Vote Coalition for which the intent to treat effect was 9.4 percentage points overall (with a standard error of 3.9 percentage points) and the treatment on the treated effect was 11-26 percentage points depending on the definition of the contact rate. For Republicans contacted by "local" Youth Vote representatives, the effect was slightly negative but not statistically significant.

Experiment 3 (Partisan):

East Bay 2002 General Election Cal Dems Partisan Phone Drive

Electoral Context: The voter registration and voter mobilization drives were carried out prior to the November 5, 2002 general election.

Subject population: The subjects of this voter mobilization phone drive were taken from the list of new voter registrants that the Cal Dems obtained through the UC Berkeley Youth Vote Coalition registration drive described above.

Randomization Procedure: Individual registrants were randomly assigned to the treatment or the control group.

Treatment: The Cal Dems captured nearly 3,000 names of new registrants in the Youth Vote Coalition drive. Due to resource constraints, however, the Cal Dems narrowed their target list to 1,917 new registrants that lived within the Berkeley area. The treatment group consisted of 542 names (28 percent of the list). Undergraduate volunteers placed calls to treatment group registrants that appealed to partisan preferences reminding registrants to vote and urging them to vote for Democratic candidates. The call also indicated the Democratic Party's position on two ballot measures

The contact rates for both treatment and control groups were approximately 80 percent including voice mail messages and approximately 40 percent counting only direct conversations with voters.

Findings: The results indicate that in 2002 there were some differences between the treatment and control groups in overall turnout and among Democrats in particular. However, the effect is in the wrong direction. For all voters, the estimates of the effects imply that individuals assigned to the treatment group were less likely to vote relative to individuals assigned to the control group. The estimated effect for all voters assigned to the treatment group was -3.7 percentage points (standard error: 2.5 percentage points); for the major partisan affiliations, it was -6.1 percentage points (standard error: 3.5 percentage points), -3.7 percentage points (standard error: 4.2 percentage points), and -11.8 percentage points (standard error: 12.1 percentage points), for Democrats, "Decline to State" voters, and Republicans, respectively. Those in the treatment group seem less likely to vote than those not in the treatment group, although, given the large standard errors, this result is likely to be due to chance.

Experiment 4 (Partisan):

East Bay 2003 Gubernatorial Recall Election Cal Dems Partisan Phone Drive

Electoral Context: This registration and mobilization drive was carried out within the unique context of the California gubernatorial recall election in 2003. This election, McNulty notes, possibly received more media attention than any other non-presidential election. The recall ballot consisted of two parts including a vote for or against the recall of then-incumbent Governor Gray Davis and a subsequent vote for a candidate to assume the governor's office should he be recalled. In addition, one of the two ballot measures was controversial and therefore salient in the election. Because the prerequisites for becoming a candidate were not difficult to meet, 135 candidates were on the ballot of which Arnold Schwarzenegger and Cruz Bustamante were the frontrunners. While Prop. 53 was a non-descript transportation measure, Prop. 54 was a measure to prohibit the state from collecting racial classification data.

Subject population: The subjects of this drive were culled from a list of new registrants gathered by Cal Dems in a registration drive independent of the Youth Vote Coalition during the weeks leading up to the California gubernatorial recall election. Only the Democratic registrants that the group enlisted were included in a database for the subsequent voter mobilization drive - about 900 people.

Randomization Procedure: Individual registrants were randomly assigned to the treatment or the control group.

Treatment: The Cal Dems improved upon their methodology from 2002. They kept better records on party affiliation and phone numbers, and used a more conversational script than in 2002. With the objective of increasing contact rates, they targeted a smaller treatment group and called each registrant who was not initially reached at least twice. The Cal Dems attempted to call nearly two-thirds of the 900 Democratic registrants on the Wednesday, Thursday and Sunday before the election. The calls and messages asked voters to vote "No" on the recall and to vote for then Lieutenant Governor Cruz Bustamante for governor should Gray Davis be recalled. They did not declare a position on Prop 53, but they strongly urged a "No" vote on Prop. 54, the measure to prohibit the collection of racial data. The contact rate including voice mail messages was 81 percent. Counting only direct contact with the subject, the contact rate was 43 percent.

Findings: Again in this partisan phone mobilization drive, as in the 2002 Cal Dems drive, the treatment effects all turned out to be negative, none of which were statistically significant.