This tutorial is in a very rough draft form, but I will post it here when it is finalized. (An earlier draft had some errors, and so I have taken it down.)

Isarithmic map of party identification from the 2008 CCES. Click to enlarge.

This map is produced from over 30,000 individual responses to the standard 7-point party identification question. I generate dense grid of points across the map, and calculate a distance-weighted mean value for each point, as well as a distance-weighted response density for each point. This grid is then smoothed through interpolation via the akima package and plotted with Hadley Wickham’s excellent ggplot2. I use a blue-red diverging color scale to encode mean values, and a transparency/alpha parameter to encode local density (similar to this approach for choropleth maps). Thus, in the northwestern U.S., from which relatively few responses were collected, the colors can be seen to “fade to black.” The same process, with a white background, is illustrated below.

Same approach as above, but with a white background. Click to enlarge.

The feedback I have gotten thus far suggests that the black background has better  æsthetics, but the white background is more clearly interpreted. I would be very interested in hearing your impressions of the relative merits of both.

The party identification maps above show the benefit of this ZIP code-based approach. First, unlike the choropleth approach, local political-geographic features are preserved without being obscured by the shape of boundary lines, such as county borders. For example, there is a clear difference between Atlanta and surrounding, more suburban areas, as well as within Miami, where the southern end of the city stands out as more Republican-leaning than the rest. Additionally, unlike choropleth maps in which variable/color values can change abruptly at boundaries which may be unfamiliar to the viewer, the distance smoothing employed here makes color-encoded information very clear around easily-identifiable major communities.

The other major benefit of this approach is that it can take as input any public opinion data for which some rough information is known about respondent location. Here I use ZIP codes, which are sufficiently granular to offer a nuanced view, but not so specific as to identify respondents. While it unsurprisingly works best with a large number of respondents, as seen here, I have also used a sample of 1,000 respondents, with useful results.

Here is a map of a variable of much current debate — family income. High income levels are concentrated along the coasts and around major cities. Again, compare the white and black backgrounds.

Isarithmic map of the distribution of family income levels. Click to enlarge.

The distribution of family income levels, with a white background to indicate response density. Click to enlarge.

These maps of abortion attitudes, as is the case with many interesting variables, fairly closely mirror the map of party identification. If you have data that you would like to see mapped in this way, especially with variables that don’t correlate so well with party identification or ideology, please get in touch with me.

Map of abortion attitudes. I should emphasize here that the legend illustrates color values describing local means, and the legend text list possible survey responses. Click to enlarge.

Abortion attitudes, with white background. Click to enlarge.

On color

Identifying optimal color schemes has been a challenge with these maps. I generally prefer to use a diverging palette to maximize color variance and the ease with which value gradations can be discerned. However, the excellent ColorBrewer diverging palettes tend to pass through white, which becomes indistinguishable from areas with low data density. As such, I have used analogous color palettes which span about 150 degrees around the hue spectrum, and tend to work well with varying alpha values over both black and white. Here again, I would be interested in hearing ideas about optimal color schemes when transparency is used to encode an additional data dimension.

Edit: The code as it stands right now is embarrassingly ugly and convoluted. Once I am able to incorporate any feedback, and clean up the code, I will be happy to share.

At the upcoming meeting of the Midwest Political Science Association, Aaron King, Frank Orlando, and I will be presenting a paper that investigates the determinants of success in Senate primary elections. We are primarily interested in whether voters are best modeled as voting by ideological proximity, or whether primary electorates strategically select candidates who offer a better chance of victory in the general election. Essentially, we are trying to identify whether ideological extremity is an advantage or a hindrance to primary electoral success.

Click for PDF version of the network graph (may be slow to load)

Unfortunately, estimating the ideology of many of these candidates can be problematic, given that many, for example, have not cast a roll-call vote which could be used in a NOMINATE-like scaling. Absent a more explicitly political record, we turn to the social networking/microblogging site Twitter, and collect data on the connections between elected officials and the mass public of Twitter users.

We use a nonmetric multidimensional scaling algorithm to estimate a space which represents users’ Twitter behavior, and find that the second dimension of that space correlates very well with Poole and Rosenthal’s NOMINATE scores for Senators and Representatives. Our main results can be seen in the figure below, and the paper is now available for download here.

Click for PDF version of the estimate summary dotplot.

A .ZIP file containing all the files necessary to follow the tutorial is available at: http://goo.gl/UrvQo.

The script goes very briefly through the loading of shapefiles and presidential election returns, and ends with the production of the choropleth below.

Click to enlarge

I don’t get into further customization of the map, as there are other more authoritative and complete sources for that. Further, much more detailed instruction on reading shape files are available from CSISS and NCEAS.

Included at the very end of the script is a brief example of a regression coefficient plot, something like a ggplot2 version of the coefplot() in Andrew Gelman‘s arm package.

I decided to develop the example into a function that takes as input a list() of model objects, and returns a ggplot2 object, which can be further modified by the user if so desired.

A coefficient plot comparing three models.

The script for the above plot can be found here. I also wrote a function that eschews arbitrarily discrete confidence bounds, instead attempting to suggest a sense of our confidence in the estimate without choosing a specific interval, since the difference between significance and insignificance is not itself significant. Code for the function is available here, and an example can be seen below.

Smoothed standard error coefficient plots

Header image from BibliOdyssey.

In this reductive scheme, then, “data displays” present variables in raw form — for use in exploratory data analysis, or perhaps just to offer the viewer access to all of the data. Where “statistical graphics” might be best served by simplicity and minimalism in design, such that a single idea might be conveyed clearly, “data displays” will tend to be inherently complex, and require effort from both the creator and viewer to parse meaning from the available information.

Where statistical graphics are ideal for presenting conclusions, data displays are useful for generating ideas, and optimally, permitting the relatively rapid identification of relationships between multiple variables. On top of this, I might add that many of the more well-regarded data displays of recent note offer macro-level insight as well as the opportunity to ascertain specific details (for this, interactivity is often valuable, as in the internet-classic New York Times box office visualization).

As several recent posts suggest, I am interested in finding ways to successfully and clearly convey multidimensional data, and have been focusing on political data as it varies across geopolitical units and time. Here I offer an approach which departs from the spatial basis of other recent efforts in favor of allowing the position of graphical objects to convey other variables.

County Vote Marimekko Plot, 1992, sorted by votes cast. Click for slideshow.

This type of plot is called, variously, a spinogram, a mosaic plot, or a marimekko — and is not dissimilar from a treemap with a different organizational structure (other examples). The utility of this plot type is that it can spatially convey four numeric variables (x position, y position, height, width), and color can be added to incorporate up to three additional variables (R, G, B). Further, there is a straightforward geometric interpretation of each cell: the areas of each (in this case, width/state turnout ×height/county proportion of state turnout) are directly comparable.

Unlike a stacked bar plot, the width of each column conveys information, permitting height to convey proportion rather than count. Further, columns and cells within columns can be sorted to express the ordering of variables of interest. In some ways, these can be seen as extreme reinterpretations of (Dorling) cartograms, in which not only the size and shape of political boundaries, but also their position, are distorted by other variables.

County Vote Marimekko Plot, 1924, sorted by Democratic share of the two-party vote. Click for slideshow.

In the plots above, cells are colored according to the strength of Democratic (blue), Republican (red), and other party (green) support, and counties whose turnout represents greater than 1% of the total turnout in an election are labeled.

I present two different layouts for the cells in each plot. The first arrays states left-to-right in order of the number of votes cast in an election, and sorts counties bottom-to-top in the same order. Thus, more populous states are on the right, and more populous counties are at the top of the plot. This arrangement allows the viewer to observe the effects of population density both within and across states, and may better facilitate tracking changes in county or state politics over time.

The second layout sorts states left-to-right, and counties bottom-to-top in order of the Democratic share of the two party vote (Dem Votes / (Dem Votes + Rep Votes)). Thus, more Democratic-leaning (relative to Republican) states are on the right, and counties that were more supportive of Democratic candidates are at the top. I believe that this arrangement makes it easier to discern overall trends in partisanship across time, as the total “sum” of red within a diagram is relatively easy to compare to the total “sum” of blue (and green).

I have attempted to make my R code fairly general, and it is available for download here, although it will obviously require some modifications for other applications. Our approaches differ, but another instructive example can be found at Learning R.

Isarithmic maps are essentially topographic or contour maps, wherein a third variable is represented in two dimensions by color, or by contour lines, indicating gradations. I had never seen such a map depicting political data — certainly not election returns, and thus sought to create them.

There is a trade-off between an isarithmic depiction versus a choroplethic depiction, in which a third variable is shown within discrete political boundaries. Namely, that though a politically-delineated presentation better facilitates the connection of the variable of interest to the level at which it was measured, the superimposition of geographically arbitrary political boundaries may cloud the existence of more general regional patterns.

Election-year maps can be seen in a slideshow here (and compared to the three-color choropleth maps here). The isarithmic depiction does an excellent job of highlighting several broad patterns in modern U.S. political history.

First, it does a good job of depicting local “peaks” and “valleys” of partisan support clustered around urban areas. In the 2008 map, for example, Salt Lake City, Denver, Chicago, Miami, Memphis, and many other cities stand apart from their surrounding environs, highlighted by a relatively intense concentration of voters with distinct partisan leanings. In 1980, this method shows that though Reagan enjoyed broad support in California, the revolution was not felt in the Bay Area.

Comparison of these maps across time also underscores well-known political trends, but offers more resolution than state-level choropleths and greater clarity than county-level choropleths. Note the nearly inverted maps for 1924 and 2004, between which elections the Solid South went from solidly Democratic to solidly Republican. Interestingly, though that particular regional pattern has been remarkably consistent since 1984, the South favored a Democratic candidate as recently as 1980.

These patterns over time are even better observed in motion. Interpolating support between elections, I have generated a video in which these maps shift smoothly from one election year to the next. The result is the story of 20th century presidential politics on a grand scale, condensed into a little 0ver a minute of data visualization.

The video can also be seen at YouTube (I recommend the “expanded” or “full screen” view), or at Vimeo. The images were rendered at 1280 x 720 pixels, to allow the video to be seen in HD.

This animated interpretation accentuates certain phenomena: the breadth and duration of support for Roosevelt, the shift from a Democratic to a Republican South, the move from an ostensibly east-west division to the contemporary coasts-versus-heartland division, and the stability of the latter.

More broadly, this video is a reminder that what constitutes “politics as usual” is always in flux, shifting sometimes abruptly. The landscape of American politics is constantly evolving,  as members of the two great parties battle for electoral supremacy.

Appendix on creating the visualization

Using county-level presidential returns from the CQ Press Voting and Elections Collection, I associated each county’s support in a given election year for the Democratic and Republican candidates with an approximation of that county’s centroid in degrees latitude and longitude, using the shapefiles loaded with the package mapdata.

I then used simple linear interpolation to create a smoothed transition from election-to-election, creating 99-interelectoral estimates of partisanship for each county. Using a custom function and the interp function from akima, I created a spatially smoothed image of interpolated partisanship at points other than the county centroids.

This resulted in inferred votes over the Gulf of Mexico, the Atlantic and Pacific Oceans, the Great Lakes, Canada and Mexico — so I had to clip any interpolated points outside of the U.S. border using the very handy pinpoly function from the spatialkernel package.

Finally, I created a custom color palette, a modification of the RdBu scheme from Colorbrewer, using colorRampPalette(), and plotted the interpolated data along with state borders using the excellent ggplot2.

I would like to note that I would have preferred using the Albers Equal Area Conic projection, but settled on the default Mercator projection, as drawing the Albers map with ggplot2 was prohibitively time-consuming, given that I was generating 2,201 individual frames.

I was able to find county-level presidential election returns going as far back as 1920, thanks to the CQ Press Voting and Elections Collection (gated). I converted the proportion of the vote garnered by Democratic, Republican, and “Other” parties’ candidates to coordinates in three-dimensional RGB color space, and used shapefiles from the mapdata package to plot these results as choropleth maps with ggplot.

Click to view slideshow.

It is interesting to observe these maps in a series, which gives historical context to the Red State/Blue State narrative. Most obviously, there is a significant shift in the geographic center of Democratic support, from a concentration in the southeast to the present equilibrium, localized on each coast and near the Great Lakes.

Among these 23 elections, landslide victories, such as Roosevelt over Landon in 1936, Johnson over Goldwater in 1964, Nixon over McGovern in 1972, and Reagan over Mondale in 1984, tend to stand out for their monochromaticity.

Also intriguing are the elections featuring substantial support for third-party candidates. Most of these are individuals who were had a strong support base in a specific region of the country, such as La Follette in the northwest, and Thurmond and Wallace in the deep south. Ross Perot’s run in 1992 is unique here, as his relatively broad geographic base of support results in a map that runs the gamut to a greater degree than any others.

Click on the image below to see a full screen version of the slideshow above, or to download any of the individual maps as PNGs.

Click for slideshow/download

To do so would be to employ something very much like k-means clustering, and produces some interesting results.

Using the population and latitude and longitude coordinates of the centroid of each (2000) census tract (a block-level reproduction was deemed too computationally intensive for the present purposes), I produced a geospatial k-means clustering for several states. Each tract was represented by its centroid as a point, weighted by population (which required a custom function, as the default kmeans() function in R does not appear to permit weighted points.

Since each run of the k-means algorithm begins with a random set of points, I replicated the function several thousand times, attempting to find a maximum inverse Herfindahl-Hirschman index of district population — the “effective number of districts,” as it were. For North Carolina, as shown below, I was able to find a maximum END of 12.17 for thirteen districts, which is a fairly even distribution of population.

Click to enlarge

Interestingly, there is still substantially wider variation in population than would be permitted under the current system. The least populous district houses fewer than 400,000 individuals, and the most populous, nearly a million. These figures are much more extreme than the extant least- (Wyoming) and most- (Montana) populous districts.

Population by district:

#  Population
1  398492
4  398896
8  423710
10 525860
2  533812
13 537417
3  618040
6  662092
11 676221
12 767249
7  785668
5  786448
9  935408

However, the district boundaries (here hastily drawn by use of chull()) are not characterized by the ragged edges and elongated shapes often seen in the existing plans.

I was interested in what the k-means-based plan would do to district partisanship, and decided to use population density as a rough proxy for local party affiliation. The distribution of population per square mile for each North Carolina census tract is shown below, with a vertical line indicating the median.

I decided to characterize any tract with greater-than-median population density as Democratic, and less-dense tracts as Republican. This resulted in the following proportion of Democrats residing in each district as plotted above:

#  % Dem.
1  0.253
4  0.265
10 0.336
8  0.350
6  0.383
7  0.474
13 0.510
3  0.589
11 0.615
9  0.628
12 0.671
2  0.673
5  0.837

As the table indicates, full turnout under such a plan would result in the election of 6 Republicans and 7 Democrats. Below, I plot “Democratic” tracts in blue and “Republican” tracts in red, scaled according to their population. Urban centers are easily identifiable. Note the difference between this plan and the current actual plan, which draws a single elongated district (the twelfth) parallel to Interstate 85.

Click to enlarge

Below, I replicate the same process for the state of Texas, generating 32 districts. One problem with the k-means algorithm is that larger states, or those with greater variance in population density, tend to generate districts with wide variations in population and inequalities of representation. The Texas plan below depicts a district with fewer than 200,000 residents and one with over 2 million. The Effective Number of Districts (maximum after 100 attempts) is a mere 21.58. Interestingly, the the district “partisanship” split is 22/10 majority Republican/Democrat — not far from the current 20/12 split. In this simulated redistricting, there are 10 districts in which the majority of residents live in higher-than-the-state-median density areas: four each in Houston and Dallas-Fort Worth, one each around San Antonio and Austin.

Click to enlarge

The slideshow below depicts the incremental steps of the weighted k-means algorithm toward convergence around alternate districts for Ohio, beginning with set of random centers, and eventually minimizing collective distances from local centroids.

Finally, I used the same algorithm to investigate what a the continental United States would look like if states were partitioned according to the k-means rule. Clicking on the image below will bring you to an interactive, scalable map of the U.S. with 48 alternate states and inferred partisanship. Instead of initializing with random centers, I started the k-means algorithm with the population centroids of the actual states, and allowed the algorithm to converge to a minimizing partition. Many of these alternative states are more compact but familiar versions of the originals, although this new plan does realize Plunkitt’s Fondest Dream.

Click to enlarge

In collaboration with Jacob Montgomery and John Aldrich, I am interested in understanding the relationship between observed (measured) and unobserved (true) dimensionality in Congress. In an ongoing project, we employ Monte Carlo simulations of legislative voting behavior, followed by dimensionality-reducing scaling techniques, to identify the parameters under which we might observe roll-call scalings similar to those we find in empirical data. Our findings suggest that the typical account of the dimensionality of ideology in Congress, “one-and-a-half dimensions,” may arise under a large variety of “true” dimensionality settings.

One of our papers, given at the 2010 APSA, is available for viewing or download here [PDF].

The slides from our paper presentation may be seen below.