Reproducibility has come a long way in political science. Many major journals now require replication materials be made available either on their websites or some service such as the Dataverse Network . Most of the top journals in political science have formally committed to reproducible research best practices by signing up to the The (DA-RT) Data Access and Research Transparency Joint Statement . This is certainly progress. But what are political scientists actually supposed to do with this new information? Data and code availability does help avoid effort duplication--researchers don't need to gather data or program statistical procedures that have already been gathered or programmed. It promotes better research habits . It definitely provides '' procedural oversight ''. We would be highly suspect of results from authors that were unable or unwilling to produce their code/data. However, there are lots of problems that data/code availability requirements do no...
Posts
A lot of progress has been made on improving political scientists’ ability to access data ‘programmatically’, e.g. data can be downloaded with source code R. Packages such as WDI for World Bank Development Indicator and dvn for many data sets stored on the Dataverse Network make it much easier for political scientists to use this data as part of a highly integrated and reproducible workflow . There are nonetheless still many commonly used political science data sets that aren’t easily accessible to researchers. Recently, I’ve been using the Database of Political Institutions (DPI) , Polity IV democracy indicators, and Reinhart and Rogoff’s (2010) financial crisis occurrence data. All three of these data sets are freely available for download online. However, getting them, cleaning them up, and merging them together is kind of a pain. This is especially true for the Reinhart and Rogoff data, which is in 4 Excel files with over 70 individual shee...
This is an abridged excerpt from Chapter 2 of my new book Reproducible Research with R and RStudio . It's published by Chapman & Hall/CRC Press . You can purchase it on Amazon . "Search inside this book" includes a complete table of contents. Researchers often start thinking about making their work reproducible near the end of the research process when they write up their results or maybe even later when a journal requires their data and code be made available for publication. Or maybe even later when another researcher asks if they can use the data from a published article to reproduce the findings. By then there may be numerous versions of the data set and records of the analyses stored across multiple folders on the researcher’s computers. It can be difficult and time consuming to sift through these files to create an accurate account of how the results were reached. Waiting until near the end of the research process to start thinking about reproducibility ca...
Reinhart & Rogoff: Everyone makes coding mistakes, we need to make it easy to find them + Graphing uncertainty
You may have already seen a lot written on the replication of Reinhart & Rogoff’s (R & R) much cited 2010 paper done by Herndon, Ash, and Pollin . If you haven’t, here is a round up of some of some of what has been written: Konczal , Yglesias , Krugman , Cowen , Peng , FT Alphaville . This is an interesting issue for me because it involves three topics I really like: political economy, reproducibility, and communicating uncertainty. Others have already commented on these topics in detail. I just wanted to add to this discussion by (a) talking about how this event highlights a real need for researchers to use systems that make finding and correcting mistakes easy, (b) incentivising mistake finding/correction rather than penalising it, and (c) showing uncertainty . Systems for Finding and Correcting Mistakes One of the problems Herndon, Ash, and Pollin found in R&R’s analysis was and Excel coding error . I love to hate on Excel as much as the next R ...
A big problem in reproducible research is that software changes. The code you used to do a piece of research may depend on a specific version of software that has since been changed. This is an annoying problem in R because install.packages only installs the most recent version of a package. It can be tedious to collect the old versions. On Toby Dylan Hocking 's suggestion, I added tools to the repmis package so that you can install, load, and cite specific R package versions. It should work for any package version that is stored on the CRAN archive ( http://cran.r-project.org ). To only install old package versions use the new repmis command InstallOldPackages . For example: # Install old versions of the e1071 and gtools packages. # Create vectors of the package names and versions to install # Note the names and version numbers must be in the same order Names <- c("e1071", "gtools") Vers <- c("1.6", "2.6.1") # Install...
I've started to put together an R package called repmis . It has miscellaneous tools for reproducible research with R. The idea behind the package is to collate commands that simplify some of the common R code used within knitr -type reproducible research papers. It's still very much in the early stages of development and has two commands: LoadandCite : a command to load all of the R packages used in a paper and create a BibTeX file containing citation information for them. It can also install the packages if they are on CRAN . source_GitHubData : a command for downloading plain-text formatted data stored on GitHub or at any other secure (https) URL. I've written about why you might want to use source_GitHubData before (see here and here ). You can use LoadandCite in a code chunk near the beginning of a knitr reproducible research document to load all of the R packages you will use in the document and automatically generate a BibTeX file you can draw on to c...
In previous posts I described how to input data stored on GitHub directly into R . You can do the same thing with source code stored on GitHub . Hadley Wickham has actually made the whole process easier by combining the getURL , textConnection , and source commands into one function: source_url . This is in his devtools package. Imagine we have a .R source code file like this: # Make cars scatter plot library(ggplot2) Plot <- qplot(cars$dist, cars$speed) + theme_bw() print(Plot) It is hosted on GitHub with the URL: https://raw.github.com/christophergandrud/christophergandrud.github.com/master/SourceCode/CarsScatterExample.R So to run this source code directly in R all we need to type is: library(devtools) SourceURL <- "https://raw.github.com/christophergandrud/christophergandrud.github.com/master/SourceCode/CarsScatterExample.R" source_url(SourceURL) There you go. You can also directly source GitHub gists (which are nice for shari...
A reader of my most recent post tried the R code I had written to download the data set of electoral disproportionality from the GitHub repository. However, it didn’t work for them. After entering disproportionality.data <- getURL(url) they got the error message: Error in function (type, msg, asError = TRUE) : SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed The Solution The problem seems to be that they didn’t have a certificate from an appropriate signing agent (see the RCurl FAQ page near the bottom) for more information. If you are really interested in SSL verification this page from redhat is a place to look). The solution to this problem is pretty straightforward. As the RCurl FAQ page points out you can use the argument ssl.verifypeer = FALSE to skip certificate verification (effectively a man-in-the-middle attack). So, if you get the above error m...
Update (6 January 2012): See this post for information on the source_GitHubData function that makes downloading data from GitHub easier. Update (15 June 2012): See this post for instructions on how to download GitHub based data into R if you are getting the error about an SSL certificate problem . GitHub is designed for collaborating on coding projects. Nonetheless, it is also a potentially great resource for researchers to make their data publicly available. Specifically you can use it to: store data in the cloud for future use (for free), track changes , make data publicly available for replication, create a website to nicely present key information about the data, and uniquely: benefit from error checking by the research community. This is an example of a data set that I’ve put up on GitHub. How? Taking advantage of these things through GitHub is pretty easy. In this post I’m going to give a brief overview of how to set up a GitHub data repos...
There has been some interest in the recent release of RStudio 0.96 and especially the ability to use combine its knitr Markdown functionality with Pandoc to integrate R and a variety of different documents types. I just wanted to add two quick things (one mostly aspirational, the other useful) Aspirational: Markdown/Ruby/deck.js I am currently using this combination to put together a presentation based on a recent working paper . Maybe out of procrastination I decided to see if there was any way to use knitr /Markdown to write a deck.js presentation. I generally prefer deck.js to the three Pandoc HTML presentation types ( slidy , S5 , and dzslides ). Deck.js presentations are a pain to write, so it would be great if there was a program like Pandoc that could quickly convert a Markdown file into a deck.js presentation. I discovered that there kind of is. There is a ruby program called deck.rb . The Markdown syntax is really simple and would be familiar to Pandoc users (...