General Guidelines
- There are several deadlines: team formation; proposal; mid-term report; final representation; and final report. Please see course schedule for details.
- Your project team should ideally consist of three members. If you want a larger group, please talk to the instructors and obtain approval in advance.
- Your project should deal with data from the “real” world—your data may come from one or more public sources, or you may collect your own data (see below for possible datasets and ideas).
- We have discussed (or will be discussing) various aspects of working with data, including but not limited to:
- Data collection and wrangling: e.g., cleaning, record linkage
- Basic querying and data processing: e.g., using SQL or MapReduce
- Statistical analysis and machine learning: e.g., testing, clustering, prediction
- Visualizing results
Your project should have a little bit of most the components above, but can focus on one of these aspects. Also, while data collection can be a component of the project, it cannot be the only focus of the project.
Datasets and Ideas
- Data.gov (http://www.data.gov/) has a huge compilation of data sets produced by the US government.
- The Supreme Court Database (http://scdb.wustl.edu/data.php) tracks all cases decided by the US Supreme Court.
- US government spending data (http://usaspending.gov/data) has information about government contracts and awards.
- Federal Election Commission (http://www.fec.gov/disclosure.shtml) has campaign finance data to download; their “disclosure portal” (http://www.fec.gov/pindex.shtml) also provide nice interfaces for exploring the data.
- GovTrack.us (http://www.govtrack.us/developers) tracks all bills through the Congress and all votes casted by its members. We have worked a fair amount with this data in this class. The Washington Post has a nice website (http://projects.washingtonpost.com/congress/112/) for exploring this type of data (in predefined ways), but you can be creative with additional and/or more flexible exploration and analysis options.
- Recovery.gov (http://www.recovery.gov/FAQ/Developer/Pages/default.aspx) provides all data related to the Recovery Act of 2009.
- The Washington Post maintains a list of datasets (http://www.washingtonpost.com/wp-srv/metro/data/datapost.html) that have been used to generate investigative news pieces. Most of these datasets hide behind some interface and may need to be scraped. Use this list for examples of what datasets are “interesting” and how to present data to the public effectively.
- National Institute for Computer-Assisted Reporting maintains a list of datasets of public interest (http://www.ire.org/nicar/database-library/). Use this list for examples of what datasets are “interesting”—they are generally not available to the public, but there may be alternative ways to obtain them.
- Google Fusion Table (http://www.google.com/fusiontables/Home/) hosts quite a number of datasets of public interest. It is a good place to find datasets or data sources to work on, and you can consider using it as a method of hosting your data for public access.
