tag:bennstancil.com,2014:/feedBenn Stancil2014-08-06T11:09:04-07:00Benn Stancilhttp://bennstancil.combenn@modeanalytics.comSvbtle.comtag:bennstancil.com,2014:Post/40000-home-runs-vs-tonights-home-run-derby2014-08-06T11:09:04-07:002014-08-06T11:09:04-07:00Are Home Run Derby Hitters Different?<p>Tonight’s <a href="http://mlb.mlb.com/mlb/events/all_star/y2014/index.jsp?content=asg_hrderby" target="_blank">Home Run Derby</a> features ten of the world’s best home run hitters. But the Derby isn’t about just hitting home runs—it’s about seeing how far they fly.</p>
<p>The graphic below explores how this year’s participants’ homers compare to the 40,000 home runs that have been hit over the last seven-and-a-half MLB seasons. Do they hit the ball harder, higher, and farther than average?</p>
<iframe src="https://modeanalytics.com/benn/reports/eb6bbba52fa3/embed" width="100%" height="800"></iframe>
<p>Data for the graphic was provided by <a target="_blank" href="http://www.hittrackeronline.com/index.php">ESPN</a>. The full dataset for all regular season home runs can be found and analyzed on <a target="_blank" href="https://modeanalytics.com/benn/tables/home_run_distance">Mode</a>.</p>
tag:bennstancil.com,2014:Post/5-interesting-datasets2014-07-23T00:40:15-07:002014-07-23T00:40:15-07:00Five Public Datasets, and Lots of Ideas for Exploring Them<p><img src="http://blog.modeanalytics.com/images/ufo.jpeg" alt="ufo"></p>
<p>The world is full of <a target="_blank" href="http://rs.io/2014/05/29/list-of-data-sets.html">interesting datasets</a>. But even though data is increasingly accessible, it’s sometimes hard think up an interesting problem to analyze. Maybe there are just too many possible questions, maybe it’s a pain to set up analytical tools, or maybe it’s just too easy to get <a target="_blank" href="http://www.theatlantic.com/technology/archive/2013/09/did-the-us-government-make-the-greatest-gif-of-all-time/280053/">distracted by animal GIFs</a>.</p>
<p>Whatever the case, we want to make it easier to start working on interesting problems right away. Here are five datasets, already loaded into Mode’s public database, that you can query, analyze, and visualize <em>right now</em>. </p>
<p>For each dataset, I’ve provided a link to the table in Mode’s public data warehouse. If you’re feeling lazy and only want to work with a tiny amount of data (as in, one row), I found the best single row of data from each dataset. And if you’re feeling ambitious—and want to get <a target="_blank" href="http://www.reddit.com/r/MapPorn/">popular on the internet</a> or <a target="_blank" href="http://www.washingtonpost.com/blogs/worldviews/wp/2013/08/12/40-maps-that-explain-the-world/">explain</a> <a target="_blank" href="http://www.businessinsider.com/the-global-economy-explained-in-17-maps-2013-9?op=1">some</a> <a target="_blank" href="http://www.zerohedge.com/news/2014-02-20/ukraine-situation-explained-one-map">things</a>—I added some ideas for turning these datasets into maps.</p>
<h3>FEC Campaign Finance Data</h3>
<p>The Federal Election Commission requires candidates to make their campaign expenditures public. This dataset includes over 200,000 campaign expenditures from the 2012 U.S. presidential campaign, and is full of fascinating discoveries. Like <a target="_blank" href="https://modeanalytics.com/benn/reports/3f8dbe7645a5">Herman Cain’s $150,000 expense on Herman Cain</a>. And <a target="_blank" href="https://modeanalytics.com/benn/reports/de9ebaedd193">the $5,000–the most of any candidate by far–Mitt Romney spent at liquor stores</a>. And Ron Paul’s and Romney’s <a target="_blank" href="https://modeanalytics.com/benn/reports/c8549d69b56e/runs/901d91c50326">addiction</a> <a target="_blank" href="https://modeanalytics.com/benn/reports/c8549d69b56e/runs/3cfb682552ba">to</a> <a target="_blank" href="https://modeanalytics.com/benn/reports/c8549d69b56e/runs/6117b404a3ee">fast</a> <a target="_blank" href="https://modeanalytics.com/benn/reports/c8549d69b56e/runs/b3d71d451038">food</a> (and Obama’s clear preference for Subway).</p>
<p><em>Herman Cain be like:</em><br>
<img src="http://blog.modeanalytics.com/images/treat-yo-self.gif" alt="treat yo self"></p>
<ul>
<li>
<strong>What kinds of questions can I ask?</strong> What do candidates spend the most money on? Do some candidates get a spending lead early, while others save for the end? How do spending patterns differ for the incumbent compared to challengers?</li>
<li>
<strong>What’s the table called?</strong><a target="_blank" href="https://staging.modeanalytics.com/cooldata/tables/fec_2012_presidential_campaign_expenditures">cooldata.fec_2012_presidential_campaign_expenditures</a>
</li>
<li>
<strong>What’s the best row?</strong> <a target="_blank" href="https://modeanalytics.com/benn/reports/3f39a90bddd0">wut.</a>
</li>
<li>
<strong>Can I use this data to make a map?</strong> Absolutely. You could see how money was spent in each state. Or you could see if different candidates focus on different regions. Or, if you were feeling particularly ambitious, you could map candidate expenses by day to sketch out how they traveled across the country during their campaigns.</li>
</ul>
<h3>Crunchbase</h3>
<p>Crunchbase is quickly becoming the dataset of record for the startup and venture capital communities. It can provide information on anything from what industries are hot (<a target="_blank" href="https://modeanalytics.com/benn/reports/5dc33294f9c2">biotech</a>) to the potential effects of founder <a target="_blank" href="http://blog.modeanalytics.com/are-experienced-founders-better/">experience</a> or <a target="_blank" href="https://modeanalytics.com/benn/reports/211f54949c00">age</a>. The dataset includes funding, investment, and acquisition data on over 40,000 companies.</p>
<ul>
<li>
<strong>What kinds of questions can I ask?</strong> Are there characteristics of a company—industry, location, etc.—that differ by VC? Do some VCs typically invest together, while others rarely do so? Are companies raising more money earlier? ARE WE IN A BUBBLE??</li>
<li>
<strong>What are the tables called?</strong> <a target="_blank" href="https://modeanalytics.com/crunchbase/tables/acquisitions">crunchbase.acquisitions</a>; <a target="_blank" href="https://modeanalytics.com/crunchbase/tables/companies">crunchbase.companies</a>; <a target="_blank" href="https://modeanalytics.com/crunchbase/tables/investments">crunchbase.investments</a>; <a target="_blank" href="https://modeanalytics.com/crunchbase/tables/rounds">crunchbase.rounds</a>.</li>
<li>
<strong>What’s the best row?</strong> <a target="_blank" href="https://modeanalytics.com/benn/reports/44a344e677ac">This one</a>, which is approaching the theoretical limit of how good a row of data can be. </li>
<li>
<strong>Can I use this data to make a map?</strong> Yes! Like this rather uninformative one, showing the number of startups by the county where they’re headquartered. </li>
</ul>
<iframe src="https://modeanalytics.com/benn/reports/86a2a6dbbf66/embed" width="100%" height="530"></iframe>
<h3>UFO Sightings</h3>
<p><a target="_blank" href="http://www.quandl.com/">Quandl</a>, which provides millions of free datasets on vast range of subjects, added data on <a target="_blank" href="https://modeanalytics.com/thomas/tables/ufo_sightings">UFO sightings</a> to Mode. The data includes the number of reported sightings by month. Quandl gets the data from the National UFO Reporting Center (and in case you need to report a sighting, they have a <a target="_blank" href="http://www.nuforc.org/Report.html">hotline</a>).</p>
<ul>
<li>
<strong>What kinds of questions can I ask?</strong> Are some months more popular for sightings? What <a target="_blank" href="http://www.tylervigen.com/">correlates</a> with UFO sightings?</li>
<li>
<strong>What’s the table called?</strong> <a target="_blank" href="https://modeanalytics.com/thomas/tables/ufo_sightings">thomas.ufo_sightings</a>
</li>
<li>
<strong>What’s the best row?</strong> <a target="_blank" href="https://modeanalytics.com/benn/reports/8b3ff191fe53">The first one</a>, on a sighting from June 1400. The first sighting of the <a target="_blank" href="http://en.wikipedia.org/wiki/Black_Knight_satellite">Black Knight</a>?</li>
<li>
<strong>Can I use this data to make a map?</strong> No. But you can probably combine it with some <a target="_blank" href="http://giphy.com/search/independence-day">Independence Day GIFs</a> and make a killer listicle. </li>
</ul>
<h3>FiveThirtyEight</h3>
<p><a target="_blank" href="http://fivethirtyeight.com/">FiveThirtyEight</a>, Nate Silver’s data journalism site, produces a lot of great analysis. For some articles, they publish the underlying data on <a target="_blank" href="https://github.com/fivethirtyeight/data">GitHub</a>. If you want to explore their data or expand on their analyses, we’ve uploaded most of their <a target="_blank" href="https://modeanalytics.com/search?utf8=%E2%9C%93&q=cooldata+fivethirtyeight+type%3Atable">datasets</a>. A few topics include classic rock radio plays, the ages of Congressional representatives, World Cup predictions, and surveys about defining U.S. geographic regions and international cuisine preferences.</p>
<ul>
<li>
<strong>What kinds of questions can I ask?</strong> From the cuisine survey, do people from different areas of the country prefer different foods? Can we predict what food someone would like based on their other preferences? From the data on Congress age, it might be interesting to see if people from different states tend to elect representatives of different ages—and are those ages related to the age of the constituents? And from the classic rock data, which classic rock songs should we be most sick of by now? Which radio stations have the laziest DJs?</li>
<li>
<strong>What are the tables called?</strong> <a target="_blank" href="https://modeanalytics.com/cooldata/tables/fivethirtyeight_region_survey">cooldata.fivethirtyeight_region_survey</a>; <a target="_blank" href="https://modeanalytics.com/cooldata/tables/fivethirtyeight_congress_age">cooldata.fivethirtyeight_congress_age</a>; <a target="_blank" href="https://modeanalytics.com/cooldata/tables/fivethirtyeight_world_cup_predictions">cooldata.fivethirtyeight_world_cup_predictions</a>; <a target="_blank" href="https://modeanalytics.com/cooldata/tables/fivethirtyeight_classic_rock_plays">cooldata.fivethrityeight_classic_rock_plays</a>; <a target="_blank" href="https://modeanalytics.com/cooldata/tables/fivethirtyeight_classic_rock_songs">cooldata.fivethirtyeight_classic_rock_songs</a>; <a target="_blank" href="https://modeanalytics.com/cooldata/tables/fivethirtyeight_food_world_cup">fivethirtyeight_food_world_cup</a>.</li>
<li>
<strong>What’s the best row?</strong> <a target="_blank" href="https://modeanalytics.com/benn/reports/f8f1676aa4f4">Too soon?</a>
</li>
<li>
<strong>Can I use this data to make a map?</strong> Yes! I made the map below to explore how people from different states defined the South and Midwest. You could map food cuisine preferences by region or show how the age of states’ Congressional representatives have changed over time. </li>
</ul>
<iframe src="https://modeanalytics.com/benn/reports/c82a11aa3bda/runs/62d8e2839ddb/embed" width="100%" height="2500"></iframe>
<h3>Holidays all over the world</h3>
<p>This dataset includes a list of all the holidays in the world over the next year. While this data is useful for analysis, it could be even more valuable for figuring out which parts of the world—and which of your customers—are on vacation.</p>
<ul>
<li>
<strong>What kinds of questions can I ask?</strong> Which countries have the most holidays? Which months and days have the most holidays? Which countries share a lot of holidays, and which only share a few?</li>
<li>
<strong>What’s the table called?</strong> <a target="_blank" href="https://modeanalytics.com/reference_lookups/tables/holidays_by_country">reference_lookups.holidays_by_country</a>
</li>
<li>
<strong>What’s the best row?</strong> <a target="_blank" href="https://modeanalytics.com/benn/reports/251699b2cc68">The freedom row</a>.</li>
<li>
<strong>Can I use this data to make a map?</strong> Yes! You could show the average number of holidays during the year by country, or where there’s a holiday on any given day. </li>
</ul>
<h3>Ideas for More?</h3>
<p>Inspired to do something fun with one of these datasets? Send us a link to your project on <a target="_blank" href="http://twitter.com/modeanalytics">Twitter</a> or <a target="_blank" href="http://facebook.com/modeanalytics">Facebook</a>, and we’ll share some of the best work! And if you want to make a map, we’ll soon be publishing a quick tutorial for how make one, but feel free to <a target="_blank" href="mailto:info@modeanalytics.com">email us</a> if you have any questions now.</p>
tag:bennstancil.com,2014:Post/nyc-taxis-and-race2014-06-23T15:57:14-07:002014-06-23T15:57:14-07:00Are Taxi Drivers Racist?<p><img src="http://i.imgur.com/eTeQSWw.png" alt="nyc taxi"></p>
<p>Last week, <a href="http://chriswhong.com/open-data/foil_nyc_taxi/">Chris Whong</a> published a massive dataset of every taxi trip taken in New York in 2013. The data, provided through a Freedom of Information Law request, includes an incredible amount of detail on where trips started, where they ended, when they occurred, how much they cost, and how many passengers there were. </p>
<p>A number of people have already done incredible things with this data, including making a <a href="https://www.mapbox.com/blog/nyc-taxi/">remarkably detailed map</a> of where cabs typically pick up and drop off passengers. A dataset of this detail opens the door for countless questions and angles of exploration. </p>
<p>One such question surrounds <a href="http://abcnews.go.com/GMA/race-cab-hailing-ride-black-white/story?id=7223511">accusations</a> that New York City cabs discriminate against potential passengers. A number of <a href="http://www.timeout.com/newyork/things-to-do/why-cant-a-black-man-get-a-cab-in-this-town">anecdotes</a> claim that New York cabs are reluctant to stop for black passengers, especially after dark. Could this new dataset shed any light on this issue?</p>
<p>The dataset, which provides no personal details on passengers and drivers (<a href="https://medium.com/@vijayp/of-taxis-and-rainbows-f6bc289679a1">sort of</a>), can’t answer this question directly. However, by looking at where cabs pick up and drop off passengers—and by considering the racial makeup of those neighborhoods—we can start piecing together the evidence. It’s not conclusive, but it could be a start.</p>
<h1 id="where-cabs-gomdashand-where-they-don39t_1">Where Cabs Go—And Where They Don’t <a class="head_anchor" href="#where-cabs-gomdashand-where-they-don39t_1">#</a>
</h1>
<p>On the surface, taxis appear to avoid picking up passengers in neighborhoods more heavily populated by minorities. The chart below show the number of taxi pickups per 1,000 people in areas bucketed by the demographic makeup of the residential population. As the chart shows—and the <a href="https://modeanalytics.com/benn/reports/43e26748b755">map above</a> makes even more clear—as minorities make up more of the population, the fewer taxi trips originate from those areas.</p>
<iframe src="https://modeanalytics.com/benn/reports/7e1270069fd8/embed" width="100%" height="365"></iframe>
<p>A few details about the numbers above—and the numbers considered in the rest of this post—are important to note. First, the data above only includes trips from one week in June (chosen because it’s not affected by major holidays or severe weather). Even though this represents less than a 50th of the dataset, it <em>still</em> includes over 3 million trips. Second, because of the <a href="http://en.wikipedia.org/wiki/Boro_taxi">complex politics</a> behind the New York City taxi system and and its services outside of Manhattan, this analysis is limited to trips that originated in Manhattan (officially, New York County).</p>
<p>In order to determine the demographic makeup of pickup and dropoff locations, I rounded the latitude and longitude of each pickup and dropoff point to the nearest thousandth, which approximates the location within about <a href="http://gis.stackexchange.com/questions/8650/how-to-measure-the-accuracy-of-latitude-and-longitude">100 meters</a>. For every rounded location, I looked up the Census tract that it falls in via the <a href="http://www.fcc.gov/developers/census-block-conversions-api">FCC Census block API</a>. The Census provides demographic and economic data by <a href="http://factfinder2.census.gov/faces/tableservices/jsf/pages/productview.xhtml">Census tract</a>, allowing the mapping between GPS coordinates and neighborhood demographics. </p>
<p>Though the chart above raise suspicions about racial profiling among taxi drivers, the result is far from conclusive. After all, pickup rates are determined by passenger demand as well as the preferences of cab drivers. Neighborhoods with more minorities could simply have fewer prospective passengers. </p>
<p>Why this might be true is a question worth its own exploration, but for the sake of this analysis, I added a couple factors that could serve as proxies for taxi demand:</p>
<p><strong>1. Residents’ incomes</strong> - Wealthier people likely take more cabs than the less-well-off. If neighborhoods with mostly white residents tend to be more affluent (<a href="https://modeanalytics.com/benn/reports/830d5daaed9f">which they are</a>), the effect above could be caused by economics rather than discrimination.</p>
<p><strong>2. Location</strong> - Central Manhattan is <a href="http://projects.nytimes.com/census/2010/explorer">largely populated by whites</a>. Though the graph above controls for population size, it doesn’t account for commercial or tourist activity, which is likely concentrated in central Manhattan. The apparent high rate of cab activity in white tracts could actually be because whites live in central areas with more commercial activity, while minorities live on Manhattan’s edges.</p>
<p>To attempt to account for these two factors, I made a simple model that estimates how many taxi pickups are expected in a Census tract given its population, median income, distance from Times Square (roughly the center of Manhattan), and non-white population. </p>
<p>A <a href="https://github.com/mode/blog/blob/master/2014-06-26-taxis/regressions.r">basic model</a> with incomes, population, distance from Times Square, and the size of the white population suggests that these other factors—and primarily distance from Times Square—account for the apparent effect shown above. Nevertheless, even controlling for these variables, the correlation between the size of the white population and taxi pickups is strong enough to at least warrant further investigation.</p>
<pre><code class="prettyprint lang-R"># This model regresses the number of pickups by Census tract against the tract's income, population, white population ratio, and distance from Times Square. As expected, incomes are positively correlated with pickups, and distance from Times Square is negatively correlated. The size of white population is also positively correlated with pickups, though, as show by the p-value in the final column, the relationship isn't highly significant.
Regression output:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.204e+04 2.149e+03 5.600 5.69e-08 ***
median_income 6.346e-02 2.006e-02 3.163 0.00176 **
population -7.785e-02 1.728e-01 -0.450 0.65282
white_percent 4.748e+03 3.244e+03 1.464 0.14455
distance_in_miles -2.840e+03 3.126e+02 -9.084 < 2e-16 ***
</code></pre>
<p>One claim worth exploring is the specific case highlighted in the anecdotes above: Cab drivers are reluctant to pick up black people (not necessarily all minorities) at night. If we limit our model to trips taken between 9:00 PM and 6:00 AM, the size of the black population appears to have a significantly negative effect on cab pickups.</p>
<pre><code class="prettyprint lang-R"># This model examines the number of pickups between 9:00 PM and 6:00 AM by Census tract. This restriction weakens the correlation between pickups and income, but the relationship between pickups and the size of the black population becomes very strong.
Regression output:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 7.188e+03 8.790e+02 8.177 1.53e-14 ***
median_income 1.510e-03 6.055e-03 0.249 0.803301
population -9.184e-02 6.859e-02 -1.339 0.181815
black_percent -3.937e+03 1.142e+03 -3.447 0.000667 ***
distance_in_miles -9.869e+02 1.194e+02 -8.266 8.56e-15 ***
</code></pre>
<p>Before concluding that this is damning evidence against cabs, it’s important to note that accounting for Manhattan’s geography in this model is a trickier statistical problem than just adding it as a variable to a regression. Because many of the non-white areas are far from Times Square, there’s a <a href="https://modeanalytics.com/benn/reports/b375af325753">strong correlation</a> between distance from Times Square and the racial makeup of the Census tract. This creates <a href="http://en.wikipedia.org/wiki/Multicollinearity">collinearity problems</a> that undermine the results above. </p>
<p>Though more complex methods can correct this, I chose to make a simple approximation. There’s <a href="https://modeanalytics.com/benn/reports/b375af325753">little correlation</a> between race and distance between 3 and 4 miles from Times Square. Limiting on these set of tracts, we can apply the <a href="https://github.com/mode/blog/blob/master/2014-06-26-taxis/regression_in_distance_range.r">model</a> as before. In this band, only income, and not racial makeup, appear to matter.</p>
<pre><code class="prettyprint lang-R"># This model considers the number of pickups between 9:00 PM and 6:00 AM by Census tract, but only includes Census tracts between 3 and 4 miles from Times Square. This limited dataset shows no correlation between race and pickups. The model only finds a significant relationship between pickups and tract income.
Regression output:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 8.294e+02 1.821e+03 0.456 0.652
median_income 2.756e-02 3.675e-03 7.501 8.73e-09 ***
population 5.658e-02 5.119e-02 1.105 0.277
black_percent -3.785e+02 6.655e+02 -0.569 0.573
distance_in_miles -4.579e+02 5.124e+02 -0.894 0.378
</code></pre>
<h1 id="considering-dropoffs_1">Considering Dropoffs <a class="head_anchor" href="#considering-dropoffs_1">#</a>
</h1>
<p>Dropoff patterns and their relationships to pickup locations could provide an additional angle of exploration. Dropoffs, as the chart below shows, are also heavily biased towards white areas. </p>
<iframe src="https://modeanalytics.com/benn/reports/643c1c179ae2/embed" width="100%" height="365"></iframe>
<p>Though this supports the argument that minority neighborhoods are underserved, it says little about racial motivations. On the one hand, if cab drivers were discriminating against minorities, we’d expect taxis to be taking fewer passengers to minority neighborhoods and therefore have fewer dropoffs in those areas. On the other hand, the low dropoff rate could be caused by the same reasons as the low pickup rate: There’s little demand to travel to and from these neighborhoods. </p>
<p>Dropoff data <em>combined with subsequent pickup data</em> could show something far more interesting. Because passengers <a href="http://nymag.com/daily/intelligencer/2013/05/whats-going-on-with-new-yorks-taxi-wars.html">can’t pre-arrange</a> for a New York yellow cab to pick them up, most trips come from street hails. If cabs wanted to avoid minority fares, they would likely move away from areas where they would get hailed by minorities once they dropped off a passenger, especially if that fare was dropped off in a minority neighborhood. </p>
<p>Nevertheless, this approach suffers from the same flaw as the chart above—non-discriminatory cabs would exhibit this same behavior if demand were higher in white neighborhoods. </p>
<p>A different angle can provide some conclusions. Cabs do pick up passengers in minority neighborhoods. If these pickups were preceded exclusively by dropoffs in the same neighborhoods, that would suggest cabs only traveled to these areas when a passenger requested they do so (the reasons for their reluctance would still unclear). However, if pickups in minority neighborhhoods followed dropoffs in white neighborhoods—in other words, if drivers voluntarily traveled to non-white areas—that could provide a piece of evidence in support of non-discriminatory practices.</p>
<p>As the chart below shows, pickups from minority neighborhoods are typically preceded by dropoffs in <strong>white neighborhoods</strong>, suggesting pickups in minority neighborhoods aren’t dependent on dropoffs in minority neighborhoods. </p>
<iframe src="https://modeanalytics.com/benn/reports/1f78e1da3534/embed" width="100%" height="365"></iframe>
<h1 id="digging-deeper_1">Digging Deeper <a class="head_anchor" href="#digging-deeper_1">#</a>
</h1>
<p>This ultimately provides one strong conclusion and one mixed one. First, by nearly every measure, cab drivers have a strong preference for white neighborhoods and non-white neighborhoods are severely underserved. This is undeniable. </p>
<p>Drivers’ motivations, however, aren’t clear. There isn’t a lot of evidence for geographically opportunistic discrimination: Cabs in white neighborhoods aren’t much more likely to stay there than cabs elsewhere. Still, the degree to which minority neighborhoods are underserved and the models above, despite their flaws, do raise questions. But the evidence supports several plausible explanations for these disparities in service.</p>
<p>Regardless of the results of this analysis, racism exists, and the experiences described by those above—and many others—are real and can’t be discounted. No matter what this smoothed, stylized, and aggregated perspective describes, we also have to acknowledge the view from the street.</p>
<p>Furthermore, these arguments are only a very cursory first step into the data. There are numerous problems with these conclusions, and many questions that could be explored further. Just to name a few:</p>
<ol>
<li><p>Data quality is a concern. This is particularly true for trip sequencing because some <a href="https://modeanalytics.com/benn/reports/14544086e996">trips appear to begin before the previous trip ended</a>.</p></li>
<li><p>Because GPS coordinates were rounded to the nearest 100 meters, Census tract matching is imprecise. </p></li>
<li><p>Demographic figures for Census tracts represent residential populations. The demographics of commuters, tourists, and others traveling to and from different tracts is unidentified. </p></li>
<li><p>Income is an imperfect measure of taxi demand. After incomes reach a certain point, residents may begin to favor other forms of transportation, like their own cars or limo services. </p></li>
<li><p>Other unmeasured factors can affect demand as well. For example, older populations may be more likely to take cabs, while residents in tracts with a high concentration of restaurants and bars may prefer to stay in their neighborhoods and demand fewer cabs. Demand in these neighborhoods could represent that of outsiders as much as that of residents. Moreover, taxi demand could be endogenous to discrimination—minority populations may hail fewer cabs because they’re concerned about racism.</p></li>
<li><p>Census tracts are not homogenous, but by identifying them as largely white or non-white, we’re flattening each tract into a single category. It’s possible that many passengers from a non-white tract are white, or many passengers from a white district are non-white.</p></li>
<li><p>Similarly, non-white districts are homogenized into a single group. A more detailed inspection of demographics could yield different results. As noted above, there seems to be evidence that black passengers face worse treatment. This analysis could be greatly expanded, or applied to other groups.</p></li>
<li><p>The data only includes one form of travel. Different parts of the city may be better served by other methods of transportation (subways, buses, <a href="http://en.wikipedia.org/wiki/Boro_taxi">Green cabs</a>); travel choices may be more affected by these options rather than the biases of cab drivers.</p></li>
<li><p>It’s possible that these conclusions change during different times of the day. The type of passenger that rides a cab at 9:00 AM on a Monday morning is likely different than the type of passenger looking for a cab at 11:00 PM on Friday night. These potential differences are largely unexplored. </p></li>
</ol>
<p>All these questions and caveats could be investigated further. To that end, all of my analysis above can be easily accessed via in-text links or links below the graphs. Additional details ares provided below. The links to Mode provide access to both the raw data and the underlying analysis. For anyone interested in exploring these ideas further—or digging into an entirely new issue with the same data—I invite you to copy and extend my work however you see fit.</p>
<h1 id="data_1">Data <a class="head_anchor" href="#data_1">#</a>
</h1>
<p>Taxi data was provided by <a href="http://chriswhong.com/open-data/foil_nyc_taxi/">Chris Whong</a> and <a href="http://www.andresmh.com/nyctaxitrips/">Andrés Monroy</a>. June data is available as <a href="http://www.andresmh.com/nyctaxitrips/">trip_data_6</a>. I trimmed this dataset down to one week in June using <a href="https://github.com/mode/blog/blob/master/2014-06-26-taxis/get_week_data.py">Python</a>. In the process, I also added a by-driver trip counter in order to identify trip order. Data on the New York Census tracts was provided by the <a href="http://factfinder2.census.gov/faces/tableservices/jsf/pages/productview.xhtml">Census</a>. This data was matched to GPS coordinates using another <a href="https://github.com/mode/blog/blob/master/2014-06-26-taxis/lat_long_lookup.py">Python script</a>. Finally, simple regression models were constructed in R. The models presented above, as well as several intermediate steps, can be found in <a href="https://github.com/mode/blog/tree/master/2014-06-26-taxis">GitHub</a>.</p>
<p><em>Benn Stancil is the chief analyst of <a href="https://modeanalytics.com/">Mode</a>.</em></p>
tag:bennstancil.com,2014:Post/where-americans-think-they-live2014-05-05T10:50:47-07:002014-05-05T10:50:47-07:00Where Americans Think They Live<p><img src="http://www.apsu.edu/sites/apsu.edu/files/imagecache/simple_page_header_image/cosm/Geography_Teacher.JPG" alt="discovering alabama"></p>
<p>Last week, FiveThirtyEight’s Walt Hickey wrote a couple of interesting articles about which states are in the <a href="http://fivethirtyeight.com/datalab/what-states-are-in-the-midwest/">Midwest</a> and <a href="http://fivethirtyeight.com/datalab/which-states-are-in-the-south/">South</a>. Being from North Carolina—where people definitely consider themselves Southern—I was surprised to see that only two-thirds of all respondents to the FiveThirtyEight survey said North Carolina was in the South. </p>
<p>This made me wonder: How do people from different states define the South and Midwest? And specifically, how do the views of people from who live in a state differ from those of people who don’t live there?</p>
<p>In keeping with their commitment to release an article’s underlying data, FiveThirtyEight published the full survey results on <a href="https://github.com/fivethirtyeight/data/tree/master/region-survey">GitHub</a>. Opening this data enables further exploration, as in the interactive graphic below. I looked at how every state in the U.S. views the South and Midwest, and how local opinions compare to national views.</p>
<p>The data reveals many additional findings:</p>
<ul>
<li><p>People tend to think the South and Midwest are close to them. Western states think the regions extend further west than states on the East Coast.</p></li>
<li><p>It’s not clear if this is because people perceive the regions this way, or if it’s because they forget about states that are farther away (though this seems unlikely, as it appears respondents were provided a list of states to choose from).</p></li>
<li><p>I’m not sure where people from Wyoming think Alabama is, or where people from Delaware think most U.S. states are (maybe they, um, I believe, and such as, don’t have <a href="http://www.youtube.com/watch?v=lj3iNxZ8Dww">maps</a>.)</p></li>
<li><p>Michigan and Oklahoma are the most geographically contentious states.</p></li>
<li><p>My intuition about North Carolina was right—95% of North Carolinians see the state as being in the South. </p></li>
<li><p>Despite people across the country having a mixed view of Arkansas, Arkansans, very clearly want to be in the South.</p></li>
<li><p>By contrast, West Virginians do not agree with it’s <a href="http://en.wikipedia.org/wiki/Buckwild_(TV_series)">Southern reputation</a>.</p></li>
</ul>
<h1 id="explore-the-data_1">Explore the data <a class="head_anchor" href="#explore-the-data_1">#</a>
</h1>
<p><em>The graphic takes a moment to load.</em></p>
<iframe src="https://modeanalytics.com/benn/reports/c82a11aa3bda/embed" width="100%" height="1000"></iframe>
<p><em>Benn Stancil is the chief analyst of <a href="https://modeanalytics.com/">Mode</a>.</em></p>
tag:bennstancil.com,2014:Post/how-long-are-the-nba-playoffs2014-04-29T17:33:56-07:002014-04-29T17:33:56-07:00Are the Playoffs Taking Forever?<p><img src="http://cdn.fansided.com/wp-content/blogs.dir/229/files/2014/01/8172906.jpg" alt="tired"></p>
<p>We’re over ten days into this year’s NBA playoffs, and several of the opening series have made it all the way to…Game 5. The NHL playoffs, which are entering their third week but only the second round, are also in no apparent hurry.</p>
<p>So far, this year’s <a target="_blank" href="http://www.youtube.com/watch?v=fSVzjUP8dVs">exciting</a> <a target="_blank" href="http://www.cbssports.com/nhl/eye-on-hockey/24533022/video-vladimir-tarasenko-scores-with-64-seconds-remaining-to-tie-game">playoffs</a> are keeping critics quiet. But if tight overtime games and buzzer beaters give way to blowouts and snoozers, the <a target="_blank" href="http://bleacherreport.com/articles/733336-nba-playoffs-are-long-excessive-and-need-to-be-fixed">annual</a> <a target="_blank" href="http://online.wsj.com/news/articles/SB10001424052702304050304577374160670012118">complaints</a> <a target="_blank" href="http://slapshot.blogs.nytimes.com/2008/05/22/do-the-nhl-playoffs-drag-on-too-long/">about the length</a> of the NBA and NHL playoffs will probably resurface. </p>
<p>So I decided to take a detailed look. Sure, the playoffs feel long, but are they really?</p>
<p>Measuring playoff length isn’t actually a straightforward question. Because each league structures its playoffs differently, direct comparisons aren’t always appropriate. However, by attempting to standardize the pace and length across leagues—and by collecting data on every regular season and playoff game in the MLB, NBA, NFL, and NHL since 1970—we can draw some conclusions. The result? The problem might not be with the playoffs, but with us.</p>
<h3>How Long Do They Take?</h3>
<p>One obvious way to measure the length of the playoffs is by figuring out how long they take from beginning to end. The chart below shows the length of each league’s playoff since 1970. The NBA and NHL playoffs now last close to 60 days, which is considerably longer than both the MLB and NFL playoffs, and the NBA and NHL playoffs of previous decades.</p>
<iframe src="https://modeanalytics.com/benn/reports/8f2cd323fa2a/embed" width="100%" height="600"></iframe>
<p>Sixty days <em>feels</em> like a long time, but is it really? When compared to the relative length of the regular season, the NBA playoffs are again the longest. In recent years, the NBA playoffs have stretched to over a third of the length of the regular season, which is slightly longer than the NHL playoffs and nearly twice that of the MLB. It also happens to be twice as long as March Madness, which is about 15% as long as college basketball’s 130-day regular season.</p>
<iframe src="https://modeanalytics.com/benn/reports/b901020ad128/embed" width="100%" height="600"></iframe>
<p>Put another way, the 60 days of the NBA playoffs take up more of the year than Mondays.</p>
<iframe src="https://modeanalytics.com/benn/reports/78c98c31519e/embed" width="100%" height="300"></iframe>
<h3>The Frequency of Play</h3>
<p>The second way to measure the playoffs is by pace. As <a target="_blank" href="http://sports.espn.go.com/nba/playoffs/2010/news/story?id=5123337">many have said</a>, it’s not the length of the playoffs that are a problem—it’s the downtime. The NBA playoffs are riddled with off days, which means a seven-game series often drags on for over two weeks.</p>
<p>The chart below shows the average pace of playoff games for all the teams that went on to win a championship in the last 40 years. Somewhat surprisingly, the rate that teams play games hasn’t changed significantly in any sport.</p>
<iframe src="https://modeanalytics.com/benn/reports/5b27605b2f32/embed" width="100%" height="600"></iframe>
<p><em>A few notes about this graph: First, delays from the 1989 <a target="_blank" href="http://en.wikipedia.org/wiki/1989_Loma_Prieta_earthquake">Loma Prieta earthquake</a> caused the MLB spike. Second, the NFL occasionally eliminated the week off before the Super Bowl, creating the downward spikes in the NFL playoffs. Finally, as a minor technical detail, pace isn’t quite defined here as the number of days in the playoffs divided by the number of games. Instead, it’s the number of days minus one, divided by the number of games minus one. To understand why, imagine than an NFL team played two games on consecutive Sundays. The pace of those games is clearly one game every seven days, not two games in eight days, or one game every four days.</em></p>
<p>A more interesting measure of pace, however, is not how pace has changed from season to season—it’s how pace changes from the regular season to the postseason. </p>
<p>The chart below, <a target="_blank" href="https://xkcd.com/1331/">inspired by XKCD’s popular “Frequency” comic</a>, illustrates the relative pace of play. Each square flashes according to how often teams in that league play games. Hover over squares to see the associated value.</p>
<iframe src="https://modeanalytics.com/benn/reports/e2d75aa21ede/embed" width="100%" height="610"></iframe>
<p>As the graphic shows, the pace of games for all sports slows down. However, despite the perception of a dramatic slowdown in the NBA, the NBA’s pace <a target="_blank" href="https://modeanalytics.com/benn/reports/a19e10a3945f/results">only slows down by 25%</a>. That’s slightly less of a slowdown than that of the NFL and far less than the MLB’s, which slows its pace by nearly 50%. Even in absolute terms, the NBA and NHL playoffs actually slow less than the MLB and NFL, suggesting that pace may be more of a perceived problem than a real one. </p>
<h3>The Number of Games</h3>
<p>Even if two months isn’t “too long” and the gaps between games don’t grow as much as they seem to, the sheer volume of games might be another reason why the NBA and NHL playoffs feel like they take forever.</p>
<p>Since 2003, NBA and NHL teams have averaged about 85 playoff games a year. For both leagues, that’s more games than single team plays in a year. NFL teams play about three-fourths of a season in the playoffs, while MLB teams only play 20 percent.</p>
<iframe src="https://modeanalytics.com/benn/reports/6a1ce0ff9554/embed" width="100%" height="430"></iframe>
<p>Because professional sports’ playoffs are almost always on national TV and heavily covered by the sports media, it’s possible that raw number of games in the NHL and NBA playoffs wears on fans. This effect could be amplified by the <a target="_blank" href="http://www.google.com/trends/explore?hl=en-US&q=/m/09p14,+/m/059yj,+/m/05jvx,+/m/05gwr&cmpt=q&content=1#q=%2Fm%2F05jvx%2C%20%2Fm%2F05gwr&cmpt=q">relative lack of interest in the regular season compared to the playoffs</a>. Unlike <a target="_blank" href="http://www.google.com/trends/explore?hl=en-US&q=/m/09p14,+/m/059yj,+/m/05jvx,+/m/05gwr&cmpt=q&content=1#q=%2Fm%2F09p14%2C%20%2Fm%2F059yj&cmpt=q">baseball or football</a>, basketball and hockey draw far more attention during the playoffs than during the regular season, which likely means a higher percentage of playoff viewers are marginal fans who might grow impatient during the early rounds of the playoffs. Of course, it’s also possible the relationship goes the other way, and interest in the regular season is low <em>because</em> fans know they have an entire season of meaningful games to watch during the playoffs. But in either case, it seems plausible that complaints about the playoffs don’t reflect a change from the sport, but a change in fanbases.</p>
<p>The NBA and NHL playoffs could also feel particularly long because there are so many teams in these playoffs, and so much “noise” is required to find a champion. Over the last ten years, NBA and NHL champions have averaged playing 23 games a year, or <a target="_blank" href="https://modeanalytics.com/benn/reports/d02f713dc3d1/results">27% of total number of playoff games</a>. Super Bowl champions participate in an average of 33% of NFL playoff games, while World Series winners play in 45% of baseball playoff games. For casual fans—which the NBA and NHL playoffs appear to attract—the early rounds may feel like a long and unnecessary preamble to the NBA and Stanley Cup Finals.</p>
<p>So in the end, it’s likely not the pace of play that makes the playoffs feel sluggish. Instead, it’s likely the volume of playoffs we have to get through that’s the problem—and the fact that many of us complaining may not be committed fans in the first place.</p>
<h3>Data</h3>
<p>MLB game data was collected from <a target="_blank" href="http://www.retrosheet.org/">Retrosheet</a>. NBA, NFL, and NHL data was collected from <a target="_blank" href="http://www.sports-reference.com/">Sports Reference</a>. Charts were created using <a target="_blank" href="http://d3js.org/">D3</a>. All the analysis, complete datasets, and visualization code can be found in <a target="_blank" href="https://modeanalytics.com/">Mode</a>, and is free for anyone to use, copy, and modify.</p>
<p><em>Benn Stancil is the chief analyst of <a target="_blank" href="https://modeanalytics.com/">Mode</a>.</em> </p>
tag:bennstancil.com,2014:Post/finding-the-most-gerrymandered-district2014-04-16T11:35:44-07:002014-04-16T11:35:44-07:00Finding the Most Gerrymandered Districts<p><img src="http://www.csulb.edu/%7Eastevens/posc100/files/notes/nc12.jpg" alt="nc"></p>
<p>Yesterday, I came across an interesting <a href="http://www.vox.com/cards/gerrymandering-explained/what-is-gerrymandering">Vox.com</a> article discussing Congressional gerrymandering. In one of the article’s cards, author <a href="https://twitter.com/awprokop">Andrew Prokop</a> highlighted several of the country’s <a href="http://www.vox.com/cards/gerrymandering-explained/what-are-the-most-gerrymandered-house-delegations">most gerrymandered districts</a>. Having recently crunched some numbers on <a href="http://mode.github.io/blog/2014-02-06-signing-day/index.html">geographic data</a>, why not try to quantitatively define the most gerrymandered districts and states? </p>
<h1 id="defining-gerrymandering_1">Defining Gerrymandering <a class="head_anchor" href="#defining-gerrymandering_1">#</a>
</h1>
<p>As Prokop noted, there’s not a great way to determine if a district is gerrymandered. Nevertheless, researchers have <a href="http://www2.hawaii.edu/%7Ematt/104/Exercises/qgis-analysis-gerrymander.html">proposed a few ideas</a> to approximate it. The proposals largely measure gerrymandering in one of two ways: By calculating how far various points on the district’s boundary are from the district’s geographic center, and by comparing the perimeter of the district to that of a similar-sized district with a regular shape (in this case, a circle). Both calculations are far from perfect—the first calculation doesn’t work for noncontinuous districts, while the second is affected by any irregular boundaries, including coastlines and state borders—but they give decent estimates.</p>
<p>Because the effect created by irregular borders is both smaller and a bit easier to adjust for, I chose to rank districts by the ratio of the perimeter of the district to the circumference of a circle of equal area. The larger that ratio, the more gerrymandered it is. </p>
<p>Importantly, this ignores any political definitions for gerrymandering and assumes the best drawn districts are all circles. This assumption clearly <a href="http://www.pages.drexel.edu/%7Ebel23/HCP1.jpg">doesn’t work geometrically</a> or politically. A more true measure of gerrymandering would compare actual lines to those that make sense given the area’s geography and population distribution, but that adds significantly more complexity to the problem.</p>
<h1 id="ranking-the-districts_1">Ranking the Districts <a class="head_anchor" href="#ranking-the-districts_1">#</a>
</h1>
<p>After calculating the area and perimeter of each district as well as the circumference of a circle with the same area (discussed in more detail below), districts can be ranked by how severe they’re gerrymandered. The list below shows the top 10 most gerrymandered districts in the United States.</p>
<iframe src="https://modeanalytics.com/benn/reports/162dc70cc053/runs/48e5459eefd8/embed" width="100%" height="380"></iframe>
<p>The winner, North Carolina’s 12th District, is hardly a surprise. (According to <a href="http://en.wikipedia.org/wiki/North_Carolina's_12th_congressional_district">Wikipedia</a>, “It is an example of gerrymandering.”) <a href="http://en.wikipedia.org/wiki/Maryland's_3rd_congressional_district">Maryland’s 3rd</a> and <a href="http://en.wikipedia.org/wiki/Florida's_5th_congressional_district">Florida’s 5th</a> are also examples of some “creative” geographic interpretations. One unexpected addition to the list is <a href="http://en.wikipedia.org/wiki/Hawaii's_2nd_congressional_district">Hawaii’s 2nd</a>. In this case, Hawaii’s geography appears to be confusing the ranking, which misinterprets the district’s several islands as severely gerrymandered boundaries. It’s worth noting, however, that this method still ranks three other contiguous districts as having been gerrymandered <em>worse</em> than a district broken into several pieces.</p>
<p>For those interested in the full ranking of all 435 districts, it is available <a href="https://modeanalytics.com/benn/reports/162dc70cc053/runs/93e1fe6ac642/results">here</a>.</p>
<h1 id="ranking-the-states_1">Ranking the States <a class="head_anchor" href="#ranking-the-states_1">#</a>
</h1>
<p>Four of the top ten most gerrymandered districts are in North Carolina. This certainly makes North Carolina the frontrunner for the most gerrymandered state, but is it actually the worst?</p>
<p>It turns out that’s not a straightforward question to answer. You can’t simply sum each district’s gerrymandering scores because states with more districts tend to have much higher scores. Furthermore, taking the average of all the districts in each state doesn’t work either, because states with irregular borders or coastlines (like Alaska, Hawaii, Louisiana, and North Carolina) will appear more gerrymandered than they actually are. </p>
<p>To attempt to correct for these issues, I adjusted each state’s average score by how gerrymandered the state would appear if it were a single district. The more irregular the state border, the less penalized that state is for having irregular districts. </p>
<p>As the chart below shows, North Carolina is in fact the worst offender. In addition to ranking states, the chart also colors each state according to <a href="http://redistricting.lls.edu/who.php">who controls the process of drawing district lines</a>. States with split executive and legislative branches or with split legislative houses are categorized as having split control. States that only have one Congressional district are grouped in with those with independent commissions. </p>
<iframe src="https://modeanalytics.com/benn/reports/9aeb9c0ffbff/embed" width="100%" height="1100"></iframe>
<p>While table above implicates both parties, it does suggest that independent commissions, unsurprisingly, help reduce gerrymandering. Half of the ten least gerrymandered states (excluding those with one district) employ independent commissions, while only two of the ten most gerrymandered districts do so. (Additionally, many of the less gerrymandered states have fewer districts. This makes sense because fewer districts provide fewer opportunities for gerrymandering.)</p>
<p>Importantly, this isn’t the only way to rank states—there are other (likely better) ways to measure district gerrymandering, other ways to aggregate it, and additional political and population data that might be useful. If anyone would like to try a different method of analysis or add additional data, <a href="mailto:benn@modeanalytics.com">I’ll be happy to provide</a> full access to my data and work in <a href="https://modeanalytics.com/">Mode</a>.</p>
<h1 id="data_1">Data <a class="head_anchor" href="#data_1">#</a>
</h1>
<p>To rank Congressional districts, I needed to figure out two things: The area and perimeter of each district. Fortunately, D3, the JavaScript visualization library, can take do this pretty easily. Using several of D3’s <a href="https://github.com/mbostock/d3/wiki/Geo-Paths">geographic functions</a> and <a href="https://github.com/jsongeo/cd113">GeoJSON data</a> on Congressional districts, I was able to calculate each district’s (as well each state’s) area and perimeter. That raw data is available <a href="https://modeanalytics.com/benn/reports/256f025255f3/results">here</a>, as is the <a href="https://github.com/mode/blog/tree/master/2014-04-16-districts">script</a> that generated it (it’s in an .html file because I’m a hack). I. Note that the area data and perimeter data are in different units. Ratios between the two can be compared, but you can’t convert one to the other. Data on who controls the redistricting process was provided by <a href="http://redistricting.lls.edu/">Justin Levitt</a>.</p>
<p><em>Benn is the chief analyst of <a href="https://modeanalytics.com/">Mode</a>.</em></p>
tag:bennstancil.com,2014:Post/plotting-the-rest-of-the-baseball-season2014-04-11T09:19:51-07:002014-04-11T09:19:51-07:00Plotting the Rest of the Baseball Season<p><a href="http://espn.go.com/mlb/photos?gameId=340409122&photoId=3758893"><img src="http://a.espncdn.com/combiner/i?img=media%2Fapphoto%2F54ed311514eee80e500f6a7067006dfe.jpg&w=715&h=516" alt="brewers"></a></p>
<p>We’re less than two weeks into the 2014 baseball season, and <a href="http://mlb.si.com/2014/04/04/seattle-mariners-san-francisco-giants-miami-marlins-surprise-teams/?xid=nl_siextra">most people</a> would say that it’s too early to make any forecasts about the rest of the year. </p>
<p>Still, <a href="http://fivethirtyeight.com/datalab/how-to-predict-mlb-records-from-early-results/">as others have noted</a>, though ten games only represents 6% of an MLB season, surely these early games provide <em>some</em> indication of how a team will finish. Does Milwaukee’s 7-2 start mean that they may not be the <a href="http://grantland.com/features/2014-mlb-season-predictions/">sub-.500</a> team they were predicted to be? Are the 4-8 Diamondbacks likely to even worse than expected?</p>
<p>The graphic below explores this question. It plots the full season for every team over the last 10 years, or 300 seasons in total. By filtering by record, you can see how teams with similar starts fared over the rest of the season, and how this compares to an average season. (Because the graphic is loading nearly 50,000 games, it takes moment to first display.)</p>
<iframe src="https://modeanalytics.com/benn/reports/c457da98ccea/embed" width="100%" height="660"></iframe>
<p><em>Note that for records with fewer than 5 teams, the graphic expands the win filter to include at least 5 teams. For example, because only one team started 6-0, the graphic shows teams that started both 6-0 and 5-1.</em></p>
<p>As the graphic shows, there’s generally a lot of noise early in the season. A small bit of history, however, is on the Brewers’ side. Based on the 15 teams that also started 7-2, the Brewers have around an 80% chance to finish above .500. In Arizona, things don’t look too desperate yet: Though teams that start 4-8 average 6 fewer wins than the overall mean, the distribution is still quite wide and many teams finish above .500.</p>
<p>Of course, forecasting (in the loosest sense of the word) future records based on teams’ current records is a very simple way to approach this problem. There are a number of other factors, such as runs scored, runs allowed, run differentials, results against other strong or weak teams, and results in home and away games, that could be indicative of future success.</p>
<p>As an example, run differentials could provide more information than just win totals. The table below shows the relationship between run differentials <em>in wins</em> and season win totals. As it shows, teams that win by more runs in their first 30 games tend to have better seasons. While that’s unsurprising, it suggests that wins alone is probably a crude metric, especially early in seasons when the sample is so small.</p>
<iframe src="https://modeanalytics.com/benn/reports/9736314b9f59/embed" width="100%" height="610"></iframe>
<p>For anyone interested in exploring this data further, <a href="http://www.retrosheet.org/">Retrosheet</a> data is available in <a href="https://modeanalytics.com/">Mode</a> for every season since 1980. This data can be analyzed, visualized, and shared directly through Mode, and <a href="mailto:benn@modeanalytics.com">I can provide access</a> to anyone who is interested. If you’d like to modify or double-check any of my <a href="https://modeanalytics.com/benn/reports/c457da98ccea/results">analysis</a> <a href="https://modeanalytics.com/benn/reports/9736314b9f59/results">above</a>, you can click through the embed links to access the work and data directly.</p>
<p><em>Benn Stancil is the chief analyst of <a href="https://modeanalytics.com/">Mode</a>.</em></p>
tag:bennstancil.com,2014:Post/fivethirtyeight-vs-the-oddsmakers2014-03-27T21:37:03-07:002014-03-27T21:37:03-07:00FiveThirtyEight vs. The Oddsmakers<p><img src="http://myspiltmilk.com/sites/default/files/the-wire-omar_1222162169_crop_506x316.jpg" alt="lame"></p>
<p><em>You come at the king, you best not miss. - Omar Little</em></p>
<p>At the start of this year’s NCAA tournament, <a href="http://fivethirtyeight.com/">FiveThirtyEight</a>, the new website of reigning forecast champion Nate Silver, <a href="http://fivethirtyeight.com/interactives/march-madness-predictions/">predicted each team’s chances</a> of making it to different rounds of the tournament. In an update yesterday, FiveThirtyEight looked into <a href="http://fivethirtyeight.com/datalab/the-ncaa-bracket-checking-our-work/">how their forecasts were doing</a>. Having made my own <a href="http://blog.modeanalytics.com/bracket-odds/">predictive bracket</a> based on Las Vegas odds, I figured I’d do the same—and see who comes out on top.</p>
<h1 id="how-did-fivethirtyeight-do_1">How Did FiveThirtyEight Do? <a class="head_anchor" href="#how-did-fivethirtyeight-do_1">#</a>
</h1>
<p>Rather than simply forecasting winners, FiveThirtyEight’s predictions—like mine—calculate each team’s probability of winning every game. To assess how well these forecasts performed, it’s not appropriate to see how many of their “favorites” won. Instead, it’s better to see if favorites win more or less often than expected. In other words, if FiveThirtyEight identified 100 games in which the favorite had a 60% chance of winning, the favorite should actually win 60 of them. If the results are substantially different from that, then it’s an indication that something’s wrong with the model.</p>
<p>Over the last several years, Silver’s predictions have performed well. The chart below—reproduced using <a href="https://github.com/fivethirtyeight/data/blob/master/historical-538-ncaa-tournament-model-results.csv">data provided by FiveThirtyEight</a>—compares game results to FiveThirtyEight’s forecasts. As it shows, if you group games by the predicted odds of the favorite winning, the actual results are close to that range.</p>
<iframe src="https://modeanalytics.com/benn/reports/b26debec2ccc/embed" width="100%" height="430"></iframe>
<p>As Silver noted in his post, though the results in each bucket don’t precisely match the forecast, they fall reasonably close and well within his confidence intervals. Silver’s model, it appears, works reasonably well.</p>
<h1 id="silver-vs-vegas_1">Silver vs. Vegas <a class="head_anchor" href="#silver-vs-vegas_1">#</a>
</h1>
<p>While FiveThirtyEight’s bracket is based on <a href="http://fivethirtyeight.com/features/nate-silvers-ncaa-basketball-predictions/">team rankings</a> and a few other factors, I based <a href="http://mode.github.io/blog/2014-03-18-bracket/2014.html">my bracket</a> solely on Las Vegas odds. Though the predictions are different, our brackets’ favorites are all the same, except for the Championship game. FiveThirtyEight gave a slight edge to Louisville over Florida, while Vegas preferred Florida by a slim margin.</p>
<p>Unfortunately, though Silver’s predictions go back to 2011, I only made forecasts for the tournaments last year and this year. To make the comparison equal, I first trimmed Silver’s data to include only 2013 and 2014 results. The chart below shows the same calculations as above for FiveThirtyEight (the buckets were made larger to adjust for the smaller sample size).</p>
<iframe src="https://modeanalytics.com/benn/reports/224d4ebecf90/embed" width="100%" height="430"></iframe>
<p>Unsurprisingly, with a smaller sample (especially one that includes the chaos of last year’s tournament), Silver’s model looks tarnished (sorry). Still, the trend is generally in the right direction. </p>
<p>Compared to my predictions using Vegas odds, however, Silver regains his luster. Vegas (or my method of interpreting Vegas) performs worse than FiveThirtyEight. As the chart shows—which overlays my model’s results with Silver’s—the model does a particularly poor job of identifying solid but not overwhelming favorites: Favorites only won half the the games in which they were expected to win 70% to 80% of the time. </p>
<iframe src="https://modeanalytics.com/benn/reports/1fad8839bf1b/embed" width="100%" height="420"></iframe>
<h1 id="why-the-difference_1">Why The Difference? <a class="head_anchor" href="#why-the-difference_1">#</a>
</h1>
<p>Before conceding to Nate Silver’s sterling record and accepting that he is just <a href="http://isnatesilverawitch.com/">better at this than me</a>, it’s worth looking into why our predictions came out so differently. Fortunately, there’s a fairly clear explanation. Silver’s recalculates his forecasts as the tournament progresses, updating the predictions after each game. This has two effects. First, his calculations respond to positive and negative signals from previous games. For instance, Virginia’s blowout win against Memphis could improve their odds against Michigan State, or Iowa State’s loss of Georges Niang to injury could lower their chances against Connecticut. My calculations were based on Vegas odds at the beginning of the tournament, and not responsive to new results.</p>
<p>Second—and more importantly—for all the games beyond the first round (when matchups were unknown), I computed game odds by comparing each team’s odds of making the Final Four. This is an imperfect calculation, most notably because <em>those odds are based on a team’s entire path to the Final Four</em>. For teams that face very challenging first games, their odds of making the Final Four are quite low. However, my model doesn’t make any adjustments for teams that overcome this first game.</p>
<p>Florida Gulf Coast’s Cinderella run last year is a perfect example of both of these cases. Not only did Florida Gulf Coast demonstrate that they were a better team than many thought by beating Georgetown and San Diego State by a total of 20 points, but they also cleared two tough hurdles between them and the Final Four. In part because of both of these factors, Silver’s model gave Florida Gulf Coast a 5.8% chance against Florida. My model—which was still based on Florida Gulf Coast’s original 1% chance of making the Final Four—only gave them a 0.2% chance against Florida.</p>
<p>This problem, however, can be partially corrected by only looking at first round games. Because these match-ups are known, forecasts are based on actual game lines rather than derived matchups.</p>
<p>As suspected, the difference in projections are less apparent in the first roud. Charting the relative predictions for each game shows that, in the first round, the differences between models are more or less random, and clustered around zero. In later rounds—when I’m deriving probabilities—Vegas odds almost universally overestimate the favorite’s chances. This makes sense, given the Florida Gulf Coast example above.</p>
<iframe src="https://modeanalytics.com/benn/reports/70ad1b9e53b8/embed" width="100%" height="430"></iframe>
<p>As this suggests, looking at only games in the first round, the models fair similarly. The chart below shows the same buckets as before, <em>but only includes first round games</em>—and in this case, the model predictions are more closely aligned.</p>
<iframe src="https://modeanalytics.com/benn/reports/9796ea87e382/embed" width="100%" height="430"></iframe>
<p>Based on this, when picking your bracket next year, it doesn’t really matter if you go with Vegas or Nate Silver in the first round. For later round games, <a href="https://www.youtube.com/watch?v=WP-lrftLQaQ">Nate Silver has left me hiding behind as car as he whistles <em>The Farmer in the Dell</em></a> (WARNING: that link is a Wire spoiler). But this does raise an interesting question: If FiveThirtyEight predicted every matchup at the start of the tournament, how would their results look? And how do FiveThirtyEight’s forecasts compare to Vegas lines at the start of each game? In other words, if we level the playing field, should I bet on Nate Silver or the true kings of sports forecasting—the oddsmakers?</p>
<h1 id="data_1">Data <a class="head_anchor" href="#data_1">#</a>
</h1>
<p>Data was collected from FiveThirtyEight and calculated using <a href="http://www.vegasinsider.com/college-basketball/odds/las-vegas/money/?s=2885">Vegas</a> <a href="http://www.vuhoops.com/2014/3/17/5518502/ncaa-tournament-2014-odds-to-win-national-championship-and-regions">odds</a>. All analysis, data, and visualization code can be found in <a href="https://modeanalytics.com/">Mode</a>. The graphs are backed by <a href="https://variancecharts.com/">Variance</a>, an excellent new visualization library.</p>
<p><em>Benn Stancil is the chief analyst of <a href="https://modeanalytics.com/">Mode</a>.</em></p>
tag:bennstancil.com,2014:Post/the-odds-of-your-ncaa-bracket2014-03-18T10:55:04-07:002014-03-18T10:55:04-07:00The Odds of Your NCAA Bracket<p><img src="http://blog.vegas.com/wp-content/uploads/2011/02/MarchMadness.jpg" alt="ncaa"></p>
<p>This year, Warren Buffett promised a billion dollars to anyone who picks a perfect bracket. Unfortunately, the odds aren’t in your favor—the chance of picking a perfect bracket if you pick every game at random is one in <a href="http://www.cbssports.com/collegebasketball/eye-on-college-basketball/24441831/bracket-voodoo-why-you-wont-win-the-billion-dollar-bracket">9 quintillion</a> (or 9,000,000,000,000,000,000).</p>
<p>But that’s just a hypothetical bracket. What are the odds of <em>your</em> bracket? The <a href="http://mode.github.io/blog/2014-03-18-bracket/2014.html">interactive</a> below lets you figure that out. Using the betting lines for <a href="http://www.vegasinsider.com/college-basketball/odds/las-vegas/money/?s=2885">each game</a> and on each team’s <a href="http://www.vuhoops.com/2014/3/17/5518502/ncaa-tournament-2014-odds-to-win-national-championship-and-regions">chances of the making the Final Four and winning the NCAA Championship</a>, the graphic calculates the odds of every possible NCAA matchup—and every possible NCAA bracket. The graphic also shows how each team affects your bracket’s odds, and which picks lower your chances of winning a billion dollars the most.</p>
<p><strong><em><a href="http://mode.github.io/blog/2014-03-18-bracket/2014.html">Click on the bracket to see the interactive</a></em></strong></p>
<p><a href="http://mode.github.io/blog/2014-03-18-bracket/2014.html"><img src="http://i.imgur.com/vsHOa0d.png" alt="bracket"></a></p>
<p>The odds for first round games are calculated using the betting lines for those games. In matchups between two teams after the first round, each game’s odds are calculated using the relative odds that the two teams make the Final Four (in the case of regional games) or the relative odds that the two teams win the championship (in the case of Final Four games). </p>
<p>To figure out how much a team contributes to lowering your bracket’s odds, I compared the odds of the selected bracket to the odds of a bracket in which that team has a 100% chance of winning every game you picked them to win. The more this adjustment increased the odds of the bracket (that is, made the bracket more likely it made it), the larger that team’s circle.</p>
<p>The data that powers the bracket, as well as the code for the visualization, are in this <a href="https://github.com/mode/blog/tree/master/2014-03-18-bracket">GitHub folder</a>.</p>
<p><em>Benn Stancil is the chief analyst of <a href="http://www.modeanalytics.com/">Mode</a>.</em></p>
tag:bennstancil.com,2014:Post/engineering-a-best-picture2014-02-27T10:00:41-08:002014-02-27T10:00:41-08:00Engineering a Best Picture<p><img src="http://extras.mnginteractive.com/live/media/site568/2013/0224/20130224_102725_OSCARS_500.jpg" alt="oscars"></p>
<p>When Netflix wanted to create a hit TV show, it turned to data. By <a href="http://www.huffingtonpost.com/2013/01/24/netflix-show-house-of-cards_n_2545332.html">analyzing its viewers habits</a>, Netflix uncovered that its customers particularly liked Kevin Spacey, director David Fincher, and political thrillers. In part because of these interests, Netflix brought the three together to create <em>House of Cards</em>—and thus far, the <a href="http://www.huffingtonpost.com/2014/02/21/house-of-cards-binge_n_4831053.html">results have been tremendous</a>.</p>
<p>Having binged our way through Season 2 of <em>House of Cards</em>, we in the entertainment world now turn our attention to the Oscars, and particularly, the race for Best Picture. In doing so, perhaps we could take a page from Netflix’s book. Perhaps, using data about movies and the relationships between them, we can identify a perfect cocktail of movie attributes—<a href="http://www.imdb.com/title/tt1504320/">PG-13-rated</a> <a href="http://www.imdb.com/title/tt0436697/">biopics</a> <a href="http://www.imdb.com/title/tt0350258/">about</a> <a href="http://www.imdb.com/title/tt1285016/">celebrities</a>, or <a href="http://www.imdb.com/title/tt0108052/">heart-wrenching World War II stories</a> <a href="http://www.imdb.com/title/tt0120815/">directed by Steven Spielberg</a>, or anything related to <a href="http://www.youtube.com/watch?v=p82viKpyrzQ">Michael Bay</a>—that strikes every Best Picture nerve. Perhaps, just like a hit TV series, a Best Picture can be engineered.</p>
<p>Using data collected from <a href="http://www.rovicorp.com/">Rovi</a>, I explored attributes that define Best Pictures. In addition to finding <a href="http://www.vanityfair.com/hollywood/2014/03/oscar-winner-predictions-nate-silver">characteristics that frequently appear in nominees for Best Picture</a>, I also looked for what made candidates for Best Picture stand out from other Oscar nominees. For example, it’s well known that most Best Picture nominees are dramas, but hundreds of dramas are released every year. Maybe there are rare, niche genres that only produce a few films a year—but always get noticed by the Academy.</p>
<p>Furthermore, concluding “we should make a drama” isn’t very instructive. Why not sketch out the entire movie, complete with a plot, themes and tones, and a cast and crew?</p>
<p>The following does exactly that. I first define the rough outline and plot of the movie, and cast it and pick a crew. I then built a model to blend together movie titles and synopses from Oscar nominated-films. The result is the ultimate Frankensteinian Oscar-bait—and like Frankenstein, it could be a <a href="http://en.wikipedia.org/wiki/Young_Frankenstein">triumph</a> or an <a href="https://www.youtube.com/watch?v=sCUZd1iuq3o">abomination</a>.</p>
<h1 id="sorting-out-the-basics_1">Sorting out the Basics <a class="head_anchor" href="#sorting-out-the-basics_1">#</a>
</h1>
<p>When engineering a Best Picture, a few elements are essential. First, popular opinion about dramas is correct—dramas do much better than other genres. Fifty percent of movies nominated for any Oscar are dramas, compared to 75 percent for Best Picture nominees. However, there’s a stronger bias for crime dramas and biographical films, suggesting that some specialization could be beneficial. </p>
<p>On the other end of the spectrum, comedy, science fiction, action, and horror movies are all under-represented in Best Pictures. Out of the roughly 550 movies of these types nominated for Oscars, less than 30 were nominated for Best Picture.</p>
<p>Second, the film shouldn’t be a sequel. Though sequels do well at the box office (for example, <em>Transformers</em> 2 and 3; <em>The Matrix</em> 2 and 3; <em>Spider-Man</em> 2 and 3; <em>Batman</em> 2 and 3; <em>Pirates of the Caribbean</em> 2, 3, and 4; <em>Twilight</em> 2, 3, 4, and 5; and <em>Harry Potter</em> 2, 3, 4, 5, 6, 7, and 8), they rarely impress Oscar voters. Only three of over 80 sequels nominated for Oscars were nominated for Best Picture.</p>
<p>While the Academy shuns sequels for Best Pictures, it actually favors adapted writing over original writing. Over 50 percent of the nominees for Best Adapted Screenplay (or a past variant of the award) were also nominated for Best Picture, but less than 40 percent of Best Original Screenplays garnered Best Picture nominations.</p>
<h1 id="constructing-a-plot_1">Constructing a Plot <a class="head_anchor" href="#constructing-a-plot_1">#</a>
</h1>
<p>This gives us the basic outline of a film—it should be well-written adapted biopic about crime. But <a href="http://www.independent.co.uk/arts-entertainment/films/features/perfect-picture-what-makes-a-movie-most-likely-to-be-nominated-for-an-academy-award-9065504.html">other studies</a> confirmed these discoveries. Surely we can be more specific. Fortunately, Rovi provides detailed classifications of films’ tones and moods, and identifies specific characteristics and keywords related to their plots. Using these attributes, I developed a more well-defined Best Picture.</p>
<p>The table shows the themes and plot elements most over-represented in Best Picture nominees (it excludes characteristics that only appear in very few films). Broadly, Best Picture nominees should take on sweeping themes and be bittersweet and compassionate. They should never be goofy, silly, or campy.</p>
<p><img src="http://i.imgur.com/js5LR9W.png" alt="elements"></p>
<p>Though the tone of the films are best in a minor key, they should end triumphantly. Light, just-for-fun adrenaline rushes struggle (apologies to Michael Bay).</p>
<p>Regarding the specifics of the plot, movies about cross-cultural relations and forbidden loves are strong performers. The best cultural differences to explore are those that emerge from economic inequality—stories about class differences and servants and employees are among the Academy’s favorites. Social injustices, particularly those that address racism and mental illness are well-received, though movies that touch on injustices done to Native Americans may be ignored. Other important characteristics to avoid are kidnapping, self-referential movies about filmmaking, pregnancy, and evil aliens (apologizes to Michael Bay).</p>
<p>Though these plotlines are solid bets, they’re all also well-trodden—what if we want to try something a little edgier? Though they’re infrequently made (and excluded from the table above), films about wheelchairs and farm life have played well to Best Picture voters. A movie about the struggles of a wheelchair-ridden farmer, perhaps?</p>
<p>Finally, the movie should present these difficult and complex themes in depth and without censorship. Oscar-nominated films are an average of 114 minutes long, Best Picture nominees are an average of 130 minutes long, and Best Picture winners are 142 minutes long. Moreover, the film should be R-rated: 40 percent of Oscar-nominated films are R-rated, compared to 50 percent of Best Picture nominees.</p>
<h1 id="casting-a-best-picture_1">Casting a Best Picture <a class="head_anchor" href="#casting-a-best-picture_1">#</a>
</h1>
<p><a href="http://www.gigapunch.com/wp-content/uploads/2013/04/jar-jar-binks.jpg">Bad acting</a> can ruin a film. Can good acting make a movie a Best Picture?</p>
<p>Though it doesn’t appear critical, casting good actors certainly helps: About 60 percent of films nominated for best actor or best supporting actor were nominated for Best Picture. Sadly, the gendered “actor” isn’t incidental—only around 45 percent of films recognized for outstanding performances by actresses are nominated for Best Picture. </p>
<p>When filling the roles, we may be inclined to turn to the greats like Meryl Streep, Tom Hanks, and Jack Nicholson. These three—along with Harrison Ford, Dustin Hoffman, Robert De Niro and Leonardo DiCaprio—have been in a number of films nominated for Best Picture, but they’ve also been in many great films that weren’t nominated for Best Picture. When casting the film—especially if it’s on a budget—we want actors and actresses that collect Best Picture nominations as efficiently as possible. </p>
<p>Tragically, the undisputed champion of the Best Picture nod—<a href="http://www.imdb.com/name/nm0001030/">John Cazale</a>—died 35 years ago. Remarkably, Cazale appeared in five films in his career, and all five were nominated for Best Picture. Outside of Cazale, <a href="http://www.imdb.com/name/nm0000358/">Daniel Day Lewis</a> and <a href="http://www.imdb.com/name/nm0000199/">Al Pacino</a> are the best male leads. For female leads, <a href="http://www.imdb.com/name/nm0680983/">Ellen Page</a> and <a href="http://www.imdb.com/name/nm1567113/">Jessica Chastain</a> are excellent choices. Despite their commercial success, <a href="http://www.imdb.com/name/nm0000125/">Sean Connery</a>, <a href="http://www.imdb.com/name/nm0000191/">Ewan McGregor</a>, <a href="http://www.imdb.com/name/nm0000552/">Eddie Murphy</a>, <a href="http://www.imdb.com/name/nm0268199/">Colin Farrell</a>, and <a href="http://www.imdb.com/name/nm0000215/">Susan Sarandon</a> have all had little success with Best Picture nominations and are all stars to avoid.</p>
<p>To round out the supporting cast, <a href="http://www.imdb.com/name/nm0101710/">Billy Boyd</a>—a poor man’s John Cazale—is an obvious first choice. All four of the Oscar-nominated movies in which Boyd has had parts (all three <em>Lord of the Rings</em> films and <em>Master and Commander</em>) were nominated for Best Picture. After Boyd, <a href="http://www.imdb.com/name/nm0727300/">Shane Rimmer</a> and <a href="http://www.imdb.com/name/nm0148149/">Peter Cellier</a> are among the best men, while <a href="http://www.imdb.com/name/nm0001584/">Miranda Otto</a> and <a href="http://www.imdb.com/name/nm0001735/">Talia Shire</a> stand out among the women.</p>
<p>(As an aside, this film has run into a minor issue at this point. The Academy indirectly told us that we should focus on issues about race. However, the Oscars have historically favored white men. Maybe Oscar-nominee <em>The Last Samurai</em> found the solution to this dilemma—make the hero of the oppressed race…Tom Cruise.)</p>
<h1 id="finding-a-worthy-crew_1">Finding a Worthy Crew <a class="head_anchor" href="#finding-a-worthy-crew_1">#</a>
</h1>
<p>A quality crew is perhaps even more important than the cast. Seventy-five percent of films nominated for Best Director were also nominated for Best Picture, which is the strongest overlap Best Picture nominees have with any other Oscar category. </p>
<p>To lead the crew, Martin Scorsese is a clear choice for the director. He has more Best Picture nominations than any other director, and has collected them with amazing efficiency. Beyond Scorsese, Norman Jewison, Ang Lee, and James Brooks would all be strong choices. On the other side of the coin, Tim Burton and Michael Bay have been very successful at having their films nominated for Oscars, but not as Best Pictures (apologizes to Michael Bay).</p>
<p>The “Moneyball” picks for the cast are editor <a href="http://en.wikipedia.org/wiki/Thelma_Schoonmaker">Thelma Schoonmaker</a> and cinematographer <a href="http://en.wikipedia.org/wiki/Robert_Richardson_(cinematographer)">Robert Richardson</a>. The two have been involved in a total of 27 Oscar nominated films, 15 of which were nominated for Best Picture. Moreover, films nominated for best editing and best cinematography were also nominated for Best Picture at rates above 50 percent, suggesting these two may provide the most bang-for-their buck out of any cast or crew member. Notably, Schoonmaker typically works with Scorsese, so she may be riding his coattails—or he may be riding hers.</p>
<p>When looking for other crew members, we should focus on sound over visual elements. Forty-two percent of films nominated for Best Score received Best Picture nominations. (Interestingly, the rate was only 14 percent for films nominated for Best Song.) By contrast, the overlaps between Best Picture nominees and Best Costumes, Best Makeup, and Best Visual Effects nominees are among the weakest of any Oscar category, excluding those dedicated to particular genres like documentaries or shorts. </p>
<p>Finally, the Weinstein brothers easily top the list as the best candidates to bankroll the movie. Like Scorsese, they’ve been remarkably successful in getting films they produce nominated for Best Picture, and done so with impressive efficiency. </p>
<h1 id="embullet-to-the-dark-sideem_1">
<em>Bullet to the Dark Side</em> <a class="head_anchor" href="#embullet-to-the-dark-sideem_1">#</a>
</h1>
<p>Based on this outline, <em>12 Years a Slave</em> appears to be the clearest Oscar-bait among this year’s Best Picture nominees. It’s a dramatic biopic; crime is central to the plot (though one of the chief crimes is kidnapping); it was nominated for Best Directing and Best Editing; and the arc of the plot—a hopeless social injustice, accented with cross-cultural relationships, breaks the audience’s spirit before ending in poignant salvation—fits the mold perfectly. </p>
<p>But we can do better. The cast could be improved. At “only” 134 minutes, it could be longer. And imagine if Scorsese had directed it. </p>
<p>What film would be better? Using a <a href="http://en.wikipedia.org/wiki/N-gram">model</a> to blend titles and plot synopses of all the Oscar nominees over the last five decades, I generated several potential Best Picture-worthy titles and plot summaries. Among the randomly-generated plots and titles produced by the model, the two titles and six plots below best fit the themes and tones recommended from the analysis above (I paired the synopses with the titles they best matched). Some of the results could be clear winners; others, however, might need some of that Schoonmaker magic…</p>
<h3 id="suggested-title-1-embullet-to-the-dark-sideem_3">Suggested Title 1: <em>Bullet to the Dark Side</em> <a class="head_anchor" href="#suggested-title-1-embullet-to-the-dark-sideem_3">#</a>
</h3>
<ul>
<li>
<strong>Possible Plot 1:</strong> A portrait of a newlywed couple who are reunited in the Afghan mountains.</li>
<li>
<strong>Possible Plot 2:</strong> A ‘50s housewife and a disgraced cop team up to exact revenge upon her one-time lover. </li>
<li>
<strong>Possible Plot 3:</strong> A crooked cop tries to obtain the ultimate Dalmatian coat.</li>
</ul>
<h3 id="suggested-title-2-emhurt-me-the-hidden-worlde_3">Suggested Title 2: <em>Hurt Me the Hidden World</em> <a class="head_anchor" href="#suggested-title-2-emhurt-me-the-hidden-worlde_3">#</a>
</h3>
<ul>
<li>
<strong>Possible Plot 1:</strong> A suicidal former Union soldier ends up joining a Sioux tribe. He then takes up arms to defend them when they become entangled with Russian mobsters in London. </li>
<li>
<strong>Possible Plot 2:</strong> A farmer tries to woo a wealthy uncle, meets and falls for an agnostic Roman soldier during WWII. </li>
<li>
<strong>Possible Plot 3:</strong> A rich playboy who escapes from prison to reunite their divorced dad poses as an eccentric teacher at an unconventional brothel.</li>
</ul>
<p>So to all the aspiring writers and filmmakers in the world, you now know what to do. The path before you is clear. Six outstanding movies are practically written. The necessary themes and plot twists are known. All that’s left to do is assemble the right cast and crew, and collect the inevitable hardware.</p>
<h1 id="data_1">Data <a class="head_anchor" href="#data_1">#</a>
</h1>
<p>I collected data via the <a href="http://developer.rovicorp.com/page">Rovi Cloud Services API</a>. While Rovi provides an impressive amount of data on each film, the dataset still has a few holes, most notably regarding the awards each film was nominated for (data on Best Picture nominees is complete). Additionally, Rovi provided no data on about 30 of the 2,900 films that were nominated for an Oscar over the last fifty years.</p>
<p>To determine the top attributes in a Best Picture, I found which attributes were most over-represented in Best Pictures relative to all Oscar nominees. Unless otherwise noted, when finding top themes and plot elements, I only considered those attributes that appeared in at least 10 of the nearly 3,000 Oscar nominated movies. The list of Oscar winners was collected from the <a href="http://awardsdatabase.oscars.org/ampas_awards/BasicSearchInput.jsp">Academy Awards Database</a>.</p>
<p>While comparing Best Picture nominees to other Oscar nominees rather than all films introduces some bias (Oscar nominees aren’t a perfect sample of all movies), it has benefits as well. The dataset is restricted to movies that had some degree of critical or popular success, excludes made-for-TV movies, and largely focuses on American films (few foreign films are nominated for Best Picture). </p>
<p>The title and synopsis mash-ups were randomly generated using a Markov n-gram model trained on a dataset of all Oscar nominees. Because the set of Best Picture nominees is small, creating n-gram models using only Best Picture titles and synopses unfortunately isn’t possible. </p>
<p>To anyone who is interested in this work and would like to explore other methods for characterizing Best Picture nominees, I’m happy to share all my analysis. The conclusions presented here are a simple start to figuring out what makes a Best Picture; like so many other analyses, it could be greatly strengthened by others’ data and others’ ideas.</p>
<p><em>Benn Stancil is the chief analyst at <a href="http://www.modeanalytics.com/">Mode</a>.</em></p>