The Partnership for Public Service has expertise on the presidential transition process that few others possess. One of the key parts of this process — federal agency appointments — is an indicator not only of the priorities being set by the incoming Administration, but also of the transition’s competence.

The Partnership knew during 2016 that the appointment process was being largely neglected by the Trump Administration. However, in 2019, after Michael Lewis’s book THE FIFTH RISK had shown the issues arising from an Administration that did not provide adequate continuity support to federal agencies, the Partnership wanted to know what the state of personnel vacancies was. How did it stack up to past Administrations? Was the Trump Administration just slow to start?

All federal appointment and nomination data is public record, available on congress.gov, but is not downloadable or viewable in batch. The Partnership asked me a simple data efficiency question: was there any way to save a staffer from having to trawl through the nomination search screen over and over again and just get this information in a flat file?

On Amtrak from NYC back to DC, I built a rather simple web scraper that first generated the search URLs necessary based on params passed to the program. Then, the scraper does what scrapers do best — processed the resulting HTML pages into a digestible CSV of nomination records. My contact at the Partnership wasn’t Python literate, so I created extensive documentation that instructed her how to rerun this script daily or weekly to refresh the data, or to get it to pull down data for, say, previous presidents. At their request, I haven’t publicly posted the code (so drop me a line if you’re interested).

We found, not particularly to anyone’s surprise, that the state of federal appointments was tracking far behind the Bush and Obama Administrations in both pace and magnitude.

Read more about the Partnership’s work on improving the USA's civil service here.