Tommy Morrison

Tommy Morrison

Research and Data Officer @ NRGI; opening up data and building the tools to use it

@GreenGridUSA: a green data twitter bot

1 minute read

Published:

I launched a new Twitter account last week: @GreenGridUSA! This is an automated Twitter bot that parses data and posts insights from the U.S. Energy Information Administration (EIA) data portal. The EIA has a ton of in-depth and timely data on the US electric grid, including on power plants, transmission, power demand and power generation, and I wanted to put something together using it.

Currently, @GreenGridUSA posts a daily update on how much utility-level power generation came from wind and solar power plants the prior day. This data is available here on the hourly electric grid monitor, with about a twelve hour delay for this dataset.

I plan to add in a monthly and perhaps weekly review with some more detailed analysis of the green grid, perhaps drilling down by region or balancing authority. But I need to do a little research first to find out what might be worthwhile to look into. Beyond that, I’ll look into combining the EIA data with some other data sources.

This is the first Twitter bot I’ve made. I use the rtweet package to interact with the Twitter API and post the tweets, while the data transformation, analysis and visualization is done primarily with data.table and ggplot2. To use the Twitter API, you need to create a developer account and get access to your accounts API keys. The EIA does offer an API for data access, which is convenient, but it is one-dimensional; you can download most any of their data series, but you get the whole thing, there is no way to limit the days or query within the series. For the hourly electric grid data, this means you get 20,000+ rows each time you pull down data, even if you just want yesterday’s figures. The program runs via cronjob on a small AWS EC2 instance.

As of this writing, we are at 2 followers, including myself! Here’s to a few more, hopefully.

You May Also Enjoy

The best thing I ever did with R

2 minute read

Published:

A few years back, I created an RStudio shortcut that has ended up being my most-used function in R, probably by far. What does the shortcut do? It wraps the current line in the RStudio editor (or the console, if the cursor is there) in a View() function, which opens the data.frame in RStudio’s built-in data viewer.

COVID-19 in U.S. Schools[’ websites]

3 minute read

Published:

Mid-way through the summer, it was looking pretty clear that the pandemic would not be over by the academic year, and school re-opening was uncertain all over the country. I was interested in how many public schools would be having in-person or virtual classes, and whether there would be a shift in schools re-opening or re-closing as the pandemic progressed. I couldn’t find any hard data on this, so decided to curate my own by collecting and analyzing unstrucuted data from school websites.

R to Slack from Scratch

5 minute read

Published:

I was recently automating a number of steps to gather, clean, and process data in a data pipeline at work. These steps are run daily from a Docker container and do things like scrape web pages for new data and check the status of data in our pipeline. Eventually, these automated steps will lead to some needed human intervention, like manually reconciling data values. I used to run these steps myself and see their output, so I would know what needed to be done; now, I’d need some other way to keep up-to-date on the pipeline without checking myself every so often.