The Prediction of Flight Delays within the Data Science pipeline: Part I
Introduction
This is the first of a series of posts dealing with a complete example of data analysis, starting from data gathering and finishing with the publication or dissemination of the results. It is the first post of the planned series dealing with the entire Data Science work flow, see The Prediction of Flight Delays within the Data Science pipeline, and is seen as the first version of steps 1 to 10 in that list.
We use a few data sets taken from the Airline on-time performance contest. Note that results are present on the website, but as far as I know none have been posted which use R, so this is the reason for using it here. In passing we might touch on specific libraries and also on different ways of doing things — in fact we start with a few command line instructions under Linux and may use different tools to show some graphical results.
Because “the data consists of flight arrival and departure details for all commercial flights within the USA, from October 1987 to April 2008. This is a large dataset: there are nearly 120 million records in total, and takes up 1.6 gigabytes of space compressed and 12 gigabytes when uncompressed,” I arbitrarily selected two datasets, the 2007 and 2008, to perform similar analyses on these. You may ask why not just one: the 2007, downloaded first, is made of over 7 million observations and 29 variables. 2008, as we shall see, is bigger. To spot differences in the recorded data over the years, I will have a look at older data, the oldest in fact, from 1987. We note that this dataset is smaller, so (as you may guess) data increased over time. We will try to verify whether this is just due to increased traffic or whether more information (features) was recorded, or perhaps both.