The Prediction of Flight Delays within the Data Science pipeline: introduction

Calm down: I am not saying I’ll predict the delay of the plane you just boarded (although someone might do…). I am just showing an example of a prediction problem which just happens to deal with plane delays (and past ones).

The aim is to present implementation examples of the Data Science work-flow. The web is full of examples of machine learning algorithms and code chunks on various aspects of exploration and prediction. However, there are not that many examples following the whole pipeline, from inception down to publication into a notebook (the valuable Titanic case aside).

Assume that a problem has been posed — call this “step 0” — a long and difficult task involving a clear-cut definition from the elicitation phase, the interaction with customers/stakeholders who usually don’t know the technical details (and that’s OK, it’s not their job). We get back ‘home’ after a few rounds and are faced with “the question(s).” Now what?

I split the work flow into several steps, grouped into phases, with code chunks dealing with the entire pipeline:

1. Data gathering
2. Data Exploration (get acquainted)
3. Tidy up
4. Questions you may want to ask (and hopefully answer)
5. A bit of engineering
6. Model the data (train)
7. Evaluate your prediction
8. The real prediction
9. Tell somebody what you did (and how)
10. Clean it up!

Steps 1–3 are (at least partially) absent from competitions like Kaggle’s, and often involve an iterative procedure, especially point 2, as you may discover new aspects even later down this list.

Step 4 is what your boss is asking, or what you think should be asked, relevant to the business based on the knowledge you acquired. This should be checked carefully against Step 0 — “what does the customer actually want?” Any mismatch should stop us from proceeding further; we’re likely to loop back to point 1 (or even 0).

Steps 5 and 6 might be swapped, as engineering depends on the chosen model. In step 6 some features may need to change format (class), be dropped, or split.

Steps 7 and 8 are seldom available in competitions, since some external algorithm establishes your accuracy and places you on the leaderboard, out of your control. In real cases you may iterate 6 and 7 before getting to 8, whose performance is assessed by domain knowledge.

Step 9 is often overlooked because you’re so deep into technical aspects that you forget another human being needs to understand, evaluate (and maybe sell) your outcomes. It should be split into two parts: one for the stakeholder (the ‘what’) and one for yourself (the ‘what + how’), as internal reference.

Step 10… so you thought you were finished? In a fast-paced environment you might be, and there may be little time to polish things to a satisfying point (I never am). But if you’re able to polish by generalizing, standardizing (e.g. transforming code chunks into functions), and documenting what you’ve done, you’ll be ready for re-use next time. This is also a good occasion to measure time spent on each activity — not to measure your own performance, but to improve future estimates.

I am sure each of these parts can be done better, more efficiently. But they represent the effort to standardize a work flow, which is why I propose step 10. Time permitting, I will transform the whole lot into a package.

So then, the first will be Air Flights 2007 and 2008 – Part I, which contains all the mentioned steps. However, it is a working document, from where the analysis of the individual steps continues in the posts that follow.