The Unquantified Risk

In decision science pipelines, we typically first estimate data, such as demand, travel times, price sensitivity etc. Then, we optimize our plans taking these estimates as inputs for our optimization models. In prior editions of this newsletter, we argued at length why this estimate should be a distributional forecast rather than a point forecast to account for the risk of estimating wrong.

The question arises whether we fully account for the risk associated with our plan in this way. Or are we forgetting something?

Consider the imaginary road map above. While this one was made up, only a few days ago I found myself in a similar situation using the Google Maps routing app. Thankfully, I had time.

Imagine you are sitting in the car in the lower left corner (marked 1), and you need to go to the address on the middle right (marked 2). All streets are drivable in both directions, except the three streets with arrows which mark one-way streets.

We have two reasonable routes that can get us to our destination. The first-go-straight green route is a bit longer, but avoids left turns. The first-go-right purple route is a bit shorter but requires two left turns. The short black dashes on each route mark ten-second intervals required for travel. As we can see, the left turns take ten extra seconds, while right turns do not cause any delays. Therefore, taking the green route, we would need 80 seconds, while the purple route requires 90 seconds.

If we route by shortest travel time, we would take the green route. Assuming you have the goal to get to your destination as quickly as possible, which route would you take?

Well, it depends.

If you know the area really well, e.g., if you live at the destination address and you need to pick up your pregnant sister from your home to bring her to the hospital, you would probably take the green route. But imagine your sister is visiting someone else and you don't know the area at all. Arguably, you are much better off taking the purple route, as it is virtually impossible to take a wrong turn while, on the green route, it will be very hard to choose the correct streets as you navigate through the suburban maze. And if you do make a mistake, it will take a long time to correct your error.

Execution Risk

The example shows that there is another risk that we must consider when optimizing plans. We may have a good handle on the travel times, but how well can we quantify the risk that the plan we prescribe will not be carried out minutely?

This execution risk is real and one reason why optimized plans may be met with resistance or even outright rejection by those who need to execute those optimized plans. After all, when something goes wrong, the blame will fall on those who made a "human error" no matter how complex the plan was.

Counterfactual Analysis

We can and should do better. In this paper, we demonstrate how we can utilize counter-factual analysis to assess the performance of 'what-if' plans for past scenarios. Using this estimated performance data, we can learn when one of two candidate plans can be expected to perform more robustly. Then, rather than assessing the quantified performance of a plan, we generate a number of plans and rank them using pairwise comparisons.

In this way, we enable the machine to learn patterns in plans that are associated with robust performance. Just as humans do, who assess potential courses of action not just based on costs but also according to their resilience to disruptions. We consider time buffers and slack in resources, for example, and we assess if there exist cheap and effective remedies to common types of disruptions, including mistakes we might make when executing the plan.

Moreover note that, by using counterfactual analysis, the machine not only learns from prior scenarios what kind of disruptions we are likely to face, but also how well we are able to recover from them. And that not just in theory, but in actual practice. It is not necessary to assume that we will find the optimal response to a disruption, for example. Instead, the counter-factual analysis can assess performance of a plan based on the actual protocol that is used to recover by the operators.

Equipped with such a learnt model, we are then able to choose, from a set of potential plans, the one that is expected to execute the best.

Generating Candidates

The problem is therefore reduced to creating a candidate set of high-quality plans. In the publication above, we simply generated these candidates from the deterministic point-estimated optimization problem. However, for more complex problems, we cannot hope that a random near-optimal solution to the point-estimated model will perform robustly under model and execution uncertainty.

Therefore, we suggest to use InsideOpt Seeker to generate the candidate set. Seeker is the only commercial optimization solver that offers a direct API for stochastic optimization.

How To Handle Execution and Other Unquantified Risks

So the next time you are facing an aggressive dynamic pricing application, a rush-hour routing problem, or a sophisticated inventory optimization problem, follow these steps:

Offline:

• Build distributional forecasting models for the optimization model inputs and base a stochastic Seeker model on these forecasts.
• Use Seeker to generate high-quality solutions for past problem instances, using the distributional estimates that were known before the execution, and the real values of the inputs that then materialized for counter-factual evaluation of how well these plans would have fared had they been used.
• Use this labeled data to train a const-sensitive classification model that can pick the preferred plan out of a given set of potential plans.

In Deployment:

• Use InsideOpt Seeker and generate a set of high-quality solutions that perform well under the quantified risk based on the model created under 'Offline (1)' above.
• Use the model learnt under 'Offline (3)' above to choose the plan that exhibits features that are associated with robust practical performance.

Do you want reliable efficiency? It is our job to provide it. InsideOpt.