ngineering
ttiunb158364
Notesforcoursework10SenstivityAnalysis.pptx
Sensitivity Analysis
Outline
Here is what is covered
Sensitivity Analysis
Sampling
Latin Hypercube Sampling
Input Variable Distribution Selection
Analysis
Sensitivity analysis
Input Selection
A simulation will have an output and several input parameters.
It is not always true that we know exactly what the input parameters should be.
Need to conduct ‘sensitivity analysis’ to see the affects of our choices of input variables.
If dealing with a stochastic simulation:
Output will derive from a distribution.
Thus, usually concerned with its average.
Sensitivity Analysis
“Varying the input values of a simulation to determine their affect on the output values”
Actual what you are really concerned with is: does varying the input values change the conclusions drawn from the simulation
For example, in a combat scenario simulation does varying the input parameters change who wins the battle
Why?
Test robustness of simulation results
Determine which inputs really matter
Simplification of model by removing factors that have no effect on output
Helps communicate results to customers
Verification of model as errors can be potentially found using SA
Explore simulation results for emergent behavior
Sensitivity Analysis
Sensitivity Analysis
Sensitivity analysis is not an exact science
Why?
To vary an input parameter you need to make assumptions about what values it can take
What is its maximum and minimum?
What frequency distribution does it take?
Sometimes these can be obvious
See later on discussion on rounding errors
… but most of the time you do not know
You are uncertain about the input value so why would you know its exact frequency distribution?
What do we want from SA?
We hope that results are insensitivity to minor variation in input parameters
Or we could explain any changes
In the old days (pre-1970s), it was assumed that little changes did not matter much
Foundations of Calculus (before limit theory)
…then Chaos Theory arrived!
Sensitivity Analysis
Modern Chaos Theory was discovered by Edward Lorenz in 1961 when “errors” were observed in his weather predication simulation!
Lorenz wanted to repeat results
So entered in input feed from weather simulation back into the computer
Got completely different results!
Assumed made a mistake with either entering input values or a mistake with the printed out or a vacuum tube had gone bad (yes, that old type of computer)
Only “mistake” found was that printed out inputs were round-up from 6 decimal places to 3 decimal places
Small things can and do matter
Chaos Theory
Chaos theory is related to the Butterfly affect
Look this up if you have never heard of it
Watch out for the hurricane butterfly!
Gleick, James. 2008. Chaos: Making a New Science. Penguin: New York.
Chaos Theory
For want of a nail the shoe was lost. For want of a shoe the horse was lost. For want of a horse the rider was lost. For want of a rider the message was lost. For want of a message the battle was lost. For want of a battle the kingdom was lost. And all for the want of a horseshoe nail.
For Want of a Nail
Mandelbrot set
Sensitivity Analysis
If only one input you are uncertain of then vary input value and present range of outputs
i.e. show if having the exact value really matters
e.g. ‘5’ or ‘5.1’
If multiple inputs then could:
Consider all feasible inputs combinations
Not possible is variables are continuous
Then use Latin Hypercube Sampling
Example
Consider the iterative function 4.x.(1-x)
Apply this function three times to 2.2
Ans: -955679
Now apply to 2.21
Ans: -1003752
The differences is 48073!
Now image all the iterative functions working in an ABMS!
Check go add-ins now for later.
Modern Times
Knowing about chaos theory means we are more sophisticated in our modern times
… or are we?
Check go add-ins now for later.
Modern Times
Pretty much all accepted Sensitivity Analysis techniques assume:
A linear relationship between the input variable and output variable
Using an approximation to test the accuracy of an approximation
Compound errors
Independence of input variables
Check go add-ins now for later.
Multiple Input Variables
Most ABMS use multiple inputs
These multiple inputs can interplay with each other:
They might be dependent on each other
Causes problems when determining what cause variant of the output parameter
Third-party correlation
Compound effects of inputs
The inputs might compound the effects on the output, i.e., Y = f(a,b) ~ a.b
All standards SA techniques do not take this into account!
Check go add-ins now for later.
Application Areas
ABMS is not the only field that uses sensitivity analysis
Pretty much any field that uses empirical data to calibrate models uses sensitivity analysis
Mathematical programming also uses sensitivity analysis
Check go add-ins now for later.
Linear Programming
Linear Programming has a mature collection of sensitivity analysis methods:
Range of Optimality: determines the exact range of coefficient variables that the current solution remains optimal
Dual Price: determines the worth of one extra resource in the problem formulation
Simulation usually too complex to find these exact ranges hence explore using multiple runs
Check go add-ins now for later.
Sensitivity Analysis
There are always two steps in sensitivity analysis for simulations:
Sampling
The selection of which values, and combination of values, will used to produce runs from the simulation
Analysis
Looking at the runs results and drawing conclusions
These two steps are not independent!
Check go add-ins now for later.
Sampling
Multiple Input Variables
Focus discussion on multiple input variables
Sampling is concerned:
Choosing the right neighborhood for the input variables
Standard approach: Ask a Subject Matter Expert (SME)
(All kinds of problems with this but will not discuss here)
Selecting the input values for the simulation runs
Check go add-ins now for later.
Example: Fast Food Restaurant
Fast Food Restaurant:
Concerned with the maximum length of queue within in the restaurant over 100 time periods.
Queue includes individual being served.
We do not know:
Do not know the exact service rate of customers.
Estimate probability a server will complete a customers order is about 0.3.
We think that this values lays within [0.1, 0.5].
Do not know the exact arrival rates of customers.
Estimate probability a new customer will arrive is about 0.2.
We think that this values lays within [0.1, 0.3].
We need to conduct sensitivity analysis..
Example: Input Parameters
Ran simulation 30 times and found for each run the ‘Maximum Queue Length’:
Average maximum queue length was ‘3.7’.
Maximum maximum queue length was ‘8’.
But what if we are wrong about the input variables?
Example: Input Parameters
Varied the probability that a customer is served in a time step and got:
But all of these results assumed the same customer arrival probability of 0.2.
The Wrong Way
When dealing with multiple input, the wrong way to do sensitivity analysis is:
One-Factor-At-A-Time (OFAT)
Why?
Because all sensitivity runs would be based on model that contained a input variable that we are not sure about.
So if that second input variable is wrong, then all our sensitivity results would invalid.
Sampling
Therefore, we should consider all possible combinations of input variables.
However:
The number of runs required can get quite large.
So if have 10 variables with 10 possibilities each then: 1010 = 10,000,000,000 = 10 trillion runs.
Even worse if any variable is a continuous one. Why?
Luckily, there is a way around this.
Latin Hypercube Sampling
Latin Hypercube Sampling
Latin Hypercube Sampling is a way to sample the space of all combinations of the input variables s.t.:
As sustainably smaller number of runs is needed.
You get similar results to if you had sampled the whole space!
It was derived especially for simulation and the problems of sensitivity analysis.
Papers
McKay, M.D.; Beckman, R.J.; Conover, W.J. (1979). "A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code“. Technometrics (American Statistical Association) 21 (2): 239–245.
Iman, R.L.; Helton, J.C.; and Campbell, J.E. (1981). "An approach to sensitivity analysis of computer models, Part 1. Introduction, input variable selection and preliminary variable assessment". Journal of Quality Technology 13 (3): 174–183.
Blower, S.M.; Dowlatabadi, H. (1994). “Senstivity and Uncertainty Analysis of Complex Models of Disease Transmission: An HIV Model, as an Example”. International Statistical Review 62 (2): 229-243
Marino, S.; Hogue, I.B.; Ray, C.J.; Kirschner, D.E. (2008); “A Methodology for Performing Global Uncertainty and Sensitivity Analysis in Systems Biology”. Journal of Theoretical Biology 254: 178-196
Latin Hypercube Sampling
Method imposes a grid over the space of all combinations of the input variables. This is used to take samples from
Comes from ‘latin square’, which is generalized to ‘Latin Hypercube’
Hypercube
Stick to Euclid’s Elements:
We are only going to visualize 2-D in examples
Second most published book in history
First is the bible
LHS Mechanism
STEP 1: Select the number of runs you will to carry out, say ‘n’.
STEP 2: For each ‘m’ input variable, choose a probability distribution that represents the range of possible values it could take.
STEP 3: For each input variable:
Divide each probability distribution into ‘n’ equal-probable bins.
Choose a value from each bin.
STEP 4: Randomly combine these values, take one value (without replacement) from each input variable.
STEP 5: Run the simulation ‘n’ times using the combinations from step 4.
LHS Mechanism
STEP 1: Select the number of runs you will to carry out, say ‘n’.
There is no exact rule on this but you will be performing statistical test on the results later, so I would say:
n > m
At least 30, ideally 100
- Unless the simulation runs are too long
Your initial judgment to:
- How complex is the simulation?
More complex implies more runs needed.
- How sensitivity do you think the simulation is to your input parameters?
More sensitivity implies more runs needed.
Example: Fast Food Restaurant
For the fast food restaurant results, I believe that the output is quite sensitive to input parameters. I will, however, choose a sample size of ‘4’ for this example, because:
Each runs has to be repeated at least 30 times because we are dealing with a stochastic simulation.
This is a toy example in a lecture and I am feeling lazy.
LHS Mechanism
STEP 2: For each ‘m’ input variable, choose a probability distribution that represents the range of possible values it could take.
See later in the lecture …
Remember: Yon need to be able to calculate probability of the bins.
Example: Fast Food Restaurant
From the earlier description, I have:
P(customer service completion in a time-step) [0.1, 0.5]
P(customer arrives in a time-step) [0.1, 0.3]
As I do not have any further information about these input parameters, I will assume that:
P(service) ~ U[0.1, 0.5]
P(arrives) ~ U[0.1, 0.3]
LHS Mechanism
STEP 3: For each input variable:
Divide each probability distribution into ‘n’ equal-probable bins.
The uniformed distribution is quite easy to do this. However, other distribution can be hard to determine this.
Given ‘n’ bins for input parameter ‘X’ s.t.
P(X bin “i”) = P(X bin “j”) i,j {1, 2, …, n}
i {1, 2, …, n} P(X bin “i”) = 1
Example: Fast Food Restaurant
The bins for P(service) are:
[0.1, 0.2], [0.2, 0.3], [0.3, 0.4], [0.4, 0.5]
The bins for P(arrival) are:
[0.1, 0.15], [0.15, 0.2], [0.2, 0.25], [0.25, 0.3],
These bins could be anything I like i.e.
[0.1, 0.125] [0.275, 0.3], [0.125, 0.15] [0.25, 0.75],
[0.15, 0.175] [0.225, 0.25], [0.175, 0.225]
LHS Mechanism
STEP 3: For each input variable:
Divide each probability distribution into ‘n’ equal-probable bins.
Choose a value from each bin.
This can done as follows:
Define the a new probability distribution for each bin:
2) Determine the inverse function ‘F-1bin(x)’
3) Use the inverse method to generate a random value
Generate U~U(0,1)
F-1bin(U)
Example: Fast Food Restaurant
For the uniformed distribution bins this is relatively easy to do for a bin [a,b] , just:
Generate U~U[0,1]
Take u.(b-a)+ a
The sample I got:
From the bins of P(service) are:
0.148, 0.216, 0.374, 0.444
From the bins of P(arrival) are:
0.135, 0.191, 0.247, 0.282
LHS Mechanism
STEP 4: Randomly combine these values, take one value (without replacement) from each input variable.
There are several ways that this can be done. In Excel, I put the parameter values next to uniformly random number in a list, then ordered the list using the ‘sort’ function.
There are many ways to do this sorting.
Bin Combinations
Sample Size: 2
Bin Combinations
Sample Size: 3
Example: Fast Food Restaurant
After ordering the values, the parameter pairs are:
P(service) P(arrival)
0.374 0.247
0.444 0.135
0.216 0.282
0.148 0.191
Questions about Method
Why not just take mid-point?
Because might be some bias introduced due to the way we construct our bins
Cycling effects could occur with input variables that just happen to be your midpoints values
Y = Sin(X) is an example of a cycling relationship
Dealing with Complex Systems!
Sampling does not mean the we cover every possibility
LHS Mechanism
STEP 5: Run the simulation ‘n’ using the combinations from step 4.
Now just enter the sample input parameters into the simulation and run it the required time
Example: Fast Food Restaurant
Choosing input variable Distributions
One of the problems of SA is that you are uncertain about the input variable so why would you be certain about the input variables distribution?
Potentially jumping out of the frying pan into fire!
There are some rules-of-thumb you can follow for choosing the input variables distribution.
Input Variable Distributions
There are varies qualities that you might already know about the input variables distribution:
I know the distribution of the input variable
Then use that distribution!
I know the shape of the distribution
Then approximate to a standard distribution or create an empirical distribution of the shape
Choosing Input Distribution
I am only worried about rounding errors
Determine the maximum, M, and minimum, m, that the input variable could be
Use X~U(m, M)
I only know the Maximum and minimum
Then use a Uniform Distribution as above
Choosing Input Distribution
I only know the Maximum, minimum and average
Then use a Triangular Distribution for these values
I only know average and variance
The use the Normal Distribution for these values
Choosing Input Distribution
Use your common sense:
What should the distribution look like?
Remember descriptions of distributions also included what they could represent in the real world
E.g., Bernoulli distribution can represent a coin toss
Does the variable have to be positive?
Then the minimum is at least zero.
Choosing Input Distribution
(Sensitivity) Analysis
Now you completed the runs, what next?
You need to analysis the results to determine if there are any abnormalities with your results
There are various standard ways to do this
Analysis
One-Factor-At-A-Time (OFAT)
Look at the range of outputs compared to the input
What is acceptable variation? SME
Scatter graphs
Local methods
Take partial differentiates of output variable to input variable
Effectively OFAT
Analysis Approaches
Correlation between output variable and input variable
Linear regression
Creating a meta-model of the simulation
Variance-based methods
Look at variance of conditional expectation
Analysis Approaches
All above methods assume independence between input variables. Some don’t:
Emulators
Create model of relationship between input and output variables
Making a metamodel!
(Sounds like work and lots of assumptions to me)
Partial Correlation Coefficient (PCC)
PCC not deal with Compound effects of inputs
PCC assume linearity effects
Independence
Partial Correlation Coefficient (PCC)
(not evaluated)
Partial Correlation Coefficient
Partial Correlation Coefficient is a correlation coefficient that tries to remove the effects on correlation between input variables when working out the correlation between the input variable and output variable.
Matrix
A matrix: A = {aij}
AMnxm(R)
‘n’ rows
‘m’ column
‘aij ‘ is an element of ‘A’
It represent the value in the;
i-th row
j-th column
P(Customer Served)
0.1
0.2
0.3
0.4
0.5
Average Maximum Queue Length
10.77
5.47
3.7
6.06
2.56
Maximum Maximum Queue length
75
15
8
6
6
(
)
(
)
(
)
(
)
0/
bin
PXx
xbin
FxPXxxbinPxbin
ow
ì£
Î
ï
=£Î=Î
í
ï
î
P(service)
P(arrival)
Average Maximum
Queue Length
Maximum
Maximum
Queue
0.374
0.247
4.033
8
0.444
0.135
1.867
4
0.216
0.282
10.500
18
0.148
0.191
7.470
19
1000
0100
0010
0001
æö
ç÷
ç÷
ç÷
ç÷
èø
