Monday, April 25, 2005
IMPORTANT ADDENDUM TO THINKING STRATEGICALLY/SYSTEMICALLY [ UPDATED]
Robin Burk of Random Probabilities and Winds of Change had the following comment in regards to large-system analysis and design that I thought would be as of much interest to my readers as it was to me:
"Speaking as someone who started in software engineering and is now working in decision analysis, I think you need to separate out several isses in large system design.First, there is the issue of causal feedback loops. That's what you're getting at when talking about "planning for success". This is the domain of systems dynamics modeling.
Systems dynamics modeling is highly regarded by many public policy analysts for the very good reason that it captures 2nd and 3rd order effects of decisions.However, many decision analysts will say that the difficulty with systems dynamics modeling is that for it to be predictive of system behavior you often have to be able to estimate various rates of change in key factors -- and that can be very difficult to do. However, it can be used to generate hypothetical best / worst / most likely scenarios, useful for diagnosing whether ones assumptions were on target.
A different approach that is useful when thinking about complex systems is multiple objective decision analysis. This technique recognizes that hard problems are often hard because we have multiple objectives we want to achieve. MODA offers ways to identify those objectives, prioritize them and tie them to specific evaluation measures. Alternative system designs are then simulated or otherwise modeled to predict expected behavior on each evaluation measure and a resulting weighted value score identifies which alternative provides the best expected overall outcome. There are other steps in this process, including (very critically) analysis of the degree to which ones conclusions are sensitive to small changes in priorities.There's lots more that can be said about other disciplines as well.
People who do decision analysis for mission-critical military and other systems are often members of INFORMS and (for military apps) MORS. "
MY COMMENTARY:
That was quite timely. These computer modelling techniques will also be helpful in drawing a clearer line between intelligence analysis and policy decision-making. Currently the border between the two domains tends to get fuzzed over as IC analysts and 1st-4th tier intelligence consumers often tend to rely upon similar mental " tools" for looking at data. In fact, some policy appointees have been, by training, world-class analysts in their own right. There really wasn't a whole lot the CIA was going to give George Schultz or Lawrence Summers, in terms of economic forecasts, that either man could not see on their own, without benefit of using a series of computer models to run scenarios as Robin suggested above.
UPDATE: A Pundita reader who is a decision analyst develops Robin's explanation further. ¶ 9:25 AM
Robin Burk of Random Probabilities and Winds of Change had the following comment in regards to large-system analysis and design that I thought would be as of much interest to my readers as it was to me:
"Speaking as someone who started in software engineering and is now working in decision analysis, I think you need to separate out several isses in large system design.First, there is the issue of causal feedback loops. That's what you're getting at when talking about "planning for success". This is the domain of systems dynamics modeling.
Systems dynamics modeling is highly regarded by many public policy analysts for the very good reason that it captures 2nd and 3rd order effects of decisions.However, many decision analysts will say that the difficulty with systems dynamics modeling is that for it to be predictive of system behavior you often have to be able to estimate various rates of change in key factors -- and that can be very difficult to do. However, it can be used to generate hypothetical best / worst / most likely scenarios, useful for diagnosing whether ones assumptions were on target.
A different approach that is useful when thinking about complex systems is multiple objective decision analysis. This technique recognizes that hard problems are often hard because we have multiple objectives we want to achieve. MODA offers ways to identify those objectives, prioritize them and tie them to specific evaluation measures. Alternative system designs are then simulated or otherwise modeled to predict expected behavior on each evaluation measure and a resulting weighted value score identifies which alternative provides the best expected overall outcome. There are other steps in this process, including (very critically) analysis of the degree to which ones conclusions are sensitive to small changes in priorities.There's lots more that can be said about other disciplines as well.
People who do decision analysis for mission-critical military and other systems are often members of INFORMS and (for military apps) MORS. "
MY COMMENTARY:
That was quite timely. These computer modelling techniques will also be helpful in drawing a clearer line between intelligence analysis and policy decision-making. Currently the border between the two domains tends to get fuzzed over as IC analysts and 1st-4th tier intelligence consumers often tend to rely upon similar mental " tools" for looking at data. In fact, some policy appointees have been, by training, world-class analysts in their own right. There really wasn't a whole lot the CIA was going to give George Schultz or Lawrence Summers, in terms of economic forecasts, that either man could not see on their own, without benefit of using a series of computer models to run scenarios as Robin suggested above.
UPDATE: A Pundita reader who is a decision analyst develops Robin's explanation further. ¶ 9:25 AM
