Tuesday, June 28, 2005
CRAFTING BETTER ANALYTICAL INTELLIGENCE PRODUCTS
Excellent declassified article from the mid-1990's Studies in Intelligence.
" Making the Analytic Review Process Work" by Martin Peterson.
"The problem with the review process is not the layers of review but rather the quality of the review. In an imperfect business, this is the one thing that intelligence officers need to get right. My 30-plus years of experience leads me to conclude that there should be three levels of review and three broad areas of review for each piece of finished intelligence.
Editing is NOT review. Editing is a mechanical task that should be accomplished by the first-level reviewer or by a staff. Review is about thinking, about questioning evidence and judgments. It focuses on the soundness of the analytic points that are being made and the quality of the supporting evidence. Levels of review is NOT synonymous with layers of review. Layers of review speaks to how many cooks are involved with the broth; levels of review is about ascertaining the quality of the soup.
Each level of review has a different focus. The strength of the review process is directly related to the different perspective that each level brings, with succeeding levels focusing on ever broader issues that are hard for the author and firstline reviewer to see because they are so close to the substance. "
Peterson has a chart that gives an overview to his proposed process that I am unable to copy and fit into PPT, so you'll have to go look for yourself.
My quibble here is with Peterson's bold font, second level, monitoring tradecaraft " Is the piece consistent with previous analysis? ". There's nothing wrong with that question, which is in fact a vital one to ask - the caution stems from the environment in which it is asked.
IC products are not the same thing as academic world, peer-review, though there are a lot of similarities because analysts are analysts. The question asked by Peterson very easily translates in the bureaucratic world to a driver to impose conformity on the new product in light of the position of internal vested interests who have authored the prior assessments. The IC misses new developments - the major paradigmatic ones - primarily because they are new and not part of the established pattern of experience that forms the IC conventional wisdom on a given subject ( in fairness, some ppl in the IC usually catch these new developments but their insights do not always survive the review to make it to intelligence consumers).
To maintain objectivity, at a minumum you would need to ask a few, similar, questions of the old IC products that preceded the new one being reviewed to try and minimize analytical bias. Questioning one's own analytical perspective needs to become as second-nature as questioning the the argument and the accuracy of the data. To use a metaphor, if the IC's model perspective on every subject is a magnifying glass then they ought to also get out a microscope, a telescope and use their naked eye once in a while. ¶ 8:00 AM
Excellent declassified article from the mid-1990's Studies in Intelligence.
" Making the Analytic Review Process Work" by Martin Peterson.
"The problem with the review process is not the layers of review but rather the quality of the review. In an imperfect business, this is the one thing that intelligence officers need to get right. My 30-plus years of experience leads me to conclude that there should be three levels of review and three broad areas of review for each piece of finished intelligence.
Editing is NOT review. Editing is a mechanical task that should be accomplished by the first-level reviewer or by a staff. Review is about thinking, about questioning evidence and judgments. It focuses on the soundness of the analytic points that are being made and the quality of the supporting evidence. Levels of review is NOT synonymous with layers of review. Layers of review speaks to how many cooks are involved with the broth; levels of review is about ascertaining the quality of the soup.
Each level of review has a different focus. The strength of the review process is directly related to the different perspective that each level brings, with succeeding levels focusing on ever broader issues that are hard for the author and firstline reviewer to see because they are so close to the substance. "
Peterson has a chart that gives an overview to his proposed process that I am unable to copy and fit into PPT, so you'll have to go look for yourself.
My quibble here is with Peterson's bold font, second level, monitoring tradecaraft " Is the piece consistent with previous analysis? ". There's nothing wrong with that question, which is in fact a vital one to ask - the caution stems from the environment in which it is asked.
IC products are not the same thing as academic world, peer-review, though there are a lot of similarities because analysts are analysts. The question asked by Peterson very easily translates in the bureaucratic world to a driver to impose conformity on the new product in light of the position of internal vested interests who have authored the prior assessments. The IC misses new developments - the major paradigmatic ones - primarily because they are new and not part of the established pattern of experience that forms the IC conventional wisdom on a given subject ( in fairness, some ppl in the IC usually catch these new developments but their insights do not always survive the review to make it to intelligence consumers).
To maintain objectivity, at a minumum you would need to ask a few, similar, questions of the old IC products that preceded the new one being reviewed to try and minimize analytical bias. Questioning one's own analytical perspective needs to become as second-nature as questioning the the argument and the accuracy of the data. To use a metaphor, if the IC's model perspective on every subject is a magnifying glass then they ought to also get out a microscope, a telescope and use their naked eye once in a while. ¶ 8:00 AM
