chapter 9 accuracy assessment in remotely sensed categorical information...
TRANSCRIPT
Chapter 9
Accuracy assessment in remotely sensed categorical information
遥感类别信息精度评估
Jingxiong ZHANG
张景雄
Chapter 9
Accuracy assessment in remotely sensed categorical information
遥感类别信息精度评估
Jingxiong ZHANG
张景雄
• Thematic mapping based on mage classification
1) use spectral (radiometric) differences to distinguish spatial classes
2) supervised / unsupervised mode• Change detection
1) binary maps of change or no-change
2) categorized change
A quick review of Chapters 7 & 8
(a) (b)
(c) (d)
Figure 1. Simulation: studies: (a) and (b) mean areal-class maps, for time 1 and time 2; (c) and (d) distorted
versions for (a) and (b), respectively.
We will discuss
• Why ?• What ?• How ?
for accuracy assessment
• Key points:
to construct confusion matrix (混淆矩阵 )
to compute accuracy measures
• Difficult points:
Statistics for sampling design and
spatial analysis
Why do we bother accuracy assessment?
• Thematic maps of land cover, forest types, and others can be derived from classification of remotely sensed imagery in combination with ancillary data sets
• You need to tell map users how well it actually represents what’s out there
• “Without an accuracy assessment, a classified map is just a pretty picture.”
What is accuracy assessment?
• Assess how well a classifier works
• Interpret the usefulness of someone else’s classification
How do we do accuracy assessment?
• Collect reference data, i.e., “ground truth”
determining class types at specific locations• Compare a map with the reference to
compute accuracy measures• Interpretation of the results
Reference Data - possible sources
• Aerial photo interpretation• Ground truthing with GPS• GIS layers
• Make sure we can actually extract from the reference source the information needed
• For discriminating four species of grass, we may need ground surveys not aerial photographs
Determining size of reference plots
• Match spatial scale of reference plots and remotely-sensed data
• Ground plots (5 meters on a side) may not be useful for remotely-sensed imagery (1km)
may need aerial photos or even other satellite imagery.
• Take into account spatial frequencies of image
consider photo reference plots that cover an area 3 pixels on a side
Example 1: Low spatial frequency Homogeneous
Example 2: High spatial frequency Heterogeneous
• Implication for positional accuracy
consider the situation where accuracy of position of the image is +/- one pixel
Example 1: Low spatial frequency
Example 2: High spatial frequency
Determining number and position of samples
• Make sure to adequately sample the landscape• Variety of sampling schemes:
Random, stratified random, systematic, etc. • The more reference plots, the better
You can estimate how many you need statistically
In reality, you can never get enough
Lillesand and Kiefer: suggest 50 per class as rule of thumb
}{ dPpprob
n
PP
N
nNttd p
)1(
1
2
2
0
)1(
d
pptn
Sample size (please see the references)
n << N p = 0.5
Sampling Methods
:
Stratified Random Sampling
a minimum number of observations are randomly placed in each stratum.
observations are randomly placed.
Simple Random Sampling
Sampling Methods
Systematic Sampling:Observations are placed at equal intervals according to a strategy
Systematic Non-Aligned Sampling a grid provides even distribution ofrandomly placed observations
Sampling Methods
Cluster Sampling
Randomly placed “centroids”used as a base of several nearby Observations, which can be Selected randomly or systematically
Accuracy assessment
• Collect reference data, i.e., “ground truth”
determining class types at specific locations
• Compare a map with the reference to compute accuracy measures
• Interpretation of the results
• Compare (through sample locations):
class type on classified map =
class type determined from reference ?
• Summarize (cross-tabulation) into an confusion matrix
• Compute accuracy measures:
overall classification accuracy (percent correctly classified pixels, PCC)
producer’s / user’s accuracy
kappa coefficient of agreement (KHAT)
An example confusion matrix (混淆矩阵 )
Class types determined from referencereference
User’s AccuracyClass types
determined from
classified map
# Plots Conifer deciduous grass Totals
Conifer 50 5 2 57 88%deciduous 14 13 0 27 48%grass 3 5 8 16 50%Totals 67 23 10 100
Producer’s Accuracy 75% 57% 80% Total (PCC): 71%
Kappa coefficient of agreement
• Kappa of 0.463 means there is 46.3% better agreement than by chance alone
(0.71 - 0.46) / (1 - 0.46) = 0.463• Chance agreement =
[Product of row and column marginals for each class]
0.46 for the example
agreement chance - 1
agreement chance -accuracy observedˆ K
Accuracy assessment
• Collect reference data, i.e., “ground truth”
determining class types at specific locations• Compare reference to classified map
class type on classified map = class type determined from reference ?
• Interpretation of the results
Interpreting results of accuracy assessment
• Misclassification in remotely-sensed data:Classes are land use, not land coverClasses not spectrally separableSpatial scale of remote sensing instrument does
not match classification scheme• Error in reference data:
Interpreter errorSubjectivity
Improving Classification• Land use/land cover: incorporate other data
Elevation, temperature, ownership, etc.Context
• Spectral inseparabilityHyperspectralMultiple dates
• ScaleDifferent sensorAggregate pixels
• ClassifiersUse HIERARCHICAL CLASSIFICATION schemeIn Maximum Likelihood classification, use Prior
Probabilities to weigh minority classes more
Summary
• Accuracy assessment to add value to remote sensing information products and to ensure their proper use
• Ground truth is itself difficult to acquire due to the non-trivial task of class definition
• Sampling design is important for cost-effectiveness in accuracy assessment
• Accuracy assessment as an integral component in the information process
References• Cochran, W.G. 1977. Sampling techniques. Wiley, New
York.
• Congalton, R. 1991. A review of assessing the accuracy of classifications of remotely sensed data. Remote Sensing of Environment 37:35-46.
• Nusser, S.M., and E.E. Klaas. 2002. Final performance report to EPA Region 7, Part II: GAP accuracy assessment pilot study. Environmental Protection Agency Contract X997387-01 Final Report. Iowa Cooperative Fish and Wildlife Research Unit, Iowa State University, Ames, Iowa. 77 pp.
• Stehman, S.V. 1997. Selecting and interpreting measures of thematic classification accuracy. Remote Sensing of Environment 62:77-89.
Questions:
1. The challenges of accuracy assessment in change detection include obtaining reference for:
images taken in the pastsampling sufficiently the areas that will change in the future
Why is this?
2. Compute accuracy measures from the hypothetical example error matrix.
