Download - Pontius y Malason (2005).pdf
-
7/30/2019 Pontius y Malason (2005).pdf
1/24
This article was downloaded by:[Universidad De Chile 2205]
On: 14 July 2008
Access Details: [subscription number 791781823]
Publisher: Taylor & Francis
Informa Ltd Registered in England and Wales Registered Number: 1072954
Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
International Journal of GeographicalInformation SciencePublication details, including instructions for authors and subscription information:
http://www.informaworld.com/smpp/title~content=t713599799
Comparison of the structure and accuracy of two land
change modelsGil R. Pontius a; Jeffrey Malanson a
a Clark University, Department of International Development, Community and
Environment, Graduate School of Geography, Worcester MA 01610-1477, USA
Online Publication Date: 01 February 2005
To cite this Article: Pontius, Gil R. and Malanson, Jeffrey (2005) 'Comparison of the
structure and accuracy of two land change models', International Journal ofGeographical Information Science, 19:2, 243 265
To link to this article: DOI: 10.1080/13658810410001713434
URL: http://dx.doi.org/10.1080/13658810410001713434
PLEASE SCROLL DOWN FOR ARTICLE
Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf
This article maybe used for research, teaching and private study purposes. Any substantial or systematic reproduction,
re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly
forbidden.
The publisher does not give any warranty express or implied or make any representation that the contents will be
complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be
independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings,
demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or
arising out of the use of this material.
http://www.informaworld.com/smpp/title~content=t713599799http://dx.doi.org/10.1080/13658810410001713434http://www.informaworld.com/terms-and-conditions-of-access.pdfhttp://www.informaworld.com/terms-and-conditions-of-access.pdfhttp://dx.doi.org/10.1080/13658810410001713434http://www.informaworld.com/smpp/title~content=t713599799 -
7/30/2019 Pontius y Malason (2005).pdf
2/24
-
7/30/2019 Pontius y Malason (2005).pdf
3/24
therefore the scientist would want to select a model that has predictive power.
Hence, it is important to have a method to compare the accuracy of two models and
of several runs for any particular model to see how the models structure relates to
its accuracy.
It can be a challenge to compare various runs of a single model, because a model
can have a large number of options and implied assumptions that interact.Consequently, when one run is different than another run, it is not necessarily
immediately clear which of the possible interactions among which assumptions is
responsible for the difference. Comparison of runs from different models can be
even more challenging, because it can be difficult to tell whether the difference in the
runs is attributable to the models detailed assumptions or to the models more basic
assumptions. LUCC scientists are likely to learn most efficiently when we examine
the influence of the most basic assumptions before the highly-complex assumptions.
To this end, this paper compares several runs of two models that are different in
fundamental ways.
Overview of two models
The first model, Cellular Automata Markov (CA_Markov), allows any number of
categories and can simulate the transition from any category to any other category.
The second model, Geomod, uses exactly two categories and can simulate only the
transition from the first category to the second category. That is, Geomod can not
simulate an additional simultaneous transition of the second category to the first. In
this respect, CA_Markov is more complex than Geomod. For some applications,
the added complexity of CA_Markov may be desirable, for other applications the
opposite may be true. Therefore, it is important to compare models head-to-head. In
the process of comparing these two models, we illustrate a new generalized methodto measure the predictive power of land change models.
Markov-type models constitute some of the historically most common methods of
predicting change among various categorical states. At their core is the mathematics
of Markov chains, derived by the Russian Scientist Andrei A. Markov (1907). The
first Markov models were not spatially explicit, but since the middle of last century,
the Markov model has been used to simulate changes in maps (Baltzer et al. 1998).
Spatio-temporal Markov chain (STMC) models can be used to model changes over
time among categories in a landscape, whereby each pixel on the landscape is
classified as exactly one category, and each pixel has some probability of
transitioning to some other category at every time step. The Cellular Automata(CA) component of the CA_Markov model allows the transition probabilities of
one pixel to be a function of neighbouring pixels. Baltzer (2000) gives many
examples of how Markov-type models are used.
Geomod is a newer land use change model that was originally designed to
simulate the loss of tropical forests and to estimate the resulting carbon dioxide
emissions (Pontius 1994). Pontius et al. (2001) offers the single most complete
description of Geomod, which used to be known as Geomod2. Geomod predicts a
one-way conversion from one category to one other category, similar to other
popular land change models such as SLEUTH (Clarke et al. 1996, Clarke 1998). The
Geomod approach has been used to simulate deforestation in Massachusetts(Schneider and Pontius 2001), Costa Rica (Hall 2000, Pontius 2002), India (Pontius
and Batchu 2003, Pontius and Pacheco in press) and the tropics globally (Hall et al.
1995a, Hall et al. 1995b). It is rapidly gaining popularity in designing baseline
244 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
4/24
scenarios of tropical deforestation in the context of carbon offset projects called for
by the Clean Development Mechanism of the Kyoto Protocol (Brown et al. 2002b).
It is now being used in private businesses, non-governmental organizations, and
research institutes (Brown et al. 2002c).
The methods section of this paper describes the qualitative and mathematical
details of each model, and then applies each model to a landscape in central
Massachusetts, USA. Several runs of each model are compared, so naturally, there
must be a criterion for comparison. The next two subsections of this introduction
motivate the selection of a criterion, which the methods section then explains in
technical detail.
Modelling strategy of calibration and validation
In land-use change modelling, there is no agreed upon criterion to assess the
performance of one model versus another model, or to compare one run versus
another run of the same model. This lack of a uniform method of assessment isrelated to the fact that there is not agreement among scientists concerning the
purpose of land change modelling. Some scientists prefer models that express the
theory of the mechanisms of the processes of land change, while others place more
weight on a models ability to extrapolate the observed pattern of change based on
past empirical patterns. It either case, most scientists would like for models to be
able to simulate true patterns. Therefore, there is a need to quantify a models
predictive power.
In order to quantify the predictive power, scientists should examine the goodness-
of-fit of the validation, which should not be confused with goodness-of-fit of
calibration. Calibration is the process whereby the scientist uses information aboutthe landscape to help select the parameters of the model. The information used for
calibration should be at or before some specific point in time (t1), which is the point
in time at which the predictive extrapolation begins. For example, CA_Markov
models are typically calibrated with reference maps from two points in time, t0 and
t1; whereas Geomod requires maps from only one point in time, t1. In both cases,
the model then extrapolates land change beyond time t1 to some subsequent time t2.
Validation is the process of comparing the models prediction for time t2 to a
reference map of time t2, where the reference map is considered a much more
accurate portrayal of the landscape at time t2. In order for the extrapolation to be a
legitimate prediction, the information used in the calibration must have existed at orbefore t1.
Pontius and Pacheco (in press) stress that a good fit of calibration does not
necessarily imply a good fit of validation, and that the later is the appropriate
indicator of a models predictive power. Mertens and Lambin (2000) and Brown et
al. (2002a) are two of the few examples that we have been able to find that clearly
distinguish calibration from validation. Rykiel (1996) and Oreskes et al. (1994) give
a more detailed and philosophical discussion of validation.
Criteria for accuracy assessment
Separation of the calibration process from the validation process is necessary, but
not sufficient, in order to compute the goodness-of-fit of validation. The scientist
must also select a criterion to compare the predicted map of time t2 to the reference
Comparison of two land change models 245
-
7/30/2019 Pontius y Malason (2005).pdf
5/24
map of time t2. Perhaps the most popular criterion is the percent of pixels classified
correctly.
The percent correct is popular because of its ease of computation and apparent
ease of interpretation; however a high level of sophistication should be used to
interpret the percent correct. Most importantly, the interpreter must realize that a
large percent correct does not necessarily imply that the model has good predictive
power, due to a variety of reasons. Most importantly, it is common to attain a large
percent correct from a null model that predicts pure persistence (i.e. no change)
between time t1 and time t2, due to temporal autocorrelation between the reference
maps of t1 and t2. In most of the land change modelling literature that we examined,
the typical amount of change on the landscape was about 10%, so a null model of
pure persistence would be 90% correct, which is a measure of accuracy that most
nave interpreters would consider high. Hagen (2002, 2003) is the only other author
that we have found that articulates the need to compare a predictive model to a null
model of pure persistence.
The criterion of percent correct has additional complications. The percent correctis usually based on a pixel-by-pixel analysis, so it is possible that percent correct will
not correspond to a visual assessment of the maps, because pixel-by-pixel analysis
fails to consider spatial patterns of the pixels. For example, if a pixel is classified as
the wrong category, then the entire pixel is in error, regardless whether the correct
category is found in the neighbouring pixel or nowhere near the pixel (Pontius 2002).
Lastly, a large portion of the percent correct can be attributable to random chance
(Pontius 2000).
Consequently, this paper uses the null resolution as the criterion for the
goodness-of-fit of validation. The null resolution is the resolution at which the
accuracy of the predictive model is the same as the accuracy of a null model, whereaccuracy is measured in terms of percent correct of the entire landscape. The finer
the null resolution the more accurate and precise is the models prediction. The
methods section describes the multiple-resolution procedures necessary to compute
the models predictive power in terms of the null resolution.
Methods
Data
The data for this paper derives from the State of Massachusetts Executive Office of
Environmental Affairs, which makes GIS files available through the agency calledMassGIS. Vector files of historic land use are available for three times: t051971,
t151985, and t251999. Each map shows 20 categories. For each year, these
categories are reclassified into two categories, built or non-built, based on the
Anderson level 1 classification system (Anderson et al. 1976). For this study area,
built consists mostly of Residential, and includes Commercial, Industrial,
Transportation, Recreation and Waste Disposal. All of the maps are converted to
raster format on a 30-by-30 meter grid of 1116 rows and 1008 columns, which
contains 651,951 pixels in the study area.
The study site consists of the town of Worcester and the nine adjacent towns in
central Massachusetts. Worcester is a common example of a post-industrialAmerican city that has undergone suburbanization over the last three decades.
Recently, Worcester has become one of the nations fastest growing housing
markets, as many people prefer to live in central Massachusetts as opposed to the
246 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
6/24
nearest major population centre, Boston, where the housing prices are approxi-
mately double what they are in the Worcester area. As a result, real estate developers
are converting much of the land in the Worcester region to residential uses, which is
causing tremendous concern for the existing residents of central Massachusetts.
Figure 1 shows that from 1971 to 1985, approximately 3.7% of the landscape
converted from non-built to built and a negligible amount converted from built tonon-built. Figure 1 is the signal used to calibrate the model. Figure 2 is the true
pattern used eventually to assess the accuracy of the models prediction through the
validation process. Figure 2 shows that between 1985 and 1999, approximately 5.4%
of the landscape converted from non-built to built and 0.8% converted from built to
non-built, resulting in a net increase in quantity of built on the landscape of 4.6%.
The legend of figure 3 shows the detailed categories of land use of 1971. Historic
land use is important as a predictive factor of future change because historic land
use is related to the ease with which land can be converted to built uses. For
example, old agricultural land has a higher propensity to transition to built than
does forest. This is probably related to the fact that agricultural land is alreadycleared and has a closer proximity to infrastructure such as roads and water services.
Figure 4 shows the categories of legal protection that restrict the conversion of
land to new built uses. In Massachusetts, legal restrictions are clearly important in
determining which parcels of land convert to the built category. Unfortunately the
map of legal restrictions does not specify the years in which the laws were enacted;
therefore it is unclear whether the map of legal restrictions is legitimate for
calibration, since some of the laws portrayed in the map were probably created
Figure 1. Change in built category from 1971 to 1985, which is the signal used forcalibration.
Comparison of two land change models 247
-
7/30/2019 Pontius y Malason (2005).pdf
7/24
Figure 2. Change in built category from 1985 to 1999, which is the pattern used forvalidation.
Figure 3. Land use of 1971, which is a driver used to calibrate the suitability maps.
248 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
8/24
subsequent to 1985. This paper uses the map of legal constraints in half of the model
runs in order to demonstrate both the impacts that such constraints can have on
land change predictions, and also to show the maximum level of accuracy attainable
if we were to use this potentially illegitimate data for model calibration.
Structural comparison
CA_Markov and Geomod have some major qualitative differences in basic
structure, and other minor differences in more subtle aspects. Table 1 states thedifferences between the models in general order of most fundamental to most
detailed. The first two characteristics in table 1 show that CA_Markov can simulate
two-way transitions among any number of categories, whereas Geomod simulates
either a one-way gain or a one-way loss from exactly one category to one alternative
category. In this sense, CA_Markov can theoretically model a wider variety of
simultaneous processes. There are a suite of challenges and complications associated
with simulating transitions among several categories. For example, many different
categories might simultaneously compete to claim any particular pixel. To deal with
this situation, CA_Markov implements a multiple objective land allocation
algorithm. Geomod avoids these complications by assuming only one type oftransition from one category to one other category. For many applications, the
process of interest can be expressed in terms of a transition from a non-disturbed
category to a disturbed category, in which case Geomods approach could suffice.
Figure 4. Contemporary legal constraints to development, which is an optional additionaldriver used to calibrate the suitability maps.
Comparison of two land change models 249
-
7/30/2019 Pontius y Malason (2005).pdf
9/24
The remaining characteristics of table 1 are grouped in terms of the two basic
tasks that a model must perform. Specifically, the model must predict the quantity
of each category and the model must predict the location of each category. BothCA_Markov and Geomod specify the quantity of the predicted categories
independently from the location of the predicted categories. In table 1,
characteristics 34 refer to the rules by which each model predicts the quantity of
each category. Characteristics 57 refer to the rules by which each model predicts
the location of each category. The sections below give the details of how each
characteristic manifests in the example.
The characteristics in table 1 are related closely to figure 5, which shows the
specific options that this paper uses to model the land change for the example.
The first option of figure 5 concerns which model to use: CA_Markov or Geomod.
The next option concerns the method to extrapolate the quantity of each land
category. The Markov approach extrapolates both the gain and loss of each
category based on the proportion estimated from the calibration information.
Geomod recommends linear extrapolation to predict the net quantity of the built
category. The remaining options concern the decision rules that each model can use
to determine the location of the predicted land change. One important decision
concerns the factors to include in the creation of a suitability map. Another option
concerns whether or not to implement a contiguity rule that encourages pixels of the
same category to be adjacent. Yet another option concerns the time step, which is
the shortest interval of time over which the model predicts change. This final option
is relevant only if the contiguity rule is selected.
The options shown in figure 5 give a range of the most important options
available, but they are not exhaustive. For Geomod, an additional option allows for
stratification of the analysis. For CA_Markov, an additional option allows for
consideration of the accuracy of the reference data. The sections below discuss the
details of each option. In total, this paper examines the 24 different model runs
shown in figure 5 to extrapolate the change between 1985 and 1999, based on
calibration from 1971 to 1985.
CA_Markov
Predicting Quantity of Change. CA_Markov predicts the quantity of each categoryat time t2 by extrapolating both gain and loss of each category from time t1. This
extrapolation is calibrated by computing the proportional gain and loss between the
years of calibration, which are t051971 and t151985 in our example. The
Table 1. Qualitative characteristics of CA_Markov and Geomod.
Characteristic CA_Markov Geomod
1) Number of Categories Two or more Two2) Transitions Gains and Losses Gain or Loss
3) Information for Quantities Computed from 2 maps Must be supplied4) Quantity Propagation Multiplicative Additive5) Suitability Map Must be supplied for
each transitionCan be created from drivermaps and a land-use mapof 1 point in time
6) Proximity Method Filter Constraint7) Stratification Not part of the design Part of the design
250 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
10/24
extrapolated quantity for each category is a function of the transition matrix asdictated by the matrix algebra of Markov chains. Specifically, the Markov
calculation assumes that for every 14-year time interval, the proportion of land
that transitions from one category to another category is the same as the proportion
of land that made that particular transition during the 14-year calibration period.
Figure 6 shows this extrapolation where both the calibration interval and the
extrapolation interval are 14 years. During the calibration interval, about 3.7% of
the landscape shows gain in the built category and negligible loss of built. Therefore,
over the subsequent 14 years, CA_Markov predicts gain of built and negligible loss
of built. The open triangles of figure 6 show the true gross gain in built, whereas the
closed triangles show the true net gain in built after the loss of built is considered.The open diamond shows the extrapolated gain of built, and the closed diamond
below the open diamond indicates the net gain of built after the loss is taken into
consideration. The extrapolation of quantity is computed independently of the
decision of where to locate the change among categories on the landscape.
Predicting Location of Change. There are two principles that determine the location
where CA_Markov predicts land change. The first is based on the concept of a
suitability map and the second is based on the concept of a contiguity rule.
CA_Markov uses a suitability map for each transition that it extrapolates. At
every time step, CA_Markov determines the number of pixels that must undergo
each transition, then selects the pixels according the largest suitability for theparticular transition. The user can generate the suitability maps in a variety of ways,
for example by using a deductive approach such as Multi-Criteria Evaluation, or an
inductive approach such as logistic regression. In our example, CA_Markov predicts
Figure 5. Combinations of options for model runs.
Comparison of two land change models 251
-
7/30/2019 Pontius y Malason (2005).pdf
11/24
-
7/30/2019 Pontius y Malason (2005).pdf
12/24
the transition from non-built to built. The subsection on Geomod describes the
detail of that suitability map (figure 7), including the option to use the map of legal
constraints in the creation of the suitability map. The suitability maps remain static
for the duration of the simulation.
A contiguity rule is the second principle that CA_Markov uses to determine the
location of predicted change. The contiguity rule usually has the effect of predictingthe growth of a category near locations where that category already exists. At every
time step, the suitability value in each candidate pixel is temporarily recalculated to
show spatial dependence, such that the suitability for the transition to a particular
category is influenced by whether that category exists in nearby pixels. If few of the
nearby pixels belong to the category, then the suitability value is down-weighted. If
many of the nearby pixels belong to the category, then the suitability value is
maintained. The definition of nearby is determined by a spatial filter that the user
specifies. Our example uses a filter which causes the suitability at a particular pixel
to be weighted as a function of its nearest 24 neighbouring pixels, such that closer
neighbours exert stronger influence on the calculation. This temporary recalculationis performed at each time step; therefore the duration of the time step can have some
influence on the location of predicted land change when the contiguity rule is used.
When the time step is small, the model must apply the time step for many iterations
in order to achieve the desired duration of the extrapolation. Therefore, smaller time
steps lead to more frequent applications of the filter, hence more frequent updates of
the spatial dependence. If the user does not want to simulate spatial dependence
Figure 7. Suitability for conversion from non-built to built, based on slope and land use of1971.
Comparison of two land change models 253
-
7/30/2019 Pontius y Malason (2005).pdf
13/24
explicitly, then the most appropriate filter has a 1 in the centre surrounded by 0s.
CA_Markov always assumes some spatial dependency, but this filter exerts the least
possible spatial dependency, so when it is used, the selection of time step is negligible.
Geomod
Predicting Quantity of Change. Geomod has no explicit method to extrapolate the
quantity of change from its one category to the alternative category, because the
purpose of Geomod is to predict the location of a one-way transition. Nevertheless,
Geomod must specify some quantity of change in order to make any prediction;
therefore Geomod allows the user to predict any quantity, based on any
extrapolation method that the user desires. We recommend a simple approach,
unless it is clear that a more complicated approach is justified. Therefore, the
example in this paper uses linear extrapolation of the net quantity of built, based on
the line that is interpolated through the quantities at 1971 and 1985. Figure 6 shows
that Geomods linear extrapolation of the quantity is nearly identical toCA_Markovs extrapolation of the gain and loss. The reason for the similarity of
the extrapolation is that the percent of change that took place on the landscape
during the calibration interval is small.
Half of the Geomod runs use the quantity obtained from the linear extrapolation
and the other half of the Geomod runs use the net quantity obtained from the
Markov extrapolation (figure 5). This allows eventual comparison between the
Geomod runs and the CA_Markov runs. Among all runs, there are three possible
quantities: Markov, net Markov, and linear.
Predicting Location of Change. There are four principles that determine the location
where Geomod predicts land change. First, and most importantly, Geomod predictspersistence for the category that grows during the extrapolation. For the example,
the built category grows in net quantity, therefore if a pixel is built in 1985, then
Geomod predicts that it will remain built in 1999. This is different than CA_Markov
and the true process, both of which show some loss of the built category. As a
consequence, Geomod is doomed to fail to predict any real conversion from built to
non-built, which is about 0.8% of the landscape as shown in figures 2 and 6.
Next, Geomod has the ability to stratify the entire analysis, meaning that
Geomod can specify the quantity in each stratum and calibrate the suitability map
separately in each stratum. This is helpful because it is common that data are
available by political unit, and that different political units experience differentpatterns of land change. Our example is not stratified in order to allow a more direct
comparison with CA_Markov, which does not allow for stratification.
Geomods last two principles that determine the predicted location of land change
are similar to the two principles of CA_Markov. Specifically, Geomod can use a
suitability map and/or a contiguity rule.
Similar to CA_Markov, Geomod examines a suitability map to find the pixels
with the largest suitability value, and then predicts conversion of new built at the
non-built pixels that have the largest suitability for built. The suitability map is
completely independent of the duration and the time step of the extrapolation, and
the suitability map remains static for the duration of the simulation.Unlike CA_Markov, Geomod has a method to create empirically and
automatically a suitability map based on several driver maps and a land cover
map from a single point in time. In our example, the single land cover map is the
254 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
14/24
built versus non-built map of 1985. The driver maps are slope and land use of 1971.
The information to compute the suitability map should be restricted to information
that would have been available during the calibration interval, 19711985.
The legend of figure 3 shows the various categories of land use of 1971, where the
darker shade indicates a larger proportion of each category that is built in 1985.
Categories that are pure black are categories that were built in 1971, of which more
than 90% remained built in 1985. The non-black categories were non-built in 1971.
For example, in the 1971 landscape, mining is a non-built category that transitioned
to 25% built by 1985. The non-built categories of Cropland, Water Based
Recreation, Pasture, and Woody Perennial are between 5% and 20% built in
1985. All other non-built categories experienced less than 5% conversion to built
during the interval 19711985.
Information of figure 3 is combined with information of a slope map to generate a
map of suitability for the built category. In general, flatter slopes have a larger
suitability for the built category. Figure 7 is the suitability map for conversion of
non-built to built for both Geomod and CA_Markov. Pontius et al. (2001) discussthe details of the method to calibrate the suitability map from any number of driver
maps. In the example, equal weights are applied to the slope driver and the 1971
land use driver.
A map of the present-day legal restrictions on development is available (figure 4),
so it could be used as another driver to create the suitability map. The date of the
information in the map is not well documented, as is typical of freely-available data.
Presumably many of the laws that regulate land conversion today, were in place in
1985. But several laws were probably created subsequent to 1985, which would
render our map of legal restrictions illegitimate to predict change between 1985 and
1999. Nevertheless, the legal restrictions are probably one of the most importantfactors in determining the land conversion; therefore we perform the model runs
both with and without the map of legal restrictions as a factor in order to measure
the maximum additional predictive power that the information of legal restrictions
could possibly offer. When the map of legal restrictions is used, it forces the
suitability for conversion to the built category to be zero in any pixel that has any
minimal level of legal restriction, meaning that any pixel that is classified as
permanent, limited, or unknown is given a suitability value of 0.
If the contiguity option is used, then Geomods search for the largest suitability
values is constrained to the non-built pixels that are near existing built pixels. This
option is useful for landscapes where growth of a land category occurs adjacent tothat category. The definition of near can be set by the user to include pixels that are
within any number of pixels of the edge of the two categories. Our example uses a
5-by-5 window in order to make the spatial dependency consistent with
CA_Markovs spatial filter. The 5-by-5 window forces the predicted land change
to be within two pixels of the edge including the diagonal. The definition of edge is
updated at every iteration of the time step; therefore it is possible for the time step to
have some influence on the location of the predicted landscape when the contiguity
rule is used. If the extrapolation has small time steps, then Geomod can predict
incremental growth at the locations with large suitability values. If the extrapolation
has a large time step, then Geomod is forced to locate the change near the edge, withless consideration of the suitability values. Similar to CA_Markov, smaller time
steps lead to more iterations and hence more frequent updates of the spatial
dependency. If the contiguity rule is not used, then the time step has no influence on
Comparison of two land change models 255
-
7/30/2019 Pontius y Malason (2005).pdf
15/24
the predicted results. Figure 8 gives an example of the output of a Geomod run,
which is overlaid with maps of the real landscape for 1985 and 1999. The red and green
in figure 8 show the locations where Geomod simulated additional built land between
1985 and 1999.
We also consider three model runs that distribute the predicted quantity of change
evenly across the landscape. According to the mathematics of statistical expectation,the accuracy of these runs is the accuracy that one would expect from a model that
distributes the change at randomly selected locations. The three runs correspond to
the three different methods to predict the quantity of change: Markov, net Markov,
and linear. The net Markov and linear quantities give an increase in the built category,
whereas the Markov quantities give simultaneous gain and loss of the built category.
Goodness-of-fit
Percent correct. Two criteria measure the goodness-of-fit of each model run. The
first criterion is the percent of pixels classified correctly on the entire landscape at
the resolution of the raw data, which are 30-by-30 meter pixels. The percent correct
criterion is extremely popular and frequently misinterpreted. This paper shows how
to interpret the percent correct in an appropriate and sophisticated manner.
There are two crucial issues that a scientist should consider when using the
percent correct criterion. First, the scientist should compare the percent correct of
the predictive model to the percent correct of a null model that predicts pure
persistence. In all cases that we have seen, a null model of pure persistence gives a
larger percent correct than a predictive model at the resolution of the raw data. This
indicates that the resolution of the raw data is finer than the resolution at which the
simulation model can accurately predict the process of change. Second, the scientist
should examine the percent correct at multiple-resolutions in a way similar toCostanza (1989) and Pontius (2002), whereby a pixel aggregation procedure simply
averages neighbouring pixels into coarser pixels in order to quantify the agreement
between the maps at coarser resolutions. As the resolution of the accuracy
assessment becomes coarser, the percent correct tends to rise. The percent correct
Figure 8. Overlay of the real 1985, the real 1999, and the simulated 1999 for the model runthat is most accurate. The two shades of gray and the red are classified correctly by thesimulation model, whereas the blue, green and yellow are errors by the simulation model. Anull model would correctly predict the green and the two shades of gray.
256 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
16/24
increases rapidly at resolutions that correspond to distances at which the model is
accurate. At the coarsest resolution for which one extremely coarse pixel contains
the entire study area, the only error in the accuracy assessment is the error due to
misspecification of quantity.
Figures 9 and 10 show this phenomenon for a null model of pure persistence and
the best model prediction respectively. In each figure, the reference map for 1999 iscompared to the modelled map of 1999. The horizontal axis shows resolution
changing from fine to coarse as one moves from left to right. The vertical axis is the
percent of the entire landscape. The figures show three components of agreement
(agreement due to chance, agreement due to quantity, and agreement due to
location) and two components of disagreement (disagreement due to location and
disagreement due to quantity). The border that separates the agreement due to
location from the disagreement due to location is the percent correct. At the coarsest
resolution, location no longer has meaning, so there are no components associated
with location and the only error is due to quantity. Readers who are especially
interested in the details of the calculations should read Pontius (2000), Pontius(2002) and Pontius and Suedmeyer (in press).
The change in resolution is accomplished by aggregating neighbouring pixels at
the fine resolution into coarser pixels. The aggregation rule computes the aggregate
membership of each category for each coarse pixel as the proportion of fine
resolution pixels of each category that constitute the coarse pixel. Each additional
progression in the aggregation process averages the membership of four
neighbouring pixels, therefore the size of the resolutions progress as a geometric
sequence, where the side of a pixel in each additional resolution is twice the length of
the side of the pixels at the previous resolution.
Null resolution. The second criterion to assess a model run is the null resolution.
Figure 11 shows how to determine the null resolution, which is the resolution at
Figure 9. Components of agreement and disagreement between real 1985 map and real 1999map at multiple resolutions.
Comparison of two land change models 257
-
7/30/2019 Pontius y Malason (2005).pdf
17/24
which the percent correct of the predictive model crosses the percent correct of the
null model. Figure 11 has the same axes as figures 9 and 10, since figure 11 shows an
overlay of the crucial information in figures 9 and 10. In figure 11, the solid line with
the circles (or triangles) is the percent correct for the predictive model (or null
model) respectively. The dashed line with the circles (or triangles) shows the error
due to quantity for the predictive model (or null model) respectively. The solid lines
Figure 10. Components of agreement and disagreement between best predicted 1999 mapand real 1999 map at multiple resolutions.
Figure 11. Illustration of the method to determine that the null resolution is 32 times thelength of the raw pixel side.
258 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
18/24
approach the dashed lines as resolution becomes coarser, and reach the dashed lines
by at least the coarsest possible resolution, which is the resolution at which one
coarse pixel contains the entire study area.
At fine resolutions, the percent correct for the predictive model is less than the
percent correct for the null model. As resolution becomes coarser, the percent
correct for both the predictive model and the null model rises, however the percentcorrect for the null model is constrained by its relatively large error due to quantity.
Since the predictive model has a smaller error due to quantity, the percent correct
for the predictive model becomes larger than the percent correct for the null model
at some resolution, which is called the null resolution. At resolutions finer than the
null resolution, the predictive model is less accurate than the null model of pure
persistence. At resolutions coarser than the null resolution, the predictive model is
more accurate than the null model of persistence only. More accurate models have
finer null resolutions because the null resolution is the resolution at which the
predictive model starts to be more accurate than a null model. For example, figure
11 shows a null resolution of 32 times the size of the side of a pixel of the raw data.
Results
Figure 8 displays the most important information of the best simulation run, which
is 92% correct at the 30-meter resolution. Figure 8 overlays the 1985 truth map, the
1999 truth map, and the 1999 simulated map of a Geomod run that uses net Markov
quantities, considered laws, and is unconstrained. The two shades of gray and the
red are classified correctly by the simulation model, but the gray would be classified
correctly also by a null model that predicts persistence only. Blue, green and yellow
are errors by the simulation model, but the green locations would be predicted
correctly by a null model. Both the null model and the simulation model fail topredict the blue and yellow. Figure 8 shows that the null model has a higher percent
correct than the best simulation run because there is more green than red.
Figure 12 plots the null resolution versus the percent correct at the 30-by-30 meter
resolution for all 24 model runs. Among all runs, the percent correct at the finest
resolution ranges from 91.06 to 92.15. These accuracies are less than the percent
correct of the null model, which is 93.75, but are greater than the accuracies of the
runs that distribute the change evenly across the landscape, which range from 90.75
to 90.91 percent correct. The null resolution for nearly all runs is 32 times the
resolution of the raw 30-meter data, which corresponds roughly to a 1-kilometer
resolution. Four of the runs have a null resolution of approximately 2 kilometres.Figure 12 shows that the model runs are grouped into distinct clusters that are
based more on the options for each model and less on the selection of CA_Markov
versus Geomod. The cluster that has the largest percent correct (ranging from 91.87
to 92.15 percent correct) contains the runs that do not use a contiguity rule. These
runs allow predicted change to occur at the largest suitability values, regardless of
whether those locations are near existing built areas. Within this most accurate
cluster, the runs with the finer null resolution of 32 tend to be from Geomod and to
use the laws in the suitability map. Among the runs that use the contiguity rule,
there are three clusters. Among these three clusters, the one with the largest percent
correct (ranging from 91.52 to 91.71 percent correct) contains all the constrainedGeomod runs. Among these constrained Geomod runs, the ones that predict a
smaller quantity of change have a larger percent correct. The filtered CA_Markov
runs are organized in two sub-clusters. The first of these sub-clusters (ranging from
Comparison of two land change models 259
-
7/30/2019 Pontius y Malason (2005).pdf
19/24
91.30 to 91.39 percent correct) contains the CA_Markov filtered runs that have a
one 14-year time step. The other sub-cluster (ranging from 91.06 to 91.10 percent
correct) contains the CA_Markov filtered runs that have fourteen 1-year time steps.Within these sub-clusters, the runs that predict the smaller quantity of change, give
greater accuracy. In all cases, the use of the laws improves the predictive power of
the extrapolation. The laws account for variation of percent correct within the
clusters, but not variation among the clusters.
The cluster in the far left of figure 12 shows the statistically expected results from
runs that distribute the change randomly. The null resolution of the random runs is
32. In terms of percent correct, all the CA_Markov and Geomod runs perform
better than the random runs, but they also perform worse than the null model.
Furthermore, none of the predictions were more accurate in spatial location than
would be expected by random chance.The range in percent correct among the CA_Markov and Geomod runs is about 1
percentage point. This should be viewed in the context that the range in percent
correct from the random models to the null model is 3 percentage points. A visual
assessment of the various prediction maps shows that the percent correct
corresponds with the pattern agreement, meaning that the larger the percent correct
the more similar the spatial pattern. Most of the variation in the visual assessment is
attributable to the contiguity rule.
Discussion
Interpretation of Results
Percent correct. For any particular modelling application, it is a challenge to
separate the results that are true in general from those that true for only a specific
Figure 12. Null resolution versus percent correct at finest resolution for all model runs.Diamonds denote Geomod, squares denote CA_Markov, triangles denote smooth distribu-tion of location. Gray denotes Markov quantities, black denotes linear quantities. Smallsymbols denote that laws are considered, large symbols denote laws are ignored. Unfilledsymbols denote no contiguity rule, filled symbols denote contiguity rule.
260 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
20/24
landscape. We attempt to interpret our results in terms of general concepts that are
likely to apply to other landscapes and other models.
First, it is extremely important that the modeller be able to control the contiguity
rule. The behaviour of the contiguity rule explains the differences among the clusters
and sub-clusters in Figure 12. The cluster with the largest percent correct uses no
principles of contiguity, meaning that it does not force the prediction of new built tobe near existing built. The next most accurate cluster is the unconstrained Geomod
runs. The effect of the contiguity rule in Geomod is weaker than in CA_Markov.
Geomods contiguity rule merely restricts the candidates for new Built to be near the
existing Built, then Geomod selects the pixels based on the suitability map.
CA_Markovs contiguity rule applies a spatial filter to the entire suitability map,
with strong weighting towards predicting new built to be near existing built. This
filter rule is applied at every time step. As a result, there are two distinct sub-clusters
in figure 12 for the constrained CA_Markov runs. The least accurate cluster is the
one where there are fourteen 1-year time steps, as the filter is applied at each of the
fourteen steps.Some models assume that land change can be predicted accurately using spatial
dependency as expressed in a contiguity rule. However, a quick glance at figures 2
and 3, show clearly that new built areas do not grow from existing built areas.
Therefore, any model that forces spatial dependency is doomed on such a landscape.
This is an important observation because the contiguity assumption is hard-coded
into some LUCC models. For other landscapes at other scales, the best predictor of
the location of new development may indeed be the proximity to existing
development (Mertens and Lambin 2000, Pontius and Batchu 2003, Pontius and
Pacheco in press), but the Massachusetts landscape is an example for which the land
change occurs in entirely new patches, not connected to existing patches (Schneiderand Pontius 2001). Therefore, it is important that models have an option to control
the level of spatial dependency. Both CA_Markov and Geomod have such an
option.
Second, an important basic principle is that when the model predicts change, it
usually predicts it in the wrong location with respect to the fine resolution, therefore
models that predict a small quantity of change tend to have a higher percent correct
than models that predict a larger quantity of change. This phenomenon explains the
variation in accuracy within the clusters. There are three quantities in this analysis.
They are in order of smallest quantity of change to largest: net Markov, Linear, and
Markov. In all cases, the accuracy within clusters follows this ordering, where net
Markov is most accurate and Markov is least accurate.
In no cases did the small 1-year time step improve the accuracy beyond the level
attained by the corresponding run that has one 14-year time step. This supports
Ockhams razor in terms of both accuracy and CPU time.
Null resolution. Nearly all of the runs have a null resolution of 32 times the 30-meter
pixel side, which is about 1 kilometre. The only exceptions are four of the
unconstrained runs, which have a null resolution of about 2 kilometres. It makes
sense that the unconstrained runs can have coarser null resolutions because those
runs allow predicted change to occur far from existing built locations, since the
predictions are unconstrained in space. All the models predictions are less accuratethan a null model at distances less than 1 kilometre. But at distances greater than
2 kilometres, all the models predictions are more accurate than a null models
prediction.
Comparison of two land change models 261
-
7/30/2019 Pontius y Malason (2005).pdf
21/24
The model runs that distribute the change smoothly or randomly across the
landscape also have a null resolution of 32. Therefore, there are no scales at which
any of the model predictions are simultaneously more accurate than both a null
prediction and random prediction. We suspect that this characteristic is typical of
the accuracy of contemporary LUCC models. We know of no spatially explicit
models that have been shown to predict for any landscape better than both a nullmodel and a random model at any spatial resolution.
Signal versus Noise
The results show that it is important to focus on the most important signals of
change on the landscape, as opposed to modelling the detail of the noise. In our
example, the major signal is persistence, with a modest amount of gain of the built
category arranged in patches distributed widely across the landscape. The some of
the noise is the small patches of transition from built to non-built. The predictive
power of the runs is higher for the cases where the model focuses on the signal and
ignores the noise. For example, Geomod ignores the small conversion from built tonon-built, whereas CA_Markov attempts to model the weak empirical pattern of
conversion from built to non-built. When CA_Markov tries to predict this rare
phenomenon, it usually predicts incorrectly. Consequently, Geomod gives more
accurate results by ignoring the rare phenomenon.
The importance of focusing on the signal and ignoring the noise is especially
crucial in light of the quality of the data. Noise derives from both unsystematic
trivial change on the landscape and error in the maps. We do not know exactly how
much error there is in the data because proper metadata for the maps does not exist
and a complete error assessment of the maps was not done by the map producer. It
could be that the small conversion from built to non-built is primarily error. In thiscase, it is wise to ignore this transition. Models that attempt to reproduce all the fine
detail in the calibration data run the risk of modelling the error, which would be
counter-productive.
Scientists need to improve our understanding of how map error influences the
level of certainty we should have in models (Foody and Atkinson 2002). For now,
one necessary step is to apply rigorous procedures that separate calibration
information from validation information. When scientists fail to distinguish between
the calibration process and the validation process, there is a temptation to refine the
model parameters until the model reproduces the noise in the data. This flawed
approach can cause a models output to be a close match with the reference maps,but the apparent accuracy is a result of modelling the noise, i.e. of over-fitting the
model. Such a practice can lure scientists into thinking that a model has a greater
predictive power than it really does.
Importance of goodness-of-fit analysis
Techniques to measure the goodness-of-fit of validation are the least sophisticated
tools in the standard toolbox of the contemporary land change modeller. In the past,
scientists assessed the accuracy of models by a qualitative visual assessment; more
recently modellers have been looking for more sophisticated measures (Foody 2002).
Intuitively, modellers know that a nave interpretation of percent correct is notuseful. This paper offers a sophisticated way to interpret a simple, familiar measure.
The temptation among many scientists is to restrict the accuracy assessment to
some subset of pixels, for example to only the pixels that are non-built at the
262 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
22/24
beginning of the extrapolation, or to only the pixels that truly change during the
time interval of the extrapolation. We recommend strongly against the exclusion of
any pixels from the accuracy assessment and advise in favour of including the entire
landscape in the accuracy assessment. It is best to consider the entire landscape and
compare the predictive model to a null model of pure persistence. Exclusion of pixels
from the accuracy assessment can make the scientist blind to problems in the model.For example, Geomod predicts only a one-way change from non-built to built,
which is a characteristic that some scientists might consider a weakness. If the
accuracy assessment considers only the pixels that were non-built at the beginning of
the extrapolation, then the accuracy assessment would overlook this potential
problem. Moreover, if the accuracy assessment considers only those pixels that truly
change during the time interval of the extrapolation, then there is an incentive for
the model to predict change everyplace, so as to maximize the predicted change in
the pixels that truly did change. The important message is that if a scientist wants to
gain insight into the dynamics of the entire landscape then (s)he should consider the
entire landscape in the accuracy assessment, and if (s)he would like to know theusefulness of a model, then (s)he should compare its predictions to the predictions of
a null model.
Specifics of the data
Additional investigation is required to tell whether our results are representative of a
wide variety of applications. This paper uses an example from one place in space and
time, specifically central Massachusetts during the end of last century. For this
landscape over this duration, most of the change is a steady one-way increase in the
built category. If the same models are applied to different landscapes at different
times, the conclusions might be quite different, but the appropriate methods for
analysis would be the same. Some landscapes are more dynamic and complex than
others, and the apparent complexity is related in part to the detail of the available
data.
Scientists should be aware that available data is usually much more precise in
spatial resolution and number of categories than is the precision of a models
predictive power, so the modeller should not be tempted to let the availability of
complex data be the major criterion upon which to select a model. In our example,
CA_Markov is poor at predicting the location of conversion of built to non-built.
Geomod was more accurate by following a strategy of ignoring the transition from
built to non-built, even though the reference data was sufficiently detailed to show
this rare phenomenon.
Conclusions
The most important difference between Geomod and CA_Markov is that Geomod
predicts only a one-way transition from one category to one alternative category,
whereas CA_Markov has the ability to predict any transition among any number of
categories. Therefore if the important dynamics of a landscape involve simultaneous
gain or loss of a category or involve substantial interactions among more than two
categories, then it would seem that CA_Markov is the better choice based onqualitative characteristics. Alternatively, it might be advisable to simplify the base
data in order to focus on the major signal of change, which frequently is the
conversion from a single non-disturbed category to a single disturbed category,
Comparison of two land change models 263
-
7/30/2019 Pontius y Malason (2005).pdf
23/24
especially in the case of urban landscape modelling. If the goal is to predict a single
dominant process, then Geomod can also be a good choice. Besides this major
conceptual difference, both Geomod and CA_Markov are similar in terms of the
available options that allow for specification of the predicted quantity and location
of categories on a landscape. In our example, the choice of the options accounted for
more variation in accuracy of model runs than the choice of the model.Whatever model is used, the methods of this paper allow a scientist to assess the
accuracy of any model run in a sophisticated manner that gives useful information
for model refinement. The most important aspects of assessing the predictive power
of a model are: 1) to separate the calibration process from the validation process, 2)
to assess the predictive model vis-a-vis a null model, and 3) to perform the accuracy
assessment at multiple resolutions.
Acknowledgements
This research was made possible through the HERO program that NSF supports
through the grant Infrastructure to Develop a Human-Environment RegionalObservatory Network (Award ID 9978052). We acknowledge also programmer
Hao Chen, who was funded by the Center for Integrated Study of the Human
Dimensions of Global Change. This Center has been created through a cooperative
agreement between the National Science Foundation (SRB-9521914) and Carnegie
Mellon University, and has been generously supported by additional grants from
the Electric Power Research Institute, the ExxonMobil Foundation, and the
American Petroleum Institute. The authors thank also Clarklabs, which has made
the methods of this paper available in the GIS software Idrisi.
LiteratureANDERSON, J.R., HARDY, E.E., ROACH, J.T. and WITMER, R.E., 1976, A Land Use And Land
Cover Classification System For Use With Remote Sensor Data. Geological Survey
Professional Paper 964.
BALTZER, H., 2000, Markov chain models for vegetation dynamics. Ecological Modeling, 126,
pp. 139154.
BALTZER, H., BRAUN, P.W. and KOHLER, W., 1998, Cellular automata models for vegetation
dynamics. Ecological Modeling, 107, pp. 113125.
BROWN, D.G., GOOVAERTS, P., BURNUCKI, A. and LI, M.-Y., 2002a, Stochastic Simulation of
Land-Cover Change Using Geostatistics and Generalized Additive Models.
Photogrammetric Engineering & Remote Sensing, 68 (10): pp. 10511061.
BROWN, S., SWINGLAND, I., HANBURY-TENISON, R., PRANCE, G. and MYERS, N., 2002b,Changes in the use and management of forests for abating carbon emissions: issues
and challenges under the Kyoto Protocol. Phil. Trans. R. Soc. Lond. The Royal
Society, 360, pp. 113.
BROWN, S., HALL, M., RUIZ, F., FLAMENCO, A., DEJONG, B., AUKLAND, L.,
MASERA, O. and SHOCH, D., 2002c, http://www.winrock.org/general/Publications/
LandUseAndForestryProjects.pdf.
CLARKE, K.C., 1998, Loose-coupling a cellular automaton model and GIS: long-term urban
growth prediction for San Francisco and Washington/Baltimore. International
Journal of GIS: 12, 17, pp. 699714.
CLARKE, K.C., HOPPEN, S. and GAYDOS, L.J., 1996, A self-modifying cellular automaton
model of historical urbanization in the San Francisco Bay area. Environment andPlanning B, 24, pp. 247261.
COSTANZA, R., 1989, Model goodness of fit: a multiple resolution procedure. Ecological
Modelling, 47, pp. 199215.
264 R. G. Pontius Jr and J. Malanson
-
7/30/2019 Pontius y Malason (2005).pdf
24/24
FOODY, G., 2002, Status of land cover classification accuracy assessment. Remote Sensing of
Environment, 80, pp. 185201.
FOODY, G. and ATKINSON, P., (Eds.)., 2002, Uncertainty in Remote Sensing and GIS. (West
Sussex, England: John Wiley & Sons), 307 pages.
HAGEN, A., 2002, Multi-method assessment of map similarity. 5th AGILE Conference on
Geographic Information Science, Palma, Spain. April 2527; 171182.HAGEN, A., 2003, Fuzzy set approach to assessing similarity of categorical maps. International
Journal of Geographic Information Science, 17 (3): pp. 235249.
HALL, C.A.S. (Ed.)., 2000, Quantifying Sustainable Development: The Future of Tropical
Economies. (San Diego, CA: Academic Press), 761 pages.
HALL, C.A.S., TIAN, H., QI, Y., PONTIUS, G. and CORNELL, J., 1995a, Modelling spatial and
temporal patterns of tropical land use change. Journal of Biogeography, 22 (4/5):
pp. 753757.
HALL, C.A.S., TIAN, H., QI, Y., PONTIUS, G., CORNELL, J. and UHLIG, J., 1995b, Spatially
Explicit Models of Land Use Change and Their Application to the Tropics, Carbon
Dioxide Information Analysis Center, Oak Ridge National Laboratory, DOE
Research Summary 31, February.
MARKOV, A.A., 1907, Extension of the limit theorems of probability theory to a sum of
variables connected in a chain. The Notes of the Imperial Academy of Sciences of St.
Petersburg, VIII Series, Physio-Mathematical College XXII.
MERTENS, B. and LAMBIN, E.F., 2000, Land-Cover-Change Trajectories in Southern
Cameroon. Annals of the Association of American Geographers, 90 (3): pp. 467494.
ORESKES, N., SHRADER - FRECHETTE, K. and BELITZ, K., 1994, Verification, Validation, and
Confirmation of Numerical Models in the Earth. Science, 263 (5147): pp. 641646.
PONTIUS JR, R.G., 1994, Modeling tropical land use change and assessing policies to reduce
carbon dioxide release from Africa. Graduate Program in Environmental Science,
Syracuse: SUNY-ESF, pp. 177.
PONTIUS JR, R.G., 2000, Quantification error versus location error in comparison of
categorical maps. Photogrammetric Engineering & Remote Sensing, 66 (8):pp. 10111016.
PONTIUS JR, R.G., 2002, Statistical methods to partition effects of quantity and location
during comparison of categorical maps at multiple resolutions. Photogrammetric
Engineering & Remote Sensing, 68 (10): pp. 10411049.
PONTIUS JR, R.G., CORNELL, J.D. and HALL, C.A.S., 2001, Modeling the spatial pattern of
land-use change with GEOMOD2: application and validation for Costa Rica.
Agriculture, Ecosystems & Environment, pp. 191203.
PONTIUS JR, R.G. and BATCHU, K., 2003, Using the Relative Operating Characteristic to
quantify certainty in prediction of location of land cover change in India. Transactions
in GIS, 7 (4): pp. 467484.
PONTIUS JR, R.G. and PACHECO, P., 2003, Calibration and validation of a model of forestdisturbance in the Western Ghats, India 19201990. GeoJournal, in press.
PONTIUS JR, R.G. and SUEDMEYER, B., 2003, Components of agreement between categorical
maps at multiple resolutions. In R.S. Lunetta and J.G. Lyon, (Eds). Remote Sensing
and GIS Accuracy Assessment. (Boca Raton, FL: CRC Press).
RYKIEL, E.J., 1996, Testing ecological models: the meaning of validation. Ecological
Modeling, 90, pp. 229244.
SCHNEIDER, L.C. and PONTIUS JR, R.G., 2001, Modeling land-use change in the Ipswich
watershed, Massachusetts, USA. Agriculture, Ecosystems & Environment, 85,
pp. 8394.
Comparison of two land change models 265