1 JRC – Ispra – 23 July 2004
Luis Rodríguez Lado
E-mail : [email protected]
Alpine Soil Information SystemAnalysis of the accuracy of ESBD in the Alps
region
2 JRC – Ispra – 23 July 2004
Introduction
There is a increasing demand of soil maps and of their properties in the frame of the EU. This information is needed to develop policies linked to sustainable land management practices, and to avoid the damage risk to ecosystems.
At present, the 1:1M digital soil map and of some of their properties are available at European Scale.
3 JRC – Ispra – 23 July 2004
Objective
In this exercise, we evaluate the accuracy of the ESDB maps by comparison with some reference maps derived from detailed survey (ECALP).
4 JRC – Ispra – 23 July 2004
Accurate digital soil maps were computed for 5 pilot areas in the Alps region (ECALP Project).
Methodologydata from ECALP areas
5 JRC – Ispra – 23 July 2004
Maps in the ECALP areas are available as to raster based soil maps.The pilot areas were divided in 1Km2 cells. In this analysis, the soil properties used for each cell are those of the main Soil Map Unit in the cell (% area).
Methodologydata from ECALP areas
6 JRC – Ispra – 23 July 2004
The 1:1M ESDB was rasterized into a 1Km2 cell raster grid.The soil properties for each grid cell were also those of the main Soil Map Unit in the cell (% area).
We compare the results of both maps.
Methodologydata from ESDB areas
7 JRC – Ispra – 23 July 2004
Texture.
Depth of presence of an obstacle to roots.
Depth of presence of an impermeable layer.
Methodologyproperties analyzed
8 JRC – Ispra – 23 July 2004
Methodology
The accuracy of the 1:1 M map is expressed by “naïve” measures of accuracy using “confusion matrices”.
9 JRC – Ispra – 23 July 2004
Objective
PRUEBA ECALP
Cat1 Cat2 Cat3 Cat4 Cat5 Cat6 Cat7 Cat8 Cat9 Cat10Total pi+ User accuracy Producer accuracy Kappa User Kappa Producer
ESDB
Cat14401 37 0 294 13 118 4863 0,9050 0,8008 0,8693 0,7372
0,9050 0,0076 0,0000 0,0605 0,0027 0,0243
Cat234 541 231 1263 51 0 2120 0,2552 0,4055 0,2023 0,3355
0,0160 0,2552 0,1090 0,5958 0,0241 0,0000
Cat30 284 700 542 143 0 1669 0,4194 0,3667 0,3585 0,3094
0,0000 0,1702 0,4194 0,3247 0,0857 0,0000
Cat41054 454 182 4409 130 15 6244 0,7061 0,5938 0,5341 0,4109
0,1688 0,0727 0,0291 0,7061 0,0208 0,0024
Cat57 18 782 851 626 309 2593 0,2414 0,6298 0,2020 0,5750
0,0027 0,0069 0,3016 0,3282 0,2414 0,1192
Cat60 0 14 66 31 2507 2618 0,9576 0,8501 0,9503 0,8277
0,0000 0,0000 0,0053 0,0252 0,0118 0,9576
Cat7 0
Cat8 0
Cat9 0
Cat10
0
Total 5496 1334 1909 7425 994 2949 20107
p+j
Overall Accuracy = 0,6557 Global Kappa = 0,557939
S.D. = 0,0034
10 JRC – Ispra – 23 July 2004
Methodology
The User’s accuracy expresses the probability that one class (in ESDB) is well mapped in relation to the reference dataset (ECALP).
The Producer’s accuracy indicates the proportion of cells that were correctly classified.
11 JRC – Ispra – 23 July 2004
Objective
PRUEBA ECALP
Cat1 Cat2 Cat3 Cat4 Cat5 Cat6 Cat7 Cat8 Cat9 Cat10Total pi+ User accuracy Producer accuracy Kappa User Kappa Producer
ESDB
Cat14401 37 0 294 13 118 4863 0,9050 0,8008 0,8693 0,7372
0,9050 0,0076 0,0000 0,0605 0,0027 0,0243
Cat234 541 231 1263 51 0 2120 0,2552 0,4055 0,2023 0,3355
0,0160 0,2552 0,1090 0,5958 0,0241 0,0000
Cat30 284 700 542 143 0 1669 0,4194 0,3667 0,3585 0,3094
0,0000 0,1702 0,4194 0,3247 0,0857 0,0000
Cat41054 454 182 4409 130 15 6244 0,7061 0,5938 0,5341 0,4109
0,1688 0,0727 0,0291 0,7061 0,0208 0,0024
Cat57 18 782 851 626 309 2593 0,2414 0,6298 0,2020 0,5750
0,0027 0,0069 0,3016 0,3282 0,2414 0,1192
Cat60 0 14 66 31 2507 2618 0,9576 0,8501 0,9503 0,8277
0,0000 0,0000 0,0053 0,0252 0,0118 0,9576
Cat7 0
Cat8 0
Cat9 0
Cat10
0
Total 5496 1334 1909 7425 994 2949 20107
p+j
Overall Accuracy = 0,6557 Global Kappa = 0,557939
S.D. = 0,0034
12 JRC – Ispra – 23 July 2004
Methodology
The Overall accuracy is the sum of the correctly classified cells (diagonal values) divided by the total number of cells analyzed. It indicates the proportion in which those maps agree.
The KAPPA coefficient of agreement is a measure of the chance in the agreement. It indicates whether the agreements found in the overall accuracy are due to the map accuracy of due to chance.
13 JRC – Ispra – 23 July 2004
Objective
PRUEBA ECALP
Cat1 Cat2 Cat3 Cat4 Cat5 Cat6 Cat7 Cat8 Cat9 Cat10Total pi+ User accuracy Producer accuracy Kappa User Kappa Producer
ESDB
Cat14401 37 0 294 13 118 4863 0,9050 0,8008 0,8693 0,7372
0,9050 0,0076 0,0000 0,0605 0,0027 0,0243
Cat234 541 231 1263 51 0 2120 0,2552 0,4055 0,2023 0,3355
0,0160 0,2552 0,1090 0,5958 0,0241 0,0000
Cat30 284 700 542 143 0 1669 0,4194 0,3667 0,3585 0,3094
0,0000 0,1702 0,4194 0,3247 0,0857 0,0000
Cat41054 454 182 4409 130 15 6244 0,7061 0,5938 0,5341 0,4109
0,1688 0,0727 0,0291 0,7061 0,0208 0,0024
Cat57 18 782 851 626 309 2593 0,2414 0,6298 0,2020 0,5750
0,0027 0,0069 0,3016 0,3282 0,2414 0,1192
Cat60 0 14 66 31 2507 2618 0,9576 0,8501 0,9503 0,8277
0,0000 0,0000 0,0053 0,0252 0,0118 0,9576
Cat7 0
Cat8 0
Cat9 0
Cat10
0
Total 5496 1334 1909 7425 994 2949 20107
p+j
Overall Accuracy = 0,6557 Global Kappa = 0,557939
S.D. = 0,0034
14 JRC – Ispra – 23 July 2004
Methodology
For example:An Overall Accuracy of 0.655 indicate that both maps agree in 65% of the cases.
A Kappa statistic of 0,557 indicates that 55,7% of this agreement is due to the mapper competency, and 9,3% of the agreements were due to chance.
15 JRC – Ispra – 23 July 2004
Methodology
Low values of Kappa indicate :
a) Bad Map. Errors due the mapper or to the mapping technique. We can do another map with the same accuracy simply by random assignation using the same classes.
b) An highly homogeneous area (1 class in whole area). For these areas, high values of agreement can be achieved also randomly.
16 JRC – Ispra – 23 July 2004
Results
17 JRC – Ispra – 23 July 2004
Texture
Resultsfrequency distribution (n = 1818 cells)
0
250
500
750
1000
1250
1500
0 1 2 3 4 5 6 7 8 9
Text ECALP
Text ESDB
18 JRC – Ispra – 23 July 2004
Texture
confussion and probabilities matrices; Accuracy index
N
EW
S
Peat SoilsFineMedium-fineMediumCoarseNo Information
Texture ESDB
19 JRC – Ispra – 23 July 2004
Texture class Lombardia-Switzerland
N
EW
S
Peat SoilsFineMedium-fineMediumCoarseNo Information
Texture ESDB
N
EW
S
Texture ECALPNo InformationCoarseMediumMedium-fineFinePeat Soils
N
EW
S
Texture ECALPNo InformationCoarseMediumMedium-fineFinePeat Soils
(ECALP)(ESDB)
20 JRC – Ispra – 23 July 2004
Texture Lombardia
confussion and probabilities matrices; Accuracy index
N
EW
S
Peat SoilsFineMedium-fineMediumCoarseNo Information
Texture ESDB
21 JRC – Ispra – 23 July 2004
Texture class Friuli-Slovenia
N
EW
S
Texture ECALPNo InformationCoarseMediumMedium-fineFinePeat Soils
(ECALP)(ESDB)
22 JRC – Ispra – 23 July 2004
Texture Friuli
confussion and probabilities matrices; Accuracy index
N
EW
S
Peat SoilsFineMedium-fineMediumCoarseNo Information
Texture ESDB
23 JRC – Ispra – 23 July 2004
Conclusions
Texture
0,0000
0,1000
0,2000
0,3000
0,4000
0,5000
0,6000
0,7000
0,8000
0,9000
1,0000
Lombardia Piemonte Friuli Veneto Austria Total
Overall Accuracy
Global Kappa
24 JRC – Ispra – 23 July 2004
Conclusions
User's accuracy
0,0000
0,2000
0,4000
0,6000
0,8000
1,0000
1,2000
Lombardia Piemonte Friuli Veneto Austria Total
No information
Coarse
Medium
Texture
Kappa User's accuracy
0,0000
0,0500
0,1000
0,1500
0,2000
0,2500
0,3000
0,3500
0,4000
0,4500
Lombardia Piemonte Friuli Veneto Austria Total
No information
Coarse
Medium
25 JRC – Ispra – 23 July 2004
ConclusionsTexture
Kappa Producer's accuracy
0,0000
0,0500
0,1000
0,1500
0,2000
0,2500
0,3000
0,3500
0,4000
0,4500
0,5000
Lombardia Piemonte Friuli Veneto Austria Total
No information
Coarse
Medium
Producer's accuracy
0,0000
0,2000
0,4000
0,6000
0,8000
1,0000
1,2000
Lombardia Piemonte Friuli Veneto Austria Total
No information
Coarse
Medium
26 JRC – Ispra – 23 July 2004
Depth of an obstacle for roots
Resultsfrequency distribution (n = 1818 cells)
0
250
500
750
1000
0 1 2 3 4 5 6
ROO ECALP
ROO ESDB
27 JRC – Ispra – 23 July 2004
Depth of an obstacle for roots
confussion and probabilities matrices; Accuracy indexes
Depth class of an obstacle to roots (ECALP)
Roo ECALPNo informationNo obstacle 0-80 cmObstacle 60-80 cmObstacle 40-60 cmObstacle 20-40 cmObstacle 0-80 cmObstacle 0-20 cm
N
EW
S
28 JRC – Ispra – 23 July 2004
Depth to obstacle to roots
Lombardia-Switzerland
(ECALP)(ESDB)
Depth class of an obstacle to roots (ECALP)
Roo ECALPNo informationNo obstacle 0-80 cmObstacle 60-80 cmObstacle 40-60 cmObstacle 20-40 cmObstacle 0-80 cmObstacle 0-20 cm
N
EW
S
Depth class of an obstacle to roots (ECALP)
Roo ECALPNo informationNo obstacle 0-80 cmObstacle 60-80 cmObstacle 40-60 cmObstacle 20-40 cmObstacle 0-80 cmObstacle 0-20 cm
N
EW
S
Depth class of an obstacle to roots (ESDB)
No informationNo obstacle 0-80 cmObstacle 60-80 cmObstacle 40-60 cmObstacle 20-40 cmObstacle 0-80 cmObstacle 0-20 cm
Roo ESDB
N
EW
S
29 JRC – Ispra – 23 July 2004
Conclusions
Obstacle to roots
0,0000
0,1000
0,2000
0,3000
0,4000
0,5000
0,6000
0,7000
0,8000
0,9000
Lombardia Piemonte Friuli Veneto Austria Total
Overall Accuracy
Global Kappa
30 JRC – Ispra – 23 July 2004
Conclusions
User's accuracy
0,0000
0,2000
0,4000
0,6000
0,8000
1,0000
1,2000
Lombardia Piemonte Friuli Veneto Austria Total
No information
No obstacle 0-80 cm
Obstacle 60-80 cm
Obstacle 40-60 cm
Obstacle 20-40 cm
Obstacles to roots
Kappa User's accuracy
0,0000
0,2000
0,4000
0,6000
0,8000
1,0000
1,2000
Lombardia Piemonte Friuli Veneto Austria Total
No information
No obstacle 0-80 cm
Obstacle 60-80 cm
Obstacle 40-60 cm
Obstacle 20-40 cm
31 JRC – Ispra – 23 July 2004
ConclusionsObstacles to roots
Kappa Producer's accuracy
0,0000
0,1000
0,2000
0,3000
0,4000
0,5000
0,6000
0,7000
0,8000
0,9000
Lombardia Piemonte Friuli Veneto Austria Total
No information
No obstacle 0-80 cm
Obstacle 60-80 cm
Obstacle 40-60 cm
Obstacle 20-40 cm
Producer's accuracy
0,0000
0,1000
0,2000
0,3000
0,4000
0,5000
0,6000
0,7000
0,8000
0,9000
1,0000
Lombardia Piemonte Friuli Veneto Austria Total
No information
No obstacle 0-80 cm
Obstacle 60-80 cm
Obstacle 40-60 cm
Obstacle 20-40 cm
32 JRC – Ispra – 23 July 2004
Depth of an impermeable layer
Resultsfrequency distribution (n = 1818 cells)
0
500
1000
1500
2000
0 1 2 3 4
IL ECALP
IL ESDB
33 JRC – Ispra – 23 July 2004
Depth of an impermeable layer
Results confussion and probabilities matrices; Accuracy index
(n = 1818 cells)
34 JRC – Ispra – 23 July 2004
Conclusions
Depth of an impermeable layer
0,0000
0,1000
0,2000
0,3000
0,4000
0,5000
0,6000
0,7000
0,8000
0,9000
Lombardia Piemonte Friuli Veneto Austria Total
Overall Accuracy
Global Kappa
35 JRC – Ispra – 23 July 2004
Conclusions
We found that the present 1:1M ESDB maps means a great generalization of soils and their properties, being inappropriate to derive effective policies in the EU at medium and large scales due to the uncertainty of its information.The overall accuracy of these maps is generally lower than 50%. It varies between 0,33 (obstacle to roots) to 0,8 (depth of impermeable layer) but low values of Kappa were found, indicating high influence of chance in the success of classification.This low values of Kappa are greatly due to the low discrimination in classes in ESDB (general map).
36 JRC – Ispra – 23 July 2004
Conclusions
Friuli-Slovenia was the region that showed a better agreement with the ECALP database, particularly for the depth of an obstacle to roots, where it also exhibits a high value of Kappa.
37 JRC – Ispra – 23 July 2004
Conclusions
1. Need of more accurate soil maps than ESDB
2. Provide soil sample description as metadata
3. Consensus in the description of properties
4. Implementation of accuracy tests for maps