fme workspace template collection for reading fundamental geographic data in japan

1. CONNECT. TRANSFORM. AUTOMATE. FME Workspace Template Collection for Reading Fundamental Geographic Data in Japan Takashi Iijima FME Certified Professional, pragmatica inc.

2. Background: KSJ and FGD Major Fundamental Geographic Datasets provided by Japanese Governmental Organizations KSJ: (Kokudo Suuchi Jouho) National Land Numerical Information FGD: (Kiban Chizu Jouho) Fundamental Geospatial Data CONNECT. TRANSFORM. AUTOMATE. 3. Background: KSJ Overview ! Provider: -- Ministry of Land, Infrastructure, Transport and Tourism (MLIT) ! http://nlftp.mlit.go.jp/ksj/ ! Feature Types: 60+; Geographic Features, Land Use, Political Designated Zones, Administrative Areas, Public Facilities, Transportation Facilities etc. (Point, Line or Area) ! Format: XML conforming to GML ! GML reader cannot read the data. CONNECT. TRANSFORM. AUTOMATE. 4. Background: FGD Overview ! Provider: -- Geospatial Information Authority of Japan (GSI) ! http://www.gsi.go.jp/kiban/etsuran.html ! Feature Types: Vectorized Topographical Map Elements (Point, Line or Area), Survey Reference Point, Digital Elevation Model (DEM) ! Format: XML conforming to GML ! GML reader can read the data except DEM. CONNECT. TRANSFORM. AUTOMATE. 5. Project Overview ! Purpose: To provide easy and flexible way to read KSJ and DEM in FGD with FME to increase their usability. ! Approach: Create FME workspaces to read the datasets with the XML reader and create features having appropriate geometry and attributes. ! Goal: FME Template Collection CONNECT. TRANSFORM. AUTOMATE. 6. Part 1. Reading KSJ ! General Data Structure ! Framework of Workspaces ! Geometry Creation Point ! Geometry Creation Line ! Geometry Creation Area ! Resolving Incorrect Data with xfMap CONNECT. TRANSFORM. AUTOMATE. 7. 20.0 123.046.0 154.0190034.98509412 139.860441161JR KSJ: General Data Structure CONNECT. TRANSFORM. AUTOMATE. Envelope Geometries Features GML: Point, Curve, OrientableCurve, Surface A feature links to a geometry via "href" attribute. Contains SRS name: JGD2000 or Tokyo Datum 8. KSJ: Framework of Workspaces CONNECT. TRANSFORM. AUTOMATE. Merge Features Geometries Envelope (SRS) Merge Set coordinate system Modify attribute names XML Reader feature type with Feature Type Specific xfMap XML Reader feature type with Common xfMap Creates geometries (Line, Area) with Custom Transformer XML Reader feature type created with Common xfMap Add geometry to feature Group By: fme_basename Join On: geometry ID Add SRS name to feature Group By: fme_basename Unconditional merging 9. KSJ: Geometry Creation - Point Mapping rule for CONNECT. TRANSFORM. AUTOMATE. element example 36.02232800 139.0030680036.01240300 139.0219210036.04452300 139.0543590035.98977700 139.0771060035.99583200 139.07745600 Points can be created simply with xfMap. 10. KSJ: Geometry Creation - Line CONNECT. TRANSFORM. AUTOMATE. Read elements as XML fragments. Create Line geometries with a Custom Transformer. Mapping rule for element example 35.91887573 139.92888998 35.91885861 139.92893361 35.91885472 139.92894500 35.91885472 139.92895778 35.91887972 139.92899667 35.91925667 139.92939778 35.91927722 139.92945889 11. KSJ: Geometry Creation - Line CONNECT. TRANSFORM. AUTOMATE. KsjLineBuilder (Custom Transformer) creates Line geometries. 12. Surface element example KSJ: Geometry Creation - Area CONNECT. TRANSFORM. AUTOMATE. Mapping rule for Surface Complicated hierarchy XML structure can be mapped to nested-structured list attribute. Create Area geometries with a Custom Transformer. 13. KSJ: Geometry Creation - Area CONNECT. TRANSFORM. AUTOMATE. OrientableCurve element example An OrientableCurve represents a reversed Curve, links to a Curve by its ID. Mapping Rule for OrientableCurve 14. KSJ: Geometry Creation - Area CONNECT. TRANSFORM. AUTOMATE. KsjAreaBuilder (Custom Transformer) creates Area geometries. 15. KSJ: Bus Stop (Point) CONNECT. TRANSFORM. AUTOMATE. 16. KSJ: River (Line and Point) CONNECT. TRANSFORM. AUTOMATE. 17. KSJ: Administrative Boundary (Area) CONNECT. TRANSFORM. AUTOMATE. 18. KSJ: Resolving Incorrect Data with xfMap Case 1 CONNECT. TRANSFORM. AUTOMATE. Incorrect - wrong feature type/attribute names Two Mapping rules for the same feature type Multiple mapping rules for the same feature type are allowed if match attributes (XML element names) are different. Correct 19. KSJ: Resolving Incorrect Data with xfMap Case 2 CONNECT. TRANSFORM. AUTOMATE. Correct 36.02232800 139.00306800 Incorrect - wrong GML element name 36.08985800 139.78870900 Mapping rule modified for incorrect data element in xfMap generates just an empty string if the XML path specified by expr attribute doesnt exist. This mapping rule works for both correct data and incorrect data. 20. Part 2. Reading DEM in FGD ! DEM Types ! Data Structure ! Sequence of Elevation Values ! Skeleton of Workspace ! Process of Creating Raster ! Implementation and Performance CONNECT. TRANSFORM. AUTOMATE. 21. DEM: DEM Types Type Name Cell Sizes width x height Columns x Rows per a Dataset Extent Sizes width x height 5m Mesh 0.2 x 0.2 sec. 225 x 150 45 x 30 sec. 10m Mesh 0.4 x 0.4 sec. 1125 x 750 450 x 300 sec. 250m Mesh 11.25 x 7.5 sec. 320 x 320 3600 x 2400 sec. CONNECT. TRANSFORM. AUTOMATE. Both 5m Mesh DEM and 10m Mesh DEM are covering all of Japanese land area; 250m Mesh DEM is covering a special area only. Ideally the workspace should be applicable to any type of DEM. 22. DEM: Data Structure CONNECT. TRANSFORM. AUTOMATE. 38.916666667 139.838.925 139.81250 0224 149 ,-9999. ,-9999. ,-9999. ,0.63 ,6.83 ,6.67 ,6.57 108 0 Number of columns and rows: num_columns = high{0} + 1 low{0} num_rows = high{1} + 1 low{1} List of (Type, Elevation) SRS name: jgd2000.bl or jgd2011.bl Extents: (ymin, xmin), (ymax, xmax) Index of start point in the grid matrix: start_col = startPoint{0} start_row = startPoint{1} 23. DEM: Sequence of Elevation Values CONNECT. TRANSFORM. AUTOMATE. 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 Start Column Index = 6, Start Row Index = 2 Elevation values are put in order in the turn of going to Right-Lower from Left-Upper. Location of the starting cell is specified by start column, row index described in the element. ,-9999. ,-9999. ,-9999. ,0.63 ,6.83 ,6.67 ,6.57 > 24. DEM: Skeleton of Workspace CONNECT. TRANSFORM. AUTOMATE. Dataset Extract Number of Columns / Rows Extract Coordinates of Lower / Upper Corner Create Raster Origin = (0, 0) Cell Sizes = 1 x 1 Width = Number of Columns Height = Number of Rows Resize, Move, Set CoordSys XML Reader with Feature Paths Scaler Offsetter CoordinateSystemSetter AttributeSplitter AttributeCreator AttributeSplitter NumericRasterizer parameter settings Size Specification: RowsColumns Number of Columns (cells): Number of Rows (cells): Ground Extents Minimum X: 0 Minimum Y: 0 Maximum X: Maximum Y: Scaler Offsetter (0, 0) (xmin, ymin) 25. DEM: Process of Creating Raster CONNECT. TRANSFORM. AUTOMATE. Extract Start Column / Row Index Serialize Elevation Values Create 3D Grid Points Transform 3D Points into a Raster 3D Point Coordinate Calculation Index = (0-based Index in the Sequence) + (Start Row Index) * (Number of Columns) + (Start Column Index) Column Index = Index % (Number of Columns) Row Index = INT(Index / (Number of Columns)) X = (Column Index) + 0.5 Y = (Number of Rows) 0.5 (Row Index) Z = Elevation NumericRasterizer parameter settings Size Specification: RowsColumns Number of Columns (cells): Number of Rows (cells): Ground Extents Minimum X: 0 Minimum Y: 0 Maximum X: Maximum Y: Extract Number of Columns / Rows 26. DEM: Implementation and Performance CONNECT. TRANSFORM. AUTOMATE. 1st Implementation: uses existing Transformers only. About 110 sec. for rasterizing a 10m Mesh DEM (843,750 cells). The ListExploder seems to consume long time. 27. DEM: Implementation and Performance CONNECT. TRANSFORM. AUTOMATE. 2nd Implementation: uses a Python script to extract Elevation values and create 3D Grid Points. About 25 sec.: Performance has been improved considerably. But there should be more efficient way. import fmeobjects class GridPointCreator(object): def input(self, feature): count = feature.getAttribute('_count') numCols = int(feature.getAttribute('_num_columns')) numRows = int(feature.getAttribute('_num_rows')) startCol = int(feature.getAttribute('_start{0}')) startRow = int(feature.getAttribute('_start{1}')) tuples = feature.getAttribute('DEM.coverage.rangeSet.DataBlock.tupleList').strip() index = startRow * numCols + startCol for z in [float(t.split(',')[1]) for t in tuples.split()]: x = index % numCols + 0.5 y = numRows - 0.5 - index / numCols point = fmeobjects.FMEFeature() point.setAttribute('_count', count) point.setAttribute('_num_columns', numCols) point.setAttribute('_num_rows', numRows) point.setGeometry(fmeobjects.FMEPoint(x, y, z)) self.pyoutput(point) index += 1 28. DEM: Implementation and Performance CONNECT. TRANSFORM. AUTOMATE. 3rd Implementation: inserted a PointCloudCombiner to transform 3D Points into a Point Cloud before rasterizing. About 12 sec.: Performance has been improved further more. Is it the best performance? Special Thanks: The Point Cloud Technology was suggested by Dale and Chris@Safe. 29. DEM: Implementation and Performance CONNECT. TRANSFORM. AUTOMATE. Final Implementation: updated the Python script so that it will create a 3D Line having all the 3D Grid Points as vertices. About 2.5 sec.: Quite faster than creating many point features. import fmeobjects def create3DLine(feature): numCols = int(feature.getAttribute('_num_columns')) numRows = int(feature.getAttribute('_num_rows')) startCol = int(feature.getAttribute('_start{0}')) startRow = int(feature.getAttribute('_start{1}')) tuples = feature.getAttribute('DEM.coverage.rangeSet.DataBlock.tupleList').strip() index = startRow * numCols + startCol coords = [] for z in [float(t.split(',')[1]) for t in tuples.split()]: x = index % numCols + 0.5 y = numRows - 0.5 - index / numCols coords.append((x, y, z)) index += 1 feature.setGeometry(fmeobjects.FMELine(coords)) 30. DEM: Implementation and Performance Level Processing Time Comments 1st 110 sec. (100%) Existing Transformers Only 2nd 25 sec. (23%) Create 3D Grid Points with Python 3rd 12 sec. (11%) Introduced the Point Cloud Technology Final 2.5 sec. (2.3%) Create 3D Line with Python CONNECT. TRANSFORM. AUTOMATE. Tested with FGD 10m Mesh DEM: FG-GML-5338-41-dem10b-20090201.xml FME 2014 SP2 WIN32, Windows 7 64-bit Conclusion: Python scripting is worth to consider for improving performance, especially if the ListExploder looks inefficient. Using a Point Cloud as input for the NumericRasterizer is very efficient. Summary 31. DEM: Implementation and Performance CONNECT. TRANSFORM. AUTOMATE. Rasterizing Result: Final Workspace 32. Thank You! ! Questions? ! For more information: ! Takashi Iijima [email protected] ! Pragmatica Inc. ! http://fme-memorandum-takashi.blogspot.jp/ CONNECT. TRANSFORM. AUTOMATE.