wtxandwps_wesb

8/4/2019 WTXandWPS_WESB

1/41

2006 IBM CorporationConference materials may not be reproduced in whole or in part without the prior written permission of IBM.

SucceedCollaborate

Learn

Network

SucceedCollaborate

Learn

Network

WebSphere Transformation Extender

( formerly Datastage TX )Integration with WAS, WPS and WESB

Shahryar Sedghi

Senior Consulting IT [email protected]


2/41

2

Objectives

Learn about WTX and How it solves Integration Problems

Why we need to Integrate it with other WebSphere products

Integration techniques

Usage Patterns

Best practices of Integration


3/41

3

WTX History

Formerly Mercator, 3-5% Market Share in the BI sector

Unlike all the others, a major player on Z Apps (CICS, Batch)

Acquired by Ascential to improve the ETL solutions

Ascential was acquired by SWG DB2 Davison for Its ETL

solutions

WTX transferred to AIM to improve our WebSphere

Integration story


4/41

4

Data Integration Terms

ParserParses (De-serialize) an Input Stream (aka. Byte Array) to atree or a hierarchical Object

Serialize the tree or hierarchical object to an output stream Apache Xerces is an XML parser

Mapper Converts from one tree or object to anotherXSLT is a standard mapper for XML

AdapterDeals with different Source and TargetsFile , Database, MQ, JMS,.

BrokerRuns a flow of various activities, involve Adapters, Parsing,

Mapping and Decision MakingBrokers implement ESB (Enterprise Services Bus)


5/41

5

WTX Components

Parser for various Data FormatsTag Delimited Data (Industry standard, aka. X12,HL7, HIPPA), Already has many off the shelfformats

Fixed Width (COBOL)XML (Basic and SOAP)Legacy Parsers implement lazy parsing

MapperAdapterFile, Database, SAP, PeopleSoft,.

BrokerEvent Server

ToolingWindows Based (Not Eclipse)


6/41

6

WTX Integration Patterns Standalone

WTXInput Stream

WPSWESB

WAS SIBWebSphere MQ

Pros

All adapter capabilities

Better Performance for large Messages

Cons

MQ get and Put overhead

Managing two environments


7/417

WTX Integration Patterns Integrated

WTX

Input Stream

Pros

Managing a single environment

A simpler model

Cons

No use of adapters

WPG,WPS,WESB,WBIMB, WAS SIB

Output Stream


8/418

WTX Integrated solutions - What we need to Learn

Design StudioType Designer

Creating Types with ImportersCreating Type Manually

Map DesignerCreate Maps between Input and Output Types

Testing, Debugging

Optionally, Development ToolkitJava, RMI and JCA Adapter API


9/419

A WTX Mapping Scenario A Sample BO


10/41

10

Type Designer XSD Input

Does not existin WPS/WESB BOs


11/41

11

Type Designer Type Tree from XML


12/41

12

Type Designer COBOL Input

01 Student.02 name PIC X(20).02 lastName PIC X(20).02 address1 PIC X(20).

02 address2 PIC X(20).02 numOfSchools PIC 99.02 schools OCCURS 0 TO 99 TIMES

DEPENDING ON numOfSchools.03 schoolName PIC X(20).03 schoolAddress PIC X(20).

02 FILLER PIC XX.


13/41

13

Type Designer COBOL Type Tree


14/41

14

Map Designer

Submap


15/41

15

WTX Map Artifacts

Files Type Tree .mtt Map Source File .mms Compiled map .mmc

Map Source file requires Type Trees to buildcompiled maps Compiled maps do not require any artifact,

unless a card uses a Xerces Parser (V8.0 orabove)


16/41

16

WPS/WESB + WTX ?

WPS/WESB natively support XML COBOL Copybook and C Header files( CICSand IMS adapter tooling)

WTX is required for: Industry Standard Formats (aka. TagDelimited Data)

Content based routing for Non-XML data


17/41

17

Inbound Data De-Serialization (Parsing)

BytesStream SDO

SCA Export

Data Binding

Edge

Bytes StreamWrapped in

SDO

SDO

Java Component

Middle

POJO


18/41

18

Oubound Data Serialization

Data BindingSDOByte

Stream

Edge

SCA Import

Center

Bytes StreamWrapped in

SDO

SDO

Java Component

POJO


19/41

19

WTX, WPS and WESB ISSW Offerings

WTXJMSDataBinding

Can be used in SCA JMS Import/Export binding inWPS/WESB

Fully parameterized and configurable through WID

WTXMedaition

Complements the WPS/WESB solutionNormally is not required if best practices followed

WTXMaprunner

A Wrapper for WTX Java API

Currently supports one Input and One output card which is sufficient forWPS/WESB

Designed for WebSphere Environment. Loads Maps from Java

classpath

Is not a perfect solution yet (No Caching,..)


20/41

20

ISSW Offering - JMS Enablement

JMS

Export

WPS Process

Or

WESB Mediation

JMS

Import

JMSData Binding

For WTXJMSData Binding

For WTX

Input Queue

WTX

Medaition

Output Queue

WTXMedaition

Incoming Data

SIBus

Outgoing Data

Either JMS Data Binding for WTX or WTX Mediation Can be used not bothtogether, on the same destination

WTX Mediation helps using WTX without modifying Existing Processesand Mediations

Non-JMS WMQ Applications can take advantage of the mediation

Inbound

Outbound


21/41

21

WTXJMSDataBinding - Outbound

JMS

Export

JMS

Import

EMFXML

Parser ByteStream

MapXMLDataSDO

WTXOutMapJMS Property

LegacyLegacy

WPS Process

Or

WESBMediation

WTXOutMapJMS Property


22/41

22

WTXJMSDataBinding - Inbound

JMS

Export

WPS Process

Or

WESBMediation

JMS

Import

EMF

XMLParserByteStream Map XMLData SDO

WTXInMap

JMS PropertyWTXInMap

JMS Property

LegacyLegacy


23/41

23

WTX SIBus Mediation

Destination

WTXMedaition

Map NameMessage Format

And OtherContext Properties

BusJMS Text orBytes Message

JMS Text orBytes Message

InspectMessage GenerateMessageByteStream ByteStreamMap


24/41

24

WTXMedaition Code

SIMessageContext siCtx = (SIMessageContext)ctx;SIMessage message = siCtx.getSIMessage();

String messageFormat = (String)ctx.getProperty("Message Format");String mapName = (String)ctx.getProperty("Map Name");byte[] inBytes = message.getDataGraphAsBytes();

byte[] outBytes = new WTXMapRunner().runMap(mapName, inBytes);

DataGraph newDataGraph =SIDataGraphFactory.getInstance().createDataGraph(outBytes,messageFormat);

message.setDataGraph(newDataGraph, messageFormat);

for(Iterator it = ctx.getPropertyNames();it.hasNext();) {String propName = (String) it.next();message.setUserProperty(propName,(Serializable)ctx.getProperty(propName) ); // Optional

}


25/41

25

WTX Non-JMS Inbound, Edge

Flat File

Adapter

Export

SCA

Import

EMF

XMLParserByteStream Map XMLData SDO

File

WPS POJO

WESB CustomMediation

CustomData Binding


26/41

26

WTX Non-JMS Inbound, Middle

Flat File

Adapter

Export

SCA

Import

EMFXML

Parser

Bytes Stream

Wrapped inSDO

Map

XML

Data SDO

File

WPS POJO


Bytes

Stream

Extract

Bytes


27/41

27

WTX Non-JMS Outbound, Edge

SCA

Export

Flat File

Adapter

Import

EMFXML

Parser ByteStream

MapXMLDataSDO

WPS Process

Or

WESBMediation

File

CustomData

Binding


28/41

28

WTX Non-JMS Outbound, Middle

SCA

Export

Flat File

Adapter

Import

EMF

XMLParser ByteStreamMapXMLDataSDO

File

WPS POJO


Bytes Stream

Wrapped inSDO


29/41

29

Usage Patterns

ESB Legacy to Web Service Data Integration

Content Based Routing Process Integration

Legacy to Business Processes


30/41

30

Legacy to Web Service Lab 1

InternetCurrencyExchange

Service

XSLT

XSLT

Out

In

In

Out

JMSExport

WSImport

Mediation

JMS Client

Mediation Module

1

2

3

45

7

6


31/41

31

Data Integration Lab 2

Data Map

JMSOutInterfaceJMSInInterface

JMSImport

Interface Mediation

JMS Client

WPS Module

1

2

3

4

5

6JMS

Export

JMS Client is for demonstration only

Typically Message Producers and Message Consumers are different


32/41

32

Content Based Routing

Event Aware

JMSExport Router

Component

Event Agnostic

Component

Component

Component

Generic Interface

Can be a POJO or aCustom Selector

WTXJMSDatabinding


33/41

33

Generic Interface


34/41

34

Legacy to Process Lab 3 (simulated)

Flat File

Adapter

Export

JMS

Import

File

BPEL

HTM

POJO

WTXMapRunner


35/41

35

Deployment Prerequisites

EnvironmentWPS/WESB interact with WTX Java API Java API invokes Shared Libraries through JNI layer

WTX Jar (dstxpi.jar) must be in the classpath

Shared Libraries (aka. DLLs) must be in the systemPATH (Path for WAS user profile in UNIX/Linux) Complied Maps must be accessible to the code Compiled map needs XSDs for validation if:

XML is inbound to the mapXerces parser is used (8.0 and above)


36/41

36

Deployment Typical practice

Add dstxpi.jar to the system classpath

Add WTX Install root to system PATH Move Compiled Map and all XSDs to a

specific directory

In a WAS cluster repeat step 1-3 for everyServer

Anytime a map changes replace the map in

every server


37/41

37

Deployment Best Practices

Use WAS Shared Library to add dstxpi.jar tothe classpathAdd WTX Install root to system PATHWAS shared library support for native libraries

does not work for WTX ( DLLs calling otherDLLs)Load the map from classpath, rather than afolderAllows deploying maps with process EAR file

No need to copy maps to each serve Follow XML Best practices


38/41

38

WTX and XML

WTX validates every incoming data includingXML WTX V8.0 supports Xerces Parser for XML Xerces requires XSD at runtime for validation

Xerces is faster for large XML documents Classic XML parser does not require XSDsand Type Trees for inbound XML

Mapping outbound XML is very difficult withclassic parser


39/41

39

XML Best practices

Use Classic parser for inbound XMLEliminates the need for copying XSDs toeach server

Use Xerces parser for outgoing XML

Outgoing XML does not need XSDs forvalidationUse Xerces if Classic parser is very slow forlarge documentsCopy XSDs to the map working directory

WTXMapRunner needs a System PropertyXSDs change less frequently than maps


40/41

40

Links

Lab Docs (3MB):

ftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/LabDocuments.zip

Solutions Folder without Workspaces (228 KB):

ftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions_without_workspace.zip

Solutions Folder with workspaces (30 MB)

ftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions.zip

VMWare (5.2 GB)

ftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/base_vm.zip
ftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/LabDocuments.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/LabDocuments.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions_without_workspace.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions_without_workspace.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/base_vm.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/base_vm.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/base_vm.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/base_vm.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions_without_workspace.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions_without_workspace.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions_without_workspace.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/LabDocuments.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/LabDocuments.zipftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/LabDocuments.zip


41/41

Questions