realtime bi 를 위한 실시간 데이터 통합 방안 백 철 호 기술이사tsg dnisoft co.,...

Realtime BIRealtime BI 를 위한 를 위한 실시간 데이터 통합 방안실시간 데이터 통합 방안

백 철 호백 철 호기술이사기술이사TSGTSGDNISoft Co., Ltd.DNISoft Co., Ltd.

AgendaAgenda

Data Integration QuadrantData Integration Quadrant Real Time Business IntelligenceReal Time Business Intelligence Real Time BI InfrastructureReal Time BI Infrastructure

Data Integration QuadrantData Integration Quadrant

Real-time Batch

Operation

Analysis

Data Integration QuadrantData Integration Quadrant

Real-time Batch

Operation

Analysis

EAI

ETL

HUB

Migration

Sync

ETL – Basic Building ETL – Basic Building BlocksBlocks Code-generation & Engine-basedCode-generation & Engine-based PerformancePerformance

Multi-thread & Multi-taskMulti-thread & Multi-task Pipeline & Partition Pipeline & Partition DBMS bulk-loadDBMS bulk-load

ExtensibilityExtensibility TransformationTransformation Metadata ManagementMetadata Management Data CleansingData Cleansing

ETL - ETL - 특성 특성 Data integrationData integration is the goal; user interface is the goal; user interface

enables making dramatic changes to data enables making dramatic changes to data that is used by other applicationsthat is used by other applications

Requires Requires scheduled loads of bulk datascheduled loads of bulk data from one database to another from one database to another

Transformations Transformations of data is a key function, of data is a key function, data needs to be data needs to be aggregated, and joinedaggregated, and joined with other data to create new applicationswith other data to create new applications

Metadata is importantMetadata is important, as definitions of , as definitions of metrics are constantly in fluxmetrics are constantly in flux

EAIEAI – Basic Building Blocks– Basic Building Blocks

Communication ModelCommunication Model Sync : request/reply, one-way, pollingSync : request/reply, one-way, polling Async : message passing, Async : message passing,

publish/subscribe, broadcastpublish/subscribe, broadcast

Method of IntegrationMethod of Integration Messaging & InterfaceMessaging & Interface

MiddlewareMiddleware MOM, DOT, TPMMOM, DOT, TPM

EAI - EAI - 특성특성 Business process integrationBusiness process integration is the goal; EAI is the goal; EAI

user interface enables building collaborative user interface enables building collaborative processes across applicationsprocesses across applications

Requires Requires real-time transferreal-time transfer (from one application (from one application to another), of application messages (i.e., to another), of application messages (i.e., generation of a purchase order)generation of a purchase order)

Transformation of messagesTransformation of messages is a key component is a key component because applications use different message because applications use different message formatsformats

Metadata requirementsMetadata requirements are not as important are not as important because message transformations are not as because message transformations are not as complex and do not change like the data complex and do not change like the data transformations required for data warehousingtransformations required for data warehousing

ETL-EAI ETL-EAI 통합통합

Real-time Batch

Operation

Analysis

EAI

ETLETLETL +EAI

Real Time BIReal Time BIWhat It IsWhat It IsWe define We define Real Time BIReal Time BI as: as:

Improving Improving speed and quality for decisionspeed and quality for decision ,,through managing through managing process exceptions and opportunitiesprocess exceptions and opportunities,,by delivering by delivering contextual, right-time informationcontextual, right-time informationfor for continuous business process improvementcontinuous business process improvement..

Process Exceptions and Process Exceptions and opportunitiesopportunities

Real Time BI allows you to monitor your key performance Real Time BI allows you to monitor your key performance indicators (KPIs) in reaction to change or events, either:indicators (KPIs) in reaction to change or events, either:

unexpectedunexpected: errors, outages, new orders, competitor maneuvers, : errors, outages, new orders, competitor maneuvers, surges in volume, rate changes, etc.surges in volume, rate changes, etc.

or or plannedplanned: policy changes, adjacent process changes, marketing : policy changes, adjacent process changes, marketing promotions, mergers & acquisitions, etc.promotions, mergers & acquisitions, etc.

Monitored events can be:Monitored events can be: abruptabrupt, requiring immediate response,, requiring immediate response, or or gradualgradual, making them hard to detect;, making them hard to detect; threatsthreats, disrupting revenue or increasing costs,, disrupting revenue or increasing costs, or or opportunitiesopportunities, offering upside within a limited window., offering upside within a limited window.

Business and system event data is captured in real-time, Business and system event data is captured in real-time, with zero latency.with zero latency.

Contextual InformationContextual Information Real Time BI places monitored activities in context:Real Time BI places monitored activities in context:

Historical and seasonal context – what is normal for right now?Historical and seasonal context – what is normal for right now? Business context – is this critical? where are we in the process?Business context – is this critical? where are we in the process? Organizational context – who cares about this development?Organizational context – who cares about this development?

Real-time data without full context is like knowing your position Real-time data without full context is like knowing your position but not your direction or velocity.but not your direction or velocity.

Inventory is Inventory is dropping towards dropping towards threshold – send threshold – send an alert?an alert?

In context, In context, inventory is inventory is above normal above normal and delivery of a and delivery of a new order is new order is imminent.imminent.

0

100

200

300

400

500

600

8:00 9:00 10:00 11:00

Inventory

Weekly Avg Inventory

Scheduled Deliveries

0

100

200

300

400

500

600

8:00 9:00 10:00 11:00

Inventory

Weekly Avg Inventory

Scheduled Deliveries

Right-Time InformationRight-Time Information BAM filters and processes the real-time data for timeliness BAM filters and processes the real-time data for timeliness

and relevance so that only right-time information is and relevance so that only right-time information is delivered.delivered.

Timing is everything to avoid “business-system thrashing.”Timing is everything to avoid “business-system thrashing.”

The timing of alerts and dashboards must match the The timing of alerts and dashboards must match the frequency of the “appropriate-response cycle”:frequency of the “appropriate-response cycle”:

BAM reports an event only when and where it will have a BAM reports an event only when and where it will have a directly-actionable impact on the business process.directly-actionable impact on the business process.

HourlyDailyMonthlyQuarterlyAnnually

Frequency of Information

Va

lue

of

Info

rma

tio

n

Payroll Processing

Production Scheduling

Knowledge Management

Continuous Business Continuous Business Process ImprovementProcess ImprovementReal Time BI enables you to act in order to close the Real Time BI enables you to act in order to close the

loop, increasing business agility and responsiveness.loop, increasing business agility and responsiveness. It is not the strongest of the species that survives, nor the It is not the strongest of the species that survives, nor the

most intelligent; it is the one that is most adaptable to most intelligent; it is the one that is most adaptable to change.change.

Charles Charles DarwinDarwin

KPIs are derived directly from the execution of the KPIs are derived directly from the execution of the processes themselves, ensuring accurate targets.processes themselves, ensuring accurate targets.

Business processes are continuously extended, Business processes are continuously extended, tuned, and modified in reaction to new patterns of tuned, and modified in reaction to new patterns of activity.activity.

Real Time BIReal Time BIWhat It MeansWhat It Means In Finance:In Finance:

Monitoring settlement in real-time and addressing problems well before close.Monitoring settlement in real-time and addressing problems well before close. Finding patterns of money laundering across millions of transactions.Finding patterns of money laundering across millions of transactions.

In Logistics:In Logistics: Rerouting in-transit vehicles to handle new orders for maximum load factors.Rerouting in-transit vehicles to handle new orders for maximum load factors.

In Telecommunications:In Telecommunications: Optimizing order provisioning and activation processes by shifting resources on the fly.Optimizing order provisioning and activation processes by shifting resources on the fly.

In CPG/Retail:In CPG/Retail: Predicting and preventing late orders before they happen.Predicting and preventing late orders before they happen.

In Insurance:In Insurance: Monitoring liquidity and solvency in real time as capital markets change.Monitoring liquidity and solvency in real time as capital markets change.

In Homeland Security:In Homeland Security: Alerting airport security to suspicious itineraries as travelers check in.Alerting airport security to suspicious itineraries as travelers check in.

Reduce development time (design time integration)Reduce development time (design time integration)Reduce operating expense (run time integration)Reduce operating expense (run time integration)

Comprehensive Integration Architecture

PA

BizTalk Integration Server (EAI)

BusinessProcessManagement(BPM)

• INFA DW• Analytic App’s

Siebel

PSoftSAP

JDEi2

Data Warehouse

Informatica PowerCenter RT (ETL)

Points of Integration

Po

rtal

• Management • Monitoring

• Reporting• OLAP

BizT

alk

BP

M

Real Time BI InfrastructureReal Time BI Infrastructure


Legacy

Documents

Databases

Apps Informatica

ETL

DataWarehouse

Op’l DataStore

BizTalk

Integration Server

Informatica Alerts

BizTalk Workflow

BizTalk Manager

INFA Analytics

Provides the framework for real-time monitoring and analysis, and analytic tools to calculate and display key business performance indicators.

Offers a historical/trend analysis perspective, which helps provide the context for action — (e.g., how significant is the current real-time value compared to historical trends?).

Integrates systems management data from a variety of sources, interpreting the relationship between that data and the business processes it impacts.

Provides an overall business process framework so that events can be analyzed and responded to in the context of the “state” of the business process.

Provides deep connections into underlying application systems and manages the stream of messages that flow among them.

Application Integration

Business Intelligence

Data Integration

Systems Monitoring

Business Process Monitoring

Software Component Utility for Real Time BI


Logical ArchitectureLogical Architecture Technology Technology ComponentsComponents

Event Absorption Layer

Event Processing and Filtering

Real Time Modeling

Event Delivery and Display

Technical Feeds Business Feeds

Valid Event(s)New Events

Data Analysis Tuning


Informatica PowerCenterRT provides context through sophisticated data integration and retrieval.

The BizTalk provides modeling of enterprise processes at the business level for business context.

Informatica PowerAnalyzer provides rich analytics to deliver actionable information to the right person at the right time.

The Real Time BI infrastructure provides full interoperability between all components and functions, common management tools, and consistent

meaning through shared metadata.

The BizTalk Integration Platform provides application integration and access to all your business data in real-time.


Your IT Assets

Real-time events are subscribed by PowerCenterRT

and become input to

the ETL process

The Integration Server accesses the ODS as the table

of record for transaction validations and lookups

The Integration Server calls an ETL service to execute a

batch load or transformation

Data IntegrationData Integration

Process Process IntegrationIntegration

PowerAnalyzer presents right-time information via

dashboard and alerts

Real Time BI InfrastructureReal Time BI InfrastructureBizTalkBizTalk

http://www.webmethods.com/partners/i2.html

BizTalk Server for Real Time BIBizTalk Server for Real Time BI

BizTalk® 2004 ?BizTalk® 2004 ?

기업 전산 환경을 위한 기업 전산 환경을 위한 Integration Integration 솔루션 솔루션 XML/XML/ 웹 서비스 표준 기반으로 제작웹 서비스 표준 기반으로 제작 Visual Studio.NET Visual Studio.NET 통합 개발통합 개발 오피스 솔루션을 이용한 오피스 솔루션을 이용한 View & AnalyzeView & Analyze 이기종 시스템들의 통합 및 연계 이기종 시스템들의 통합 및 연계 (EAI)(EAI) 협력업체 및 파트너 통합 및 연계 협력업체 및 파트너 통합 및 연계 (B2Bi)(B2Bi) Biz. Process Management (BPM)Biz. Process Management (BPM) 솔루션솔루션 통합 인증과 확장성의 제공통합 인증과 확장성의 제공

StandardsStandards XMLXML 웹 서비스 지원은 웹 서비스 지원은 BizTalk 2004 BizTalk 2004 기능의 핵심기능의 핵심

WS-I basic profile, WS-Security plug-inWS-I basic profile, WS-Security plug-in XML foundationXML foundation

W3C XML W3C XML 표준 기반의 스키마표준 기반의 스키마 W3C XSLT W3C XSLT 기반의 스키마 매핑기반의 스키마 매핑

Business Process Execution Language (BPEL)Business Process Execution Language (BPEL) Microsoft, IBM, BEA : Microsoft, IBM, BEA : 자유로운 자유로운 Biz.Biz. 프로세스 호환성프로세스 호환성 비즈니스 프로세스를 위한 최선의 모델비즈니스 프로세스를 위한 최선의 모델 기술정의서기술정의서 (Specification)(Specification) 가 가 OASISOASIS 에 제출됨에 제출됨

pure BPEL4WS

Microsoft OrchestrationRuntime optimized for the

.NET Framework

BPEL4WS

3rd party runtime engine

Business Process Interop

BizTalk End-To-End ArchitectureBizTalk End-To-End Architecture

Message PipelineMessage PipelineDecodeDecodeEncode Encode

DecryptDecryptEncrypt Encrypt

ParseParseSerialize Serialize

ExtractExtractProperties Properties

AuthenticateAuthenticateSign Sign

OrchestrationOrchestration

Activation & Instance ManagementActivation & Instance Management

Correlation & Content-based RoutingCorrelation & Content-based Routing

MessageBoxMessageBox

Transport HandlersTransport Handlers HTTP HTTP SOAP-RM SOAP-RM MSMQ MSMQ SAP SAP MQSeries MQSeries

BusinessBusinessIntelligenceIntelligence

ReportingReporting

MonitoringMonitoring

AdministrationAdministration

DeploymentDeployment

ProcessProcessManagementManagement

Trading PartnerTrading PartnerManagementManagement

Other Other AppsApps

……. .

BizTalk Solution FlowBizTalk Solution Flow

Universal MessageBox Universal MessageBox ArchitectureArchitecture SQL Server SQL Server 기반의 기반의

Scale-out Scale-out 확장성 제공확장성 제공 Pub/SubPub/Sub 아키텍처아키텍처 Stateless serversStateless servers 대용량 메시지 처리대용량 메시지 처리 빠른 메시지 처리와 빠른 메시지 처리와

라우팅라우팅 플러그 방식 어답터플러그 방식 어답터 보안과 감사 기능보안과 감사 기능

Message PipelineMessage Pipeline

DecodeDecodeEncode Encode

DecryptDecryptEncrypt Encrypt

ParseParseSerialize Serialize

ExtractExtractProperties Properties

AuthAuthSign Sign

OrchestrationOrchestration

Activation & Instance ManagementActivation & Instance Management

Correlation & Content-based RoutingCorrelation & Content-based Routing

UNIVERSAL MESSAGEBOXUNIVERSAL MESSAGEBOX

AdaptersAdapters

HTTP HTTP WebWeb

Services Services MSMQ MSMQ Adapters Adapters MQSeries MQSeries

……. .

Host 3 (SvcAcct3)Host 3 (SvcAcct3)

Host 2Host 2(SvcAcct2)(SvcAcct2)

Host 1 (SvcAcct1)Host 1 (SvcAcct1)

Authentication TrustAuthentication Trust

DecryptDecrypt ParseParse ResolveResolvePartyParty

MessageMessageBoxBox

SerializeSerialize SignSignEncryptEncrypt

TransportTransport TransportTransport

인증되지 않은 호스트의 인증되지 않은 호스트의 메시지 메시지 SID/PartyId SID/PartyId 를 를

차단하는 역할차단하는 역할

PartyID, Guest PartyID, Guest 메시지가 없는 메시지가 없는 메시지를 차단메시지를 차단

Scaling Out Processor Scaling Out Processor MachinesMachines

Message boxMessage box

HostHost

MachineMachine

SQL ServerSQL Server

Stored procsStored procs

Data (messages, subscriptions, etc.)Data (messages, subscriptions, etc.)

HostHost

MachineMachine

HostHost

MachineMachine

HostHost

MachineMachine

Network Load BalanceNetwork Load Balance

Scaling Out StorageScaling Out Storage

SQL Server Machine (Master)SQL Server Machine (Master)



DataData

SQL Server MachineSQL Server Machine



DataData

Network Load BalanceNetwork Load Balance

HostHost

MachineMachine

HostHost

MachineMachine

HostHost

MachineMachine

HostHost

MachineMachine

© 2003 Microsoft Corporation. All rights reserved.© 2003 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.