taking distributed tracing - jax london · 2019-11-05 · taking distributed tracing to the next...

DISTRIBUTED TRACINGTAKING

TO THE NEXT LEVEL

WHO ARE WE?

▸ Frank Pfleger@frankpflegerfrank@trasier.com

▸ Lukasz Pielak@lukaszpielaklukasz@trasier.com

WHAT IS THIS TALK ABOUT

▸ The Fundamental Concepts

▸ The Current State

▸ The Next Level

TAKING DISTRIBUTED TRACING TO THE NEXT LEVEL

LOGGINGMETRICS TRACING

OBSERVABILITY

LOGGING

LOGGING

THERE IS ALWAYS AN ALTERNATIVE ;)

LOGGING

LOGGING

LOGGING - MDC

Source: https://www.baeldung.com/mdc-in-log4j-2-logback

TRACEABILITY OF SOFTWARE

MONOLITHIC ARCHITECTURE

TRACEABILITY OF SOFTWARE

MICROSERVICES

TRACEABILITY OF SOFTWARE

MDC CONTEXT PROPAGATION

TK TK TK

TKTK

TK

TK

TK TK

DISTRIBUTED TRACING

DISTRIBUTED TRACING

BRINGING EVENTS IN

CAUSAL ORDER IN

DISTRIBUTED ENVIRONMENT

DISTRIBUTED TRACING

Span TracerTrace Context(span id, trace id)

Trace

DISTRIBUTED TRACING

WHY DISTRIBUTED TRACING?

▸ Latency visualization

▸ Service dependency visualization

▸ Hidden architecture revealed

▸ Error analysis

▸ Infrastructure check

▸ Version check

▸ Trace Context propagation

▸ in-process

▸ inter-process

▸ Clock skew / synchronization

▸ Performance and stability

▸ Data ingestion, storage, retention

▸ GDPA

DISTRIBUTED TRACING

PROBLEMS OF DISTRIBUTED TRACING

SERVICE B

SERVICE A

Latency measurement points

Host 1

Host 2

CORRELATE LOG ENTRIES

TRACE COMMUNICATION BETWEEN SERVICES

USE DISTRIBUTED TRACING

USE TRACE-ID IN THE MDC

THE CURRENT STATE OF DISTRIBUTED TRACING

BOOKING PAYMENT

USER INTERFACE Live Reporting(Performance monitoring)

OFFER

DATASTORE BigData Store(High-Performance / Indexing)

WEBS

HOP

Context propagation(Service Instrumentation)

Common language(Specification / Trace-Context)

COLLECTOR Decoupled Reporter(Asynchronous collecting)

APM - PROVIDERSSUPPORT FOR OPENTRACING ENABLES

TOOLING LANDSCAPE

GOAL: MAKE OBSERVABILITY EASY FOR MODERN APPLICATIONS

libraries used for collecting tracing and metrics dataready to use implementation for database engines, kubernetes etc.tightly coupled to the implementationsopinionated implementation for capturing observability signals

COMPLEMENTARY NOT CONTRADICTORY

vendor-neutral API with default implementations

+ =

vendor-neutral API wide vendor supporttracer implantation effort

DEMO

https://github.com/trasiercom/springboot-example/branches

TAKING DISTRIBUTED TRACING

TO THE NEXT LEVEL

DISTRIBUTED TRACING

DISTRIBUTED TRACING

credits to jaeger documentation

THE NEXT LEVEL

BUSINESS TRACING EASY BUG TRIAGE

BUSINESS ANALYSIS PREDICTIVE ANALYSIS

MOCK SERVICES REPLAY CONVERSATIONS

BUSINESS TRACING

CONVERSATION AS A WHOLE

CUSTOMER CHARGED TOO MUCH?

INSPECT PAYLOAD OF TRANSACTIONS

ON-THE-FLY ERROR DETECTION

https://ui.trasier.com/#/account/170522/space/nova-test/search/id/267cd7b7-2d6a-474b-ade2-bf0453881f72?spanId=f53e4982-a7c0-4117-b98c-91260eefbdbb&view=editor&editortab=response

DEMO

https://github.com/trasiercom/springboot-example/branches

THE NEXT LEVEL

EASY BUG TRIAGE

https://ui.trasier.com/#/account/123456/space/sys-test/search/id/267cd7b7-2d6a-474b-ade2-bf0453881f72

Allocating bugs to the responsible team can be tricks when there is only a functional description.

THE NEXT LEVEL

MOCK SERVICES

OFFER THIRD PARTY SERVICE

TEST

- CL

IENT X-CID=ABC123

HEADER

Testing can be hard when you’re integrating with many services. Especially unreliable integration systems can increase the development costs unexpectedly.

TEST 1 | MOCK ID=ABC123

BUSINESS TRACING BACKEND

THE NEXT LEVEL

MOCK SERVICESTesting can be hard when you’re integrating with many services. Especially unreliable integration systems can increase the development costs unexpectedly.

OFFER THIRD PARTY SERVICE

TEST

- CL

IENT MODE=READ

HEADER

X-CID=ABC123

TEST 1 | MOCK ID=ABC123

BUSINESS TRACING BACKEND

THE NEXT LEVEL

MOCK SERVICES

Change on the integrator side (Bug, Error, API Change)

Probably bug in our system

Run tests against real backend and against mock at the same time

Test against mock50 (100%) SUCCESS, 0 (0%) ERROR

Test against real backend40 (80%) SUCCESS, 10 (20%) ERROR

Test against real backend40 (80%) SUCCESS, 10 (20%) ERRORTest against mock40 (80%) SUCCESS, 10 (20%) ERROR

THE NEXT LEVEL

REPLAY CONVERSATIONS

OFFER THIRD PARTY SERVICE REPL

AY -

CLIEN

T

BUSINESS TRACING BACKEND

MODE=REPLAYHEADER

X-CID=ABC123

THE NEXT LEVEL

BUSINESS ANALYSISThe data stored by the business tracing system is the perfect foundation for extensive research and data analysis. Gathering the equivalent information from your existing data stores is expensive and may not even be possible.

THE NEXT LEVEL

BUSINESS ANALYSISThe data stored by the business tracing system is the perfect foundation for extensive research and data analysis. Gathering the equivalent information from your existing data stores is expensive and may not even be possible.

THE NEXT LEVEL

PREDICTIVE ANALYSISThe chance is to finally take the step to predictive analysis on your actual business data and processes.

Offer creation errors during the day per product id.

Offer creation errors during the day.

BUSINESS OBSERVABILITY IS AS IMPORTANT AS TECHNICAL TRACING

Q & MAYBE A ;-)

▸ Lukasz Pielak@lukaszpielaklukasz@trasier.com

▸ Frank Pfleger@frankpflegerfrank@trasier.com

▸ OpenTracing

▸ https://opentracing.io

▸ OpenCensus

▸ https://opencensus.io

▸ OpenTelemetry

▸ https://opentelemetry.io

▸ Zipkin

▸ https://zipkin.io

▸ Jaeger

▸ https://jaegertracing.io

▸ Trasier

▸ https://trasier.com

THANK YOU