bde sc4 hangout - hajira jabeen, general architecture

BDE Platform ArchitectureSC 4 Webinar27 June 2016

Agenda

◎ Platform architecture

◎User interfaces

Platform architecture

Platform architecture

DEPRECATED

Platform architecture

Platform architecture

Platform installation

◎ Manual installation guide

◎Using Docker Machineo On local machine (VirtualBox)o In the cloud (AWS, DigitalOcean, Azure)o Bare metal

Platform componentsBase Docker images:

◎Serves as a template for a (Big Data) technology

◎Easily extendable with custom algorithm/data

◎Automating as much as possible

◎May be opinionated

Platform components◎Published components

o Image repositories on GitHubo Automated builds on Docker Hubo Documentation on BDE Wikio docker-compose snippet in README

Developing a Pipeline

Big Data pipelines◎Run pipeline: $ docker-compose up

◎Orchestrator required for initialization processo Components may depend on each othero Components may require manual

intervention

Platform architecture

User Interfaces

User interfaces◎Workflow UIs

o Workflow Builder❖Build a workflow to configure the init daemon

serviceo Workflow Monitor

❖Monitor a pipeline's initialization process

◎Swarm UIo Deploy a pipeline on the platform

◎ Integration of existing UIso BDE CSS based on Materialize CSSo Integrator UI: present GUIs in a unified

interface

◎UI to follow-up status of the pipelineo Status of each componento Manually finish a step in the pipeline

◎ Integrated as component inthe application's docker-compose.yml

Pipeline monitor application

Thank you