introduction to rfx for backend developer
Post on 14-Jul-2015
1.085 Views
Preview:
TRANSCRIPT
Introduction to RFX for Backend Developer
Reactive Function (X) the Open Source Framework for solving Fast Data Problem
and reacting to the World with Deep Learning
By Triều Nguyễn, the creator of RFX http://mc2ads.com (Reactive Big Data Lab)
λ(x)
2008: Java Developer, develop Social Trading Network for a startup (Yopco)2011: joined FPT Online, software engineer, worked in banbe.net social network and VnExpress Mobile Restful API2012: backend engineer at Greengar Studios12/2012 to now - back to FPT Online, lead engineer, developed new version Data Analytics Platform (ad-network eclick.vn and VnExpress News)
Introduction about myself
1. What is Rfx ?2. Inception and Ideas3. How Rfx was born4. Why is Rfx ?
a. from big picture view (business)b. from business viewc. from specific problems
5. Concepts and architecture: The BIG picture6. Coding and tutorials from practical problems7. Resources for self-studying
Contents of this talk
● A framework for reactive real-time big fast data● A collection of Open Source Tools ● The mission of RFX
→ “BUILD digital data-driven brain for every company in the World”
What is RFX or Reactive Function X ?
INCEPTION and Ideas
Ideas when I was student, internship at DRD, non-profit OrganizationMore info at http://activefunctor.blogspot.com
http://singularityhub.com/2014/04/20/new-imaging-method-shows-young-neurons-making-connections-exchanging-information/
Why Rfx ?
● Ideas since 2007 (from Haskell and Actor model theory)● R&D and Deployed in Production since 2013
● Open Source: Apache License, Version 2.0 ● Full Stack: Front-end and Back-end● Apply Agile for Analytics and Data Science● Apply Reactive Lambda Architecture
● Really fast and near-real-time processing● Tested with 1.000.000 logs / second (1 million in 1 second)
● Simple development model for big data developer
Domain (in business) where Rfx can be used
● real-time data analytics for digital marketing, advertising● hospital systems● personal banking system● financial institution to detect frauds● manufacturing plant● airline systems● online trading system● emergency control system● manufacturing plant management system● road tolling system detects● social networking site
Paper: http://vialab.science.uoit.ca/textvis2011/papers/textvis%202011-rohrdantz.pdf
Problem: How to monitor Mobile Web Performance and react to slow response time
http://sixrevisions.com/mobile/pay-attention-to-mobile-web-performance
Luggage management system, events are produced by the check-in process and by the various radio-frequency identification (RFID) readers, which emit events about the movement of the luggage in the system. The events generated by the event processing system are consumed by the luggage control system itself, by airport staff, or even by the passengers themselves.
Problem: Monitoring sensor data and real-time security checking
ActorUser, Mobile, Browser, ...
Reactive Lambda Architecture
System Rfx-Topology
data + context + metadata
useful (data + relationship)
DatabaseNoSQL
1. Actor → System 2. System → Database3. Database → System4. System → Actor
Concepts● Each user, who uses the services and creates data, is the
actor in system
● Actor is the source of all events (aka: logs), (click, reading
news, sending message to friends, playing games, ...)
● Functor (aka: neuron) is a computing object, used for
storing, processing data and emitting results to subscribed
functors
● Topology is the directed graph, define how functors that
are connected with stream data and process data
There are 3 demos, from simple to advanced user story
User story 1: Counting Real-time URL PageviewUser story 2: Monitoring Social Media StatisticsUser story 3: Social Ranking for Recommendation Engine
User Story
Domain problem: Reactive Real-time MarketingUser story’s details:1. User does read news from a website
→ tracking user activities (pageview, time on site)2. User does login with Facebook Account3. User clicks on like Facebook button
→ tracking what user liked, commented4. The marketer/data analyst should see the trending most
read article in real-time● → Personalized articles for the reader ● → Native advertising in real-time
Demo user story 1: Counting Real-time URL Pageview
Input:1. The pageview logs from HTTP
Output:1. The total number of page-view2. The total number of page-view per hour3. The total number of page-view per minute4. The total number of page-view per second5. The total number of page-view for URL
Demo user story 2: Monitoring Social Media Statistics
Input:1. The pageview logs from HTTP
Output:The social media statistics from:1. Facebook: Like, Share, Comment2. Twitter: Tweet Count3. LinkedIn: Share Count4. Geolocation heat-map report
Demo user story 3: Social Ranking for Recommendation EngineInput:● Data: the URL of article● Context: where (User's Location), when (time visit), from
where (referer url)● Metadata: keywords, category of article
Output:Real-time Statistics about pageview, social media statistics (Share, Like, Comment), recommended articles
The list of articles are ranked by:● most liked and same category● most viewed and same category● most liked, same category and near user's location
Reference ResourcesMain website for Rfx: http://www.mc2ads.comIdeas:
● http://journal.frontiersin.org/Journal/10.3389/fninf.2010.00112/full● http://singularityhub.com/2014/04/20/new-imaging-method-shows-young-neurons-making-
connections-exchanging-information● http://en.wikipedia.org/wiki/Actor_model_theory● http://java.dzone.com/articles/introduction-event-processing● http://www.technologyreview.com/featuredstory/526501/brain-mapping● http://www.technologyreview.com/featuredstory/513696/deep-learning
Apache Storm: http://storm.incubator.apache.orgApache Kafka: http://kafka.apache.orgIn-memory NoSQL: http://redis.ioDeep Learning for Java: http://deeplearning4j.orgDistributed processing with Actor Model: http://akka.ioPapers: ● Real-Time Visualization of Streaming Text Data● Hypernetworks for the Science of. Complex Systems
Main Blogs: http://www.mc2ads.org
The end and thank youhttps://github.com/mc2ads/rfx
http://www.mc2ads.org
λ(x)
top related