one size doesn’t fit all: federated data science using ampool - by nitin lamba, suhas gogate,...
TRANSCRIPT
Prepared for: BDAM
OneSizedoesn’tfitallMakingacaseforFederatedDataScience
NitinLamba&Suhas Gogate
(RiyaMakhijani,Ruta Joshi,ShantanuTulshibagwale)
Prepared for: BDAM
DataScienceisamulti-frameworkenvironment
! DataScientists! ! Business
Analysts!! DataEngineers!
language
tool
Prepared for: BDAM
DataProcessingisaniterative,multi-step process
…" " "
# #
&
#
ModelExplore ServeFlatten
Prepared for: BDAM
ModelExplore ServeFlatten
" " "
# #
&
#
…
Whatiffastobjectaccess isavailableacrossstages?
Prepared for: BDAM
ModelExplore ServeFlatten
" " "
# #
#
…&
Reducetimetoinsights,buildreal-timemodels
✅ Object-baseddataexchange
✅ NoDataCopies
✅ End-to-endspeedup
✅ IncreasedConcurrency
Prepared for: BDAM
Ampool isamemory-orientedActiveDataStore...
AprimarystoreforALLdataprocessing
StoreALLactivedata&updateit,asreqd.
Servesdataconcurrentlytomultiplestages&tenants
DataPersistence
Prepared for: BDAM
...offerschoicefordifferentanalyticalneeds...
'
( ) ) …
MPPDatabaseLocalFileSystem
Software-DefinedMemory
*Experimental; waiting feedback fromdesignpartners
…
…
Prepared for: BDAM
…anddeliversvaluetoall typesofdatausers
Nochangeinapplicationlogic
Makeconfig.changesonly
Nochange inexistingusertools
Getmemoryspeeds
NohassledeploymentUsecurrentmgmt.tools
! DataArchitectDataEngineers!
! BusinessAnalystsDataScientists!
! DataAdminsInfra/SysAdmins!
Prepared for: BDAM
In-MemoryDistributedSys
Low-latencyComms.
Key-ValueStore
FunctionPushdown
+
HighThroughput
TableStore
NativeInterfacePluggableStoreManager
JavaAPI
MASH(CLIExt)
JavaAPI
Itisbuiltonaproven in-memorytechnology
SmartDataTiering
MatureEventModel
TunableConsistency
Metadata/Catalog
SecurityAuthZ
Prepared for: BDAM
CardHolder
Init PurchasePay bill
Merchant
ScansCardProcessesTx
ABCBankOffersCardsServesBills
AnomalyDetection:Anillustrativeuse-case…
TransactionLogs
Tx Tx Tx…
Tx Tx Tx …
Acquirer
IntermediatesTx
Issuer
ProcessrequestMonitorsactivityBilling
Events
Network
*++
AggregatesTx
1 2 34 5 67 8 9
0
Prepared for: BDAM
…with Kafka,Spark,R &Ampool…
IngestRepattheIssuingBank Financial
TransactionsStream&BatchProcessing
Monitoring/App
Tx ValidateUserProfile$impact
Explore&Model
Prepared for: BDAM
RStudio
…supportingmultipleanalyticaltools
KafkaDataGenerator
Ingest Streaming Explore/Model
CrossValid Tx Log Outliers
User/AppData
FinTx Logs
SparkStreaming
Fin_Txn
Tx Confirm.UserUpdates
Dstream RDD
In_DF
Model
SparkBatch/ML
Batch_DF
Batch
Prepared for: BDAM
Demo
Prepared for: BDAM
Questions?
, /company/ampool-inc- -. /AmpoolIO@AmpoolIO/ www.ampool.io