Page 1
DIY (Do-it-yourself)
Testing
Page 2
Ruby Engineer on the drivers team.
Based in Berlin.Maintain the gems:
bson mongo mongoid bson_ext mongo_kerberos
@EmStolfo
Page 3
MongoDB DriversC C++ C# Java Node.js Perl PHP Python Ruby Scala
Page 4
MongoDB Drivers
Interface to the server.
10 official drivers available.
Many started as OS projects.
History
Page 5
No specifications.Divergent codebases
and features.
MongoDB DriversReality
Page 6
Server “interface” was not consistent.
MongoDB DriversReality
Page 7
Organizations can experience 1 interface.
Specifications guide design and provide documentation.
Rewrites with collective knowledge.
MongoDB DriversRevamp
Page 8
Server “interface” is a single product.
MongoDB DriversRevamp
Page 9
Specifications
Equally valuable to the drivers team as to the community.
Page 10
Server Discovery and Monitoring Spec
Logic required to make an application using MongoDB highly available.
Page 11
How to validate compliance?
Page 12
Requirements for validating compliance• Unit tests. • Integration tests. • Tests define as “data”. • Language-agnostic. • Reproducible scenarios.
Page 13
DIY Testing
Our custom tests: Unit / Integration Build your own test framework.
Page 14
define
Format - YAML Tests
Scenarios
Our approach:“DIY” testing
Page 15
benefitsFormat: YAML
• Describes data. • Can translate to actions. • Most languages can use a YAML parsing
library. • Driver authors write a reusable harness. • Changes in the spec can be
communicated via additional YAML tests or changes to existing ones.
Page 16
Test requirements• Unit tests. • Integration tests. • Defined as “data”. • Language-agnostic. • Reproducible scenarios.
Page 17
Unit test - YAMLsingle/direct_connection_standalone.yml
Page 18
Unit test - Test objectspec/support/server_discovery_and_monitoring.rb
Page 19
Unit test - specspec/mongo/server_discovery_and_monitoring_spec.rb
Page 20
Unit test - YAMLrs/discover_primary.yml
Page 21
Test requirements• Unit tests. • Integration tests. • Defined as “data”. • Language-agnostic. • Reproducible scenarios.
Mongo Orchestration
Page 22
Mongo Orchestration
Page 23
Define clusters using JSON.Manipulate clusters via RESTful API.
Mongo OrchestrationHTTP server providing REST interface to manage
multiple MongoDB processes on the same machine.
Implemented in python.
Page 24
What about Automation!?
• Is intended for testing. • Starts processes on one machine. • Allows fault injection. • Does not have a UI. • Has no protection against downtime. • Has one agent, so no resiliency. • Does not handle operation tasks.
Mongo Orchestration..!
Page 25
Create different cluster topologies.
Single
Mongo Orchestration
Replica Set
Sharded Cluster
Page 27
Single serverconfig options
Page 28
Replica Setconfig options
Page 29
Sharded Clusterconfig options
Page 30
Define different installations in a config file
mongo-orchestration.config
Page 31
Set up a Replica Sethttp://bit.ly/mo-usage
Page 32
Benefits of MO• Reproducible test scenarios. • Abstracts differing configuration
options across MongoDB versions. • Uniform interface regardless of OS. • Ability to define different locations
of MongoDB installations for multi-version testing.
Page 33
Multi-version testingMulti-topology testing
SSLAuthentication
Used internally
Drivers integration tests
Page 34
No need for cluster managerstest/tools/mongo_config.rb(cluster manager in 1.x ruby driver)
Page 36
Integration tests
Page 37
Integration testsTest driver behavior. Input: cluster state
Output: driver behavior
Mongo Orchestration
Page 38
Test harness (Ruby driver example)
PR #584 [WIP] Integration tests using Mongo Orchestration
Page 39
YAML integration testrs/connection/primary-not-available.yml (I)
Page 40
YAML integration testrs/connection/primary-not-available.yml (II)
Page 41
Harness structure
1. Resource (MO cluster) 2. Specification (parsed YAML) 3. Test (can be run)
PR #584 [WIP] Integration tests using Mongo Orchestration
Page 42
Resourcespec/support/mongo_orchestration/resource.rb
Page 43
Specificationspec/support/mongo_orchestration/spec.rb
Page 44
Testspec/support/mongo_orchestration/operation/client_operation.rb
Page 45
Restarted primary integration testrs/connection/primary-restarted.yml (I)
Page 46
Restarted primary integration testrs/connection/primary-restarted.yml (II)
Page 47
Restarted primary integration testrs/connection/primary-restarted.yml (III)
Page 48
How to build your own testing framework
Page 49
Ask yourself1. Unit or integration tests? 2. Need to be language-agnostic? 3. Directly related to specs or
documentation?
Page 50
Test framework entities
Resource Wrapper around
the external component.
Page 51
Test framework entities
Specification The parsed YAML.
Page 52
Test framework entities
State manipulator Processor of phases to get
to a certain state.
Page 53
Test framework entities
Tests Can be executed.
Result is ‘pass’ or ‘fail’.
Page 54
Make the test runner as generic as possible.
Page 55
Install Mongo Orchestration
Page 57
Resources
http://bit.ly/ruby-kaigi-diy-testing@EmStolfo