divide and conquer – microservices with node.js
TRANSCRIPT
Basti• Sebastian Springer
• from Munich
• work @ MaibornWolff GmbH
• https://github.com/sspringer82
• @basti_springer
• JavaScript Developer
AgendaHTTP Microservice with Express
• Routing • Structure • Database • Logging • Tests • Container
Commandbased Microservices with Seneca
• Patterns • Plugins • Queueing
Rainer Sturm / pixelio.de
MicroservicesSmall
Scaleable
Resilient
Independent Testable
Focused
JavaScript is nearly everywhere. There are a lot of OSS Modules. Node.js was built for the web
Node.js supports most of the protocols, systems and databases.
The platform is lightweight.
Example
A microservice that delivers articles for a web shop.
M. Hermsdorf / pixelio.de
Webshop
Client API
Articles
Payment
Users
DB
DB
DB
Auth
Logger
List of articlesRouter
Controller
Model
Datenbank
Logger
Packages
S. Hofschlaeger / pixelio.de
Express.js
Express is a lightweight (< 2MB) web application framework for Node.js.
Express supports HTTP and HTTPS in version 1 and 2. Express handles routing within a web application for you.
Additionally it provides you with a plugin interface (Middleware).
Installation
yarn init
yarn add express
Structure of a Microservice. ├── db │ ├── config.js │ └── db.sqlite3 ├── package.json ├── spec └── src ├── config.js ├── controller.js ├── index.js ├── logger.js ├── model.js └── router.js
Entry pointindex.js
Entry point
const express = require('express'); const router = require('./router');
const app = express();
router(app);
app.listen(8080, () => { console.log('Service is listening to http://localhost:8080'); });
Entry point
In this file all global packages are included and the app is configured.
Additional features should be put in separate files.
Routerrouter.js
Router
const controller = require('./controller');
module.exports = (app) => { app.get('/article', controller.getAll); app.get('/article/:id', controller.getOne); app.post('/article', controller.create); app.put('/article', controller.update); app.delete('/article', controller.remove); }
Router
The router defines all available routes of a microservice.
A route consists of a HTTP method and a path. By adding variables to routes they become more flexible.
To keep the code of your router readable, you should put the routing callbacks in a separate file.
Controllercontroller.js
Controllerconst model = require('./model');
module.exports = { getAll(req, res) { res.json( model.getAll() ); }, getOne(req, res) {}, create(req, res) {}, update(req, res) {}, remove(req, res) {} }
ControllerThe controller holds your routing callback functions. It extracts information from the requests and hands the
information over to the model.
To access variables of your routes, you can use the req.params object.
To deal with information within the request body, you should install the body-parser middleware. It introduces the req.body
object which represents the message body.
Modelmodel.js
Model
The model contains the business logic of your application. It also controls access to the database.
Usually every microservice has its own database, which increases independence between services.
ModelNode.js supports all common databases such as OracleDB,
MySQL, Redis, MongoDB.
To access your database, you have to install a database driver first.
yarn add sqlite3
To further simplify the access, you can use various ORMs or ODMs
yarn add orm
ORM
An ORM handles CRUD operations for you. It already implements all common use cases. You just have to
configure it with the structure of your database.
The ORM also takes care of some security features such as escaping.
ORMconst express = require('express'); const orm = require('orm'); const router = require('./router'); const dbConfig = require('./db/config.js'); const {dbPath} = require('./config.js');
const app = express();
app.use(orm.express(dbPath, dbConfig));
router(app);
app.listen(8080, () => { console.log('Service is listening to http://localhost:8080'); });
ORM
module.exports = { define(db, models, next) { models.articles = db.define('articles', { id: Number, title: String, price: Number }); next(); } }
Model
Besides encapsulating database communication your model takes care of additional tasks such as validation of user input
and various calculations.
Most of these operations are asynchronous. To provide a clean API you should think about using promises, async
functions or streams instead of plain callbacks.
async getAll(req, res) { try { const articles = await model.getAll(req); res.json(articles); } catch (e) { res.status(500).send(e); } }
getAll(req) { return new Promise((resolve, reject) => { req.models.articles.all((err, results) => { if (err) { reject(err); } else { resolve(results); } }); }); }
controller
model
Logging
Tim Reckmann / pixelio.de
LoggingWithin your microservice errors might occur at any time. Of course you should be prepared to catch and handle them.
But you should also keep a log of errors somewhere.
A logger should not be a fixed part of a certain microservice. logging is a shared service which is available for all services
in your application.
Centralised logging provides some advantages over local logging such as scalability and an improved maintainability.
Logging
You can use log4js for remote logging in your application. This library provides a plugin interface for external appenders which support various log targets such as files or logstash.
For remote logging you could use the logstash appender and a centralised logstash server.
Loggingconst log4js = require('log4js'); log4js.configure({ appenders: [{ "host": "127.0.0.1", "port": 10001, "type": "logstashUDP", "logType": "database", "layout": { "type": "pattern", "pattern": "%m" }, "category": "database" }], categories: { default: { appenders: ['database'], level: 'error' } } });
module.exports = log4js;
Loggingconst log4js = require('./logger');
module.exports = { getAll(req) { return new Promise((resolve, reject) => { req.models.articles.all((err, results) => { if (err) { reject(err); log4js.getLogger('database').error(err); } else { resolve(results); } }); }); } }
TestsQuality and stability are two very important aspects of
microservices. To ensure this, you need the ability to automatically test your services.
There are two levels of tests for your microservice application: unittests for smaller units of code and integration tests for
external interfaces.
For unittests you can use Jasmine as a framework. With Frisby.js you can test your interfaces.
Unittests
yarn add jasmine
node_modules/.bin/jasmine init
Unittestsconst model = require('../model');
describe('model', () => { it('should handle a database error correctly', () => { const req = { models: { articles: { all: (cb) => {cb('error', null);} } } }
model.getAll(req).catch((e) => { expect(e).toBe('error'); }); }) });
Unittests
To execute your tests, just issue the command npx jasmine.
You have two variants for organising your tests. You can either store them in a separate directory, usually named “spec” or
you put your tests where your source code is located.
Mockery
In your microservice application you have to deal with a lot of dependencies. They are resolved via the node module
system. Mockery is a tool that helps you with dealing with dependencies. Mockery replaces libraries with test doubles.
yarn add -D mockery
Mockeryconst mockery = require('mockery');
beforeEach(() => { mockery.enable();
const fsMock = { stat: function (path, cb) {...} }; mockery.registerMock('fs', fsMock); });
afterEach(() => { mockery.disable(); });
Integration tests
Frisby.js is a library for testing REST interfaces. Frisby is an extension for Jasmine. So you don’t have to learn an additonal
technology. In order to run Frisby.js, you have to install jasmine-node.
yarn add -D frisby jasmine-node
Integration testsrequire('jasmine-core'); var frisby = require('frisby'); frisby.create('Get all the articles') .get('http://localhost:8080/article') .expectStatus(200) .expectHeaderContains('content-type', 'application/json') .expectJSON('0', { id: function (val) { expect(val).toBe(1);}, title: 'Mannesmann Schlosserhammer', price: 7 }) .toss();
PM2
Node.js is single threaded. Basically that’s not a problem because of its nonblocking I/O nature. To optimally use all
available resources, you can use the child_process module.
A more convenient way of locally scaling your application is using PM2.
yarn add pm2
PM2pm2 start app.js
pm2 start app.js -i 4
pm2 reload all
pm2 scale <app-name> 10
pm2 list
pm2 stop
pm2 delete
Docker
All your services run in independent, self contained containers.
You can start as many instances of a certain container as you need.
To extend the features of your application, you simply add additional services by starting containers.
DockerfileFROM node:7.10
# Create app directory RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
# Install app dependencies COPY package.json /usr/src/app/ RUN yarn install
# Bundle app source COPY . /usr/src/app
EXPOSE 8080 CMD [ "yarn", "start" ]
Docker
docker build -t basti/microservice .
docker run -p 8080:8080 -d basti/microservice
API Gateway
Each service of your application has to focus on its purpose. In order to accomplish this, you have to centralise certain
services. A typical example for a central service is authentication.
An API Gateway forwards authorised requests to the services of your application, receives the answers and forwards them to
the client.
Seneca
Seneca follows a completely different approach. All services of your application communicate via messages and become
independent of the transport layer.
Installation
yarn add seneca
Service Definition
const seneca = require('seneca')(); seneca.add({role: 'math', cmd: 'sum'}, controller.getAll);
The first argument of the add method is the pattern, which describes the service. You are free to define the pattern as you want. A common best practice is to define a role and a
command. The second argument is an action, a function that handles incoming requests.
Service Handlerasync getAll(msg, reply) { try { const articles = await model.getAll(req); reply(null, JSON.stringify(articles)); } catch (e) { reply(e); } }
Similar to express a service handler receives the representation of a request. It also gets a reply function. To
create a response, you call the reply function with an error object and the response body.
Service Callseneca.act({role: 'article', cmd: 'get'}, (err, result) => { if (err) { return console.error(err); } console.log(result); });
With seneca.act you consume a microservice. You supply the object representation of a message and a callback function. With the object, you trigger the service. As soon as the answer
of the service is available, the callback is executed.
Patterns
You can create multiple patterns of the same type. If a service call matches multiple patterns, the most specific is
used.
You can use this behaviour to implement versioning of interfaces.
Plugins
Klicker / pixelio.de
Plugins
A plugin is a collection of patterns. There are multiple sources of plugins: built-in, your own plugins and 3rd party
plugins.
Plugins are used for logging or debugging.
Plugins
You organise your patterns with the use method. The name of the function is used for logging purposes.
You can pass options to your plugin to further configure it. The init pattern is used instead of a constructor.
const seneca = require('seneca')();
function articles(options) { this.add({role:'article',cmd:'get'}, controller.getAll); }
seneca.use(articles);
Pluginsconst seneca = require('seneca')();
function articles(options) { this.add({role:'article',cmd:'get'}, controller.getAll); this.wrap({role:'article'}, controller.verify); }
seneca.use(articles);
The wrap method defines features that are used by multiple patterns. With this.prior(msg, respond) the original service
can be called.
Client/Server
cre8tive / pixelio.de
Serverfunction articles(options) { this.add({role:'article',cmd:'get'}, controller.getAll); this.wrap({role:'article'}, controller.verify); }
require('seneca')() .use(articles) .listen(8080)
Just like in the ordinary web server the listen method binds the server to the port 8080. Your service can then be called by
the browser or another server:
http://localhost:8080/act?role=article&cmd=get
Client
require('seneca')() .client(8080) .act({role: 'article', cmd: 'get'}, console.log);
The client method lets you connect to a seneca microservice.
Change the transport// client seneca.client({ type: 'tcp', port: 8080 });
// server seneca.listen({ type: 'tcp', port: 8080 });
By providing the type ‘tcp’ the TCP protocol is used instead of HTTP as a communication protocol.
Integration in Express
Integration in Expressconst SenecaWeb = require('seneca-web'); const Express = require('express'); const router = new Express.Router(); const seneca = require('seneca')();
const app = Express();
const senecaWebConfig = { context: router, adapter: require('seneca-web-adapter-express'), options: { parseBody: false } }
app.use( require('body-parser').json() ) .use( router ) .listen(8080);
seneca.use(SenecaWeb, senecaWebConfig ) .use('api') .client( { type:'tcp', pin:'role:article' } );
Integration in Express
With this configuration seneca adds routes to your Express application.
You have to adopt your seneca patterns a little bit. All routes defined with seneca.act('role:web', {routes: routes}) are added
as Express routes.
Via the path-pattern a corresponding matching is done.
Communication via a queue
S. Hofschlaeger / pixelio.de
Queue
You can use various 3rd party plugins to communicate via a message queue instead of sending messages over network.
The main advantage of using a Queue is decoupling client and server. Your system becomes more robust.
Queue
yarn add seneca-servicebus-transport
Queuerequire('seneca')() .use('seneca-servicebus-transport') .use(articles) .listen({ type: 'servicebus', connection_string: '...' });
require('seneca')() .use('seneca-servicebus-transport') .client({ type: 'servicebus', connection_string: '...' }) .act({role: 'article', cmd: 'get'}, console.log);
Server
Client
Microservices, the silver bullet?
Microservices might be a good solution for certain problems but definitely not for all problems in web development.
Microservices introduce an increased complexity to your application. There is a lot of Node.js packages dealing with
this problem. Some of these packages are outdated or of bad quality.
Take a look at GitHub statistics and npmjs.com if it fits your needs.
Questions
Rainer Sturm / pixelio.de
CONTACT
Sebastian Springer [email protected]
MaibornWolff GmbH Theresienhöhe 13 80339 München
@basti_springer
https://github.com/sspringer82