defensive programming in javascript and node.js

Defensive Programming

in Javascript & node.js

Wednesday, May 29, 13

INTRODUCTION

Ruben Tan Long Zheng 陈龙正

VP of Engineering, OnApp CDN KL

Lead Engineer, 40 Square Sdn Bhd

Javascript > 5 years

@roguejs

Organizer of Nodehack KL


OVERVIEW

Dependency Awareness

Javascript Mastery

Methodology Improvements


SEASON 1

DEPENDENCY AWARENESS


Internal dependencies

Libraries

require(), include()

External dependencies

Services, files, databases, etc

socket.connect(), db.open()

DEPENDENCY TYPES


NEVER ASSUME!

Never assume a dependency is reliable!

var db = require(‘database’);

db.open();

db.write(‘foo bar’, function (err, data) { // ... do something ... });


NEVER ASSUME!


db.open();

db.write(‘foo bar’, function (err, data) { // ... do something ... });

What if this failed?will write() throw an error? will open() throwan exception?


NEVER ASSUME!


db.open(function (err) { db.write(‘mr-big’, bigData, function (err, data) { // ... unrelated logic db.close(); });

db.read(‘foo2’, function (err, data) { // ... some work done });});

Accidents happen...


NEVER ASSUME!


db.open(function (err) { db.write(‘mr-big’, bigData, function (err, data) { // ... unrelated logic db.close(); });

db.read(‘foo2’, function (err, data) { // ... some work done });});

close() might affect read()


A MORE COMPLEX EXAMPLE...


VIDEO STREAMING SERVICE

VideoStreamer

Origin

StatsLogger

VODClient

User

Accounting

UploadLiveClient

UserStream

Stream

LogReport

Render

Render


VIDEO STREAMING SERVICE

VideoStreamer

Origin

StatsLogger

VODClient

User

Accounting

UploadLiveClient

UserStream

Stream

LogReport

Render

Render1

2

3

4

5

6

7

89

1011


DEPENDENCY AWARENESS

What can fail, WILL FAIL!

Never assume a dependency is reliable!

Contingency plans - failover, redundancy, fail-fast, etc

Pro-active monitoring

Load test, stress test, chaos monkey, etc

Remember, what can fail, WILL FAIL!


SEASON 2

JAVASCRIPT MASTERY


JAVASCRIPT MASTERY

Code Execution Order

Sanitization & Validation

Scope

Control Flow


I KNOW CODE-FU!


EXECUTION ORDER

var mq = require(‘mq’);

mq.conn(...);

mq.on(‘ready’, function () { mq.send(‘batman’);

mq.on(‘message’, function (msg) { console.log(msg); mq.close(); });});

mq is never closed!

send() executes before on()


DOIN’ IT RIGHT!

var mq = require(‘mq’);

mq.conn(...);

mq.on(‘ready’, function () { mq.on(‘message’, function (msg) { console.log(msg); mq.close(); });

mq.send(‘batman’);});

Swap places


SANITIZATION & VALIDATION

function foodForKittens(num) { return num * 10;}

foodForKittens();

num is not validated, is undefined

this will fail!


TOO SIMPLE?


SANITIZATION & VALIDATION

var db = require(‘database’);var conn = db.open(...);

function writeToDb(conn, cb) { conn.write(bigData, function (err, res) { if (err) { cb(err); return; }

cb(null, res); });});

writeToDb(conn, ghostCallback);






what if open() returned undefined?

this will throw an exception!






What if ghostCallback is undefined?

These will fail too!


DOIN’ IT RIGHT!var db = require(‘database’);var conn = db.open(...);

function writeToDb(conn, cb) { if (typeof conn !== ‘object’) { // ... handle error ... }

if (typeof cb !== ‘function’) { // ... handle error ... }

conn.write(bigData, function (err, res) { if (err) { cb(err); return; }



Validate your input,especially when theyinvolve functions ormethods that you need toinvoke in your code.

These are not the time tofail-fast!


DON’T GO OVERBOARD...

Validate only necessary parameters

Method invocations (anObject.method())

Function invocations (aFunction())

Have a proper error/exception handling policy

Validate for correctness, not existence

Correctness: typeof a === ‘object’

Existence: a !== undefined


SCOPE AWARENESS

Plagues most callback-based code

Bad practice leads to costly debugging waste

New JS programmers not aware of scoping

JS scoping is a simple but weird thing (to non-JS programmers)


SCOPE!!!

var a = ‘outside’;

if (true) { var a = ‘inside’; console.log(a);}

console.log(a);

What is the output?

> node test.jsinsideinside


SCOPE!!!

Non-JS programmers:

a inside the if block is “inside”

a outside the if block is “outside”

JS programmers:

they are both “inside”

JS scope by function


SCOPE CHAINS!!!var avar = 1;

(function outer1() { var avar = 2; (function inner1() { var avar = 3; console.log(avar); // outputs 3 })();

(function inner2() { console.log(avar); // outputs 2 })();})();

(function outer2() { (function inner3() { console.log(avar); // outputs 1 })();})();

inner1()

local - found!

inner2()

local - nope

outer1() - found!

inner3()

local - nope

outer2() - nope

global - found!


HOISTING VARIABLESfunction () { for (var i = 0; i < 10; i++) { for (var j = 0; j < 10; j++) { // ... do something } }}

function () { var i, j; // now the scope is clear for i & j

for (i = 0; i < 10; i++) { for (j = 0; j < 10; j++) { // ... do something } }}

Below is far clearer what individual variable scopes are:


CONTROL FLOW

Node.js’ async nature makes it unintuitive to predict control flow

I <3 async (github.com/caolan/async)

Control flow is ugly. Welcome to Javascript.

Async will save your life. Use it.


CONTROL FLOWvar fs;

fs = require(‘fs’);

fs.readFile(‘./myfile.txt’, function (err, data) { if (err) { console.log(err); return; }

fs.writeFile(‘./myfile2.txt’, data, function (err) { if (err) { console.log(err); return; }

// ... do stuff ... });})


CONTROL FLOW

Callback hell!

Step 1

Step 2

Step 3

Step 4

Step 5


mod.step1(function () { mod.step2(function () { mod.step3(function () { mod.step4(function () { mod.step5(function () { // ... too many levels ... }); }); } });});


CONTROL FLOW

var async, fs;

async = require(‘async’);fs = require(‘fs’);

async.waterfall([ function step1(callback) { fs.readFile(‘./myfile.txt’, callback); },

function step2(data, callback) { fs.writeFile(‘./myfile2.txt’, data, callback); }], function (err) { // ... execute something in the end ...});


SEASON 3

METHODOLOGY IMPROVEMENTS


GOLDEN RULES

Golden Rules of Defensive Programming

Proper error handling policy

Intelligent logging

Design for failure


ERROR HANDLING

Never, ever HIDE errors

> node app.js 2>&1 /dev/null

ob.callback(function (err, data) { if (err) {} console.log(data);});

socket.on(‘error’, function () {});


ERROR HANDLING

I WILL FIND YOU

AND I WILL CRASH YOU


ERROR HANDLING

Standardize error handling in the app

Log to error DB

Output to error file

Output error to a stream

Use a logging library

Ask a leprechaun to manage it

etc


LOGGING

How do you feel if your “log” looks like this?

> tail -f error.log[12:01:55] ERROR - General error detected[12:01:56] ERROR - General error detected[12:01:57] ERROR - General error detected[12:01:58] ERROR - General error detected[12:01:59] ERROR - General error detected[12:02:00] ERROR - General error detected[12:02:01] ERROR - General error detected


LOGGING


LOGGING

Logs are the first place you go to find out what happened

Standardize a log location for each app

Make logs easy to access for developers


DESIGN FOR FAILURE

Common steps to designing software:

1 - what should it do?

2 - how do I do it?

3 - how do I deploy?

4 - done


DESIGN FOR FAILURE

Proper steps in defensive programming:

1 - what should it do?

2 - how many ways can it fail?

3 - how do I know when it fails?

4 - how do I prevent it from failing?

5 - write code accordingly


DESIGN FOR FAILURE

Nothing is reliable

TCP can fail

Network can go down

Servers can run out of memory

Cows might fly through the sky crashing into your datacenter and flooding the server rooms with milk and destroying everything


DESIGN FOR FAILURE

Designing for failure mindset & methodologies:

Identify SPOF (single point of failures)

Redundancy, failover, monitoring

Fail-fast, start-fast

Persist important data

Reliability & Consistency > Speed

Code is liability


~ The End ~


defensive programming in javascript and node.js

Technology