ruslan shevchenko - property based testing
TRANSCRIPT
WHAT IS IT ?
PROPERTY BASED TESTING
Eng: property властивість свойство
Other : example приклад пример
for all x: Int x < (x+1)
3 < (3 + 1)
SHOW ME THE CODE/SCALA
import org.scalacheck.Properties import org.scalacheck.Prop.forAll
object IntSpecification extends Properties("Int") { property("x < x+1") = forAll{ (x:Int) => x < x+1 } }
SHOW ME THE CODE/SCALA
import org.scalacheck.Properties import org.scalacheck.Prop.forAll
object IntSpecification extends Properties("Int") { property(“x < x+1") = forAll{ (x:Int) => x < x+1 } }
[info] ! Int.x > x+1: Falsified after 3 passed tests.[info] > ARG_0: 2147483647[info] Failed: Total 1, Failed 1, Errors 0, Passed 0[error] Failed tests:[error] IntSpecification[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
SHOW ME THE CODE/JAVA
…..
@RunWith(JUnitQuickcheck.class) public class IntPropertyTest {
@Property public void xLessIncr(int x) { assert(x < x+1); }
}
SHOW ME THE CODE/JAVA
…..
@RunWith(JUnitQuickcheck.class) public class IntPropertyTest {
@Property public void xLessIncr(int x) { assert(x < x+1); }
} [info] Running com.github.rssh.IntPropertyTestTests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.75 sec
// Problem not found. // Generation of random values are differ ;)
SHOW ME THE CODE/JAVASCRIPT
var jsc = require("jsverify")
describe('double ceiling', function() { it ('x < x + 1e-15',function() { expect( jsc.checkForall(jsc.number,function(x){ return x < x+1e-15 }) ).toBe(true) }); });
// explore the limits of the floating point ceiling.
SHOW ME THE CODE/JAVASCRIPT
var jsc = require("jsverify")
describe('double ceiling', function() { it ('x < x + 1e-15',function() { expect( jsc.checkForall(jsc.number,function(x){ return x < x+1e-15 }) ).toBe(true) }); });
// explore the limits of the floating point ceiling.
Failed after 8 tests and 1 shrinks. rngState: 0879d03f2e184b6c5a; Counterexample: 19.7693584933877; [ 19.7693584933877 ]FFailures: 1) double ceiling x < x + 1e-151.1) Expected Object({ counterexample: [ 19.7693584933877 ], counterexamplestr: '19.7693584933877', shrinks: 1, exc: false, tests: 8, rngState: '0879d03f2e184b6c5a' }) to be true.
MONKEY TESTING
PROPERTY-BASED TESTING
We generate random values. Pass one to our function or service to test Expect, that result of this function satisfy some properties
laws (classical property-based testing); crash
The Infinite Monkey Theorem
HISTORY
QuickCheck. Haskell. 1999
• https://en.wikipedia.org/wiki/QuickCheck • http://www.cse.chalmers.se/~rjmh/QuickCheck/
John Hughes founded QuivQ • commercial version of QuickCheck for Erlang • extensions/interfaces for
• C/C++ • WebServices • Industry specific standards … (appl. Volvo cars)
IMPLEMENTATIONS FOR POPULAR LANGUAGESScala: scalacheck
http://www.scalacheck.org Java: junit-quickcheck
http://pholser.github.io/junit-quickcheck/ JavaScript: jsverify
http://jsverify.github.io/ Python: Hypothesis
http://hypothesis.works/ PHP: eris; phpquickchek
https://github.com/giorgiosironi/eris https://github.com/steos/php-quickcheck
…………………… practically any language.
TYPICAL USAGE
Thinks and describe laws (dependencies between input & output)
Tune input generators Check such properties in output
Demos: Programming tasks (sorting, serialization) Form validation State checks
LET’S CHECK SORTED ALGORITHM
def sort(x:IndexedSeq[T]):IndexedSeq[T]
invariant: length
definition of sorted:
idempotency:
LET’S CHECK SORTED ALGORITHM
def sort(x:Vector[T]): Vector[T]
invariant: length
property(“sorted array must the same size of origin”) = forAll{ (x: Vector[Int]) => x.size == sort(x).size }
LET’S CHECK SORTED ALGORITHM
We need to generate: • n - max number of entries • sorted vector of the length n • i < n • j: i < j < n
CUSTOM GENERATOR
val orderedVectorGen = for { n <- Gen.choose(2,100) x <- Gen.containerOfN[Vector,Int] i <- Gen.choose(0,n-2) j <- Gen.choose(i,n-1) } yield (n, sort(x), i, j)
CUSTOM GENERATOR
val orderedVectorGen = for { n <- Gen.choose(2,100) x <- Gen.containerOfN[Vector,Int] i <- Gen.choose(0,n-2) j <- Gen.choose(i,n-1) } yield (n, sort(x), i, j)
property(“sorted array must reflect ordered indexes”) = forAll(orderedVectorGen){ case (n, x, i, j) => x(i) <= x(j) }
property test can serve as documentation for input and output.
LAWS
Ordering[T]: • reflexivity:
forall{ x:T => x <= x} • anitsimmetry:
forall{ x:T, y:T => x <= y & y <= x ==> x==y }
• transitivity: • forall{ x:T, y:T, z: T =>
x <= y & y <= z ==> x <= z }
// Laws = set of properties. // Can be used as specification for type argument.
EXAMPLE: DATA VALIDATION
case class Subsriber(id:Long, // must be unique firstName: String, // max 20 sym, no spaces. lastName:String // max 20 sym, no space )
val correctSubscriberGen = for { id <- arbitrary[Long] firstName <- Gen.alphaStr suchThat (_.length <= FIRST_NAME_LEN) lastName <- Gen.alphaStr suchThat (_.length <= LAST_NAME_LEN) } yield Subscriber(id,firstName, lastName)
val incorrectSubscriberGen = for { id <- arbitrary[Long] firstName <- incorrectNameGen(FIRST_NAME_LEN) ………
GENERAL QUESTIONS
Is property-based testing useful itself ? — definitely yes. — force to think hight-level
Are we steel need example-based testing ? — probably yes.
• What to manual construct thing, which • explore some type of bug.
• Dimension Problem. • N probes can be very tiny part.
QUESTIONS ?
THANKS FOR LISTENING
Full example code is available on github: https://github.com/rssh/property-based-testing-talk
@rssh1 Ruslan Shevchenko <[email protected]>