QuickCheck:A Lightweight Tool for Random Testing

of Haskell ProgramsKoen Claessen, John Hughes ICFP ‘00

CMSC 737 Software TestingFang Cheng javran@cs.umd.edu

Prelude - Haskell

● Functional Programming○ Functions as First-class Citizens○ More Controls over Side-effects○ Implicit Control Flow

● Strong Static Typing○ Reasoning about Types○ Rule out “bad programs” at compile time○ If it compiles, it works!

Prelude - Haskell

● Functional Programming○ First-class citizen

■ g . f = λx -> g (f x)○ Side effects

■ nextRandom :: () -> Int■ nextRandom :: Seed -> (Int, Seed)

○ Implicit control flow■ sum . map (^2) . filter (< 4) $ [1..10]

Prelude - Haskell

● Strong Static Typing○ Reasoning about Types

■ f1 :: ∀a . a -> a■ f2 :: ∀a b . a -> b■ f3 :: ∀a . [a] -> [a]

● Strong Static Typing○ Reasoning about Types

■ f1 :: ∀a . a -> a● Can only be the identity function

■ f2 :: ∀a b . a -> b● Impossible

■ f3 :: ∀a . [a] -> [a]● Theorems for free! [Wadler 1989]● map f . f3 == f3 . map f

Prelude - Haskell

● Strong Static Typing○ f: plus 2○ f3: swap the first two element if possible

Prelude - Haskell

[1,2,3,4] [3,4,5,6]

[2,1,3,4] [4,3,5,6]

map f

map f

f3 f3

Overview - QuickCheck

● Test Case Generation○ Value Generation○ Function Generation

● Output Verification○ Describing Desired Properties

● Case studies● Discussion

Test Case Generation - Type Class

● Type class: ad hoc polymorphism support○

in Prolog:

eq(bool).eq(char).…

ord(X) :- eq(X).

Test Case Generation - Arbitrary

● newtype Gen a = Gen (Rand -> a)○ reads “some computation(Gen) that will give you a”class Arbitrary a where

arbitrary :: Gen a

instance Arbitrary Int where

arbitrary = choose (-20,20)

instance (Arbitrary a, Arbitrary b) => Arbitrary (a,b) where

arbitrary = do {v1 <- arbitrary; v2 <- arbitrary; return (v1,v2)}

We split a random seed (using a primitive function) into two independent seeds, notation “<-” and “return” will take care of that.

Test Case Generation - Arbitrary

● newtype Gen a = Gen (Rand -> a)○ reads “some computation(Gen) that will give you a”instance (Arbitrary a, Arbitrary b) => Arbitrary (a,b) where

arbitrary = liftM2 (,) arbitrary arbitrary

-- shorthand:

-- liftM f m1 = do { v1 <- m1; return (f v1) }

-- liftM2 g m1 m2 = do { v1 <- m1; v2 <- m2; return (g v1 v2) }

Test Case Generation - Arbitrary

● Generate Recursive Data Structure○ data Tree a = Branch (Tree a) (Tree a)

| Leaf a deriving (Show)

instance (Arbitrary a) => Arbitrary (Tree a) where arbitrary = frequency [ (1, liftM Leaf arbitrary) , (2, liftM2 Branch arbitrary arbitrary) ]

Test Case Generation - Arbitrary

● Generate Recursive Data Structure○ data Tree a = Branch (Tree a) (Tree a)

| Leaf a deriving (Show)

instance (Arbitrary a) => Arbitrary (Tree a) where arbitrary = frequency [ (1, liftM Leaf arbitrary) , (2, liftM2 Branch arbitrary arbitrary) ]

● The generator might not terminate● Likely to produce huge trees

Test Case Generation - Arbitrary

● Generate Recursive Data Structure○ Limit the size○ The notion of “size” is hard to define, leave it to

programmers○ newtype Gen a = Gen (Int -> Rand -> a)○ sized :: (Int -> Gen a) -> Gen a

Test Case Generation - Arbitrary

● Generate Recursive Data Structure○ data Tree a = Branch (Tree a) (Tree a)

| Leaf a deriving (Show)

instance (Arbitrary a) => Arbitrary (Tree a) where arbitrary = sized arbTree

arbTree :: Int -> Gen aarbTree 0 = liftM Leaf arbitraryarbTree n = frequency [ (1, liftM Leaf arbitrary) , (4, liftM2 Branch (arbTree (n `div` 2)) (arbTree (n `div` 2)) ]

Test Case Generation - CoArbitrary

● Generate Functions○ Use input value to perturb the random generator Gen b

■ class CoArbitrary a where■ coarbitrary :: a -> Gen b -> Gen b

○ We should be able to generate an arbitrary b using Gen b○ Use input value a to perturb Gen b so that we will have a modified

Gen b○ This is a pure function because given the same generator, the output

depends merely on input value.

Output Verification - DSL

● a Domain Specific Language (DSL) embedded in Haskell○ Property○ Function Equality○ Conditional Laws○ Classify Input○ Custom Data Generator

Output Verification - DSLprop_RevUnit x = reverse [x] == [x]

prop_RevApp xs ys = reverse (xs++ys) == reverse ys++reverse xs

prop_RevRev xs = reverse (reverse xs) == xs

type Endo a = a -> a

prop_CompAssoc :: Endo Int -> Endo Int -> Endo Int -> Int -> Boolprop_CompAssoc f g h = f . (g . h) === (f . g) . h

● Property

● Function Equality (===)

Output Verification - DSL

● Conditional Laws (==>)○ “A ==> B” is different from “not A || B” ○ try checking it for 100 test cases satisfying the

condition○ generate only a limited number of test cases

prop_MaxLe :: Int -> Int -> Propertyprop_MaxLe x y = x <= y ==> max x y == y

Output Verification - DSL● Classify Input: classify, collect● Custom Data Generator: forAll

prop_Insert :: Int -> Propertyprop_Insert x = forAll orderedList $ \xs -> classify (null xs) “trivial” $ ordered (insert x xs)

Case studies

● Unification○ Random generated terms are unlikely to be unified○ New types are introduced to produce a different input distribution

● Lava○ Provide symbolic input and use external theorem prover

● Pretty Printing○ Extending Arbitrary type class to include:○ shrink :: a -> [a]

Discussion

● On Random Testing○ a Haskell test framework written in Haskell

■ Lightweight■ Doesn’t tie to a particular implementation

○ Many criteria need reinterpretation■ Reachable statement?■ Constraint solving in Haskell program is hard

Discussion

● Correctness Criteria○ a Haskell test framework written in Haskell

■ Property Language is much more general● Function Equality (e.g. current vs. known correct version)● Result Checking (e.g. mathematical properties)● Conditional Properties (e.g. insert on sorted list)● Testing High-Order Functions (e.g. function composition)

○ No published work on automatic testing of functional programs against specification

Discussion

● Test Data Generation○ a Haskell test framework written in Haskell

■ another DSL for describing test data generation● Grammar inherited from Haskell● Minimum learning effort

○ Controlling sizes to guarantee termination■ Need developer to

● Interpret the meaning of “size”● Specify generators for his/her own new types

Discussion

● Some Reflections○ Formulating formal specification improves our

understanding of our programs○ Three types of errors divided evenly

■ Errors in test generators■ Errors in the specification■ Errors in the program

References● Claessen, Koen, and John Hughes. "QuickCheck: a lightweight tool for

random testing of Haskell programs." Acm sigplan notices 46.4 (2011): 53-64.

● Wadler, Philip. "Theorems for free!." Proceedings of the fourth international conference on Functional programming languages and computer architecture. ACM, 1989.

● Weyuker, Elaine J. "On testing non-testable programs." The Computer Journal 25.4 (1982): 465-470.