GuidedSampler:Coverage-guided Samplingof SMT SolutionsRafael Dutra, Jonathan Bachrach, Koushik SenEECS DepartmentUC Berkeley

Formal Methods in Computer-Aided DesignOctober 25, 2019

A SMT solver can generate one solution:

Constraint Sampling

2

mem[0] mem[1]

σ0 1 0 0 0 1 0 0 0

∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])

Input: SMT formula φ

Goal: Generate many solutions to φ

Constraint Sampling

3

mem[0] mem[1]

σ0

σ1

σ2

σ3

σ4

σ5

1 0 0 0 1 0 0 0

0 0 0 0 1 0 1 0

1 1 0 0 1 0 0 1

1 0 0 0 0 1 0 0

0 1 0 0 1 0 1 1

0 0 0 0 0 1 1 0

∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])

Input: SMT formula φ

● Synthesis● Symbolic execution

Motivation: Sampling Solutions

● Thoroughly exercising some target functionality● Constrained-Random Verification

4

int4 x, y, z, w;int4 mem[4] = {x, y, z, w};for (int4 i = 0; i < 4; ++i) {

mem[mem[i]] *= -1;}

i < 4

mem[0] < 0∨ mem[0] ≥ 4

SMT: Satisfiability Modulo Theories

5

SMT formula φ

∧(mem[0] ≥ 0 ∧ mem[0] < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wheremem’ = store(mem, mem[0], -1 * mem[mem[0]])

mem ∈ Array(BV[4], BV[4])

mem ∈ Array(BV[4], BV[4])

SMT: Satisfiability Modulo Theories

6

SMT formula φBit-vector

∧(mem[0] ≥ 0 ∧ mem[0] < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wheremem’ = store(mem, mem[0], -1 * mem[mem[0]])

SMT: Satisfiability Modulo Theories

7

SMT formula φBit-vector

Array

mem ∈ Array(BV[4], BV[4])

∧(mem[0] ≥ 0 ∧ mem[0] < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wheremem’ = store(mem, mem[0], -1 * mem[mem[0]])

State of the art

● SMTSampler (our prior work)○ Efficient generation of solutions for SMT formulas

● Markov Chain Monte Carlo (MCMC)○ Works for linear constraints and can generate biased solutions

● Constraint solver heuristics○ Can be expensive, requiring one solver call per solution

● Universal hashing○ Expensive, but can guarantee exact distribution of solution

● Weighted Sampling○ Literal-weighted distributions: WAPS

8

Goal: Generate solutions to φ

SMTSampler

9

mem[0] mem[1]

σ0

σ1

σ2

σ3

σ4

σ5

1 0 0 0 1 0 0 0

0 0 0 0 1 0 1 0

1 1 0 0 1 0 0 1

1 0 0 0 0 1 0 0

0 1 0 0 1 0 1 1

0 0 0 0 0 1 1 0

∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])

Input: SMT formula φ

Goal: Generate solutions to φ

Coverage-guided Sampling

10

mem[0] mem[1]

σ0

σ1

σ2

σ3

σ4

σ5

1 0 0 0 1 0 0 0

0 0 0 0 1 0 1 0

1 1 0 0 1 0 0 1

1 0 0 0 0 1 0 0

0 1 0 0 1 0 1 1

0 0 0 0 0 1 1 0

∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])

mem’[1] < 0

Input: SMT formula φ

Input: Coverage predicates

mem’[1] ≥ 4 mem’[0] < 0

ψ1 ψ2 ψ3

Goal: Generate solutions to φ such that the predicates ψ1, ψ2, …, ψn are covered uniformly

Coverage-guided Sampling

11

mem[0] mem[1] ψ1 ψ2 ψ3

σ0

σ1

σ2

σ3

σ4

σ5

1 0 0 0 1 0 0 0

0 0 0 0 1 0 1 0

1 1 0 0 1 0 0 1

1 0 0 0 0 1 0 0

0 1 0 0 1 0 1 1

0 0 0 0 0 1 1 0

∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])

mem’[1] < 0

Input: SMT formula φ

0 1 0

0 1 0

0 0 1

1 1 0

0 1 0

1 1 0

mem’[1] ≥ 4 mem’[0] < 0

Input: Coverage predicates

ψ1 ψ2 ψ3

Goal: Generate solutions to φ such that the predicates ψ1, ψ2, …, ψn are covered uniformly

Coverage-guided Sampling

12

mem[0] mem[1] ψ1 ψ2 ψ3

σ0

σ1

σ2

σ3

σ4

σ5

1 0 0 0 1 0 0 0

0 0 0 0 1 0 1 0

1 1 0 0 1 0 0 1

1 0 0 0 0 1 0 0

0 1 0 0 1 0 1 1

0 0 0 0 0 1 1 0

∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])

mem’[1] < 0

Input: SMT formula φ

0 1 0

0 1 0

0 0 1

1 1 0

0 1 0

1 1 0

mem’[1] ≥ 4 mem’[0] < 0

Input: Coverage predicates

ψ1 ψ2 ψ3

Uniformity over Coverage Classes

13

Uniformity over Coverage Classes

14

Uniformity over Coverage Classes

15

Challenges

● Coverage of the formula might still not be ideal even using state-of-the-art approaches, such as SMTSampler

● User might be interested in a specific notion of coverage for the produced solutions

16

GuidedSampler

17

GuidedSampler

Our goals:

● Sample solutions from a formula φ, but have the distribution determined by the coverage predicates ψ1, ψ2, …, ψn

● Uniformly sample solutions from the different coverage classes

● Uniformly sample within each coverage class

Our approach:

● Compute simple mutations that can be applied to one solution to generate another solution from a different class

● Combine those mutations together to generate a large number of new solutions

18

19

Formula φ

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

Coverage Predicates

...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

ψ1

ψ2

ψ3

20

Formula φx = mem[0]y = mem[1]

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

21

Formula φx = mem[0]y = mem[1]

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])ψ1 ψ2 ψ3

0 1 1RandomClass

...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

22

Solution σ 1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])ψ1 ψ2 ψ3

0 1 1

ψ1 ψ2 ψ3

0 1 0...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

RandomClass

23

Solution σ 1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

ψ1 ψ2 ψ3

0 1 0

MAX-SMT ...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

24

Solution σ 1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

MAX-SMT

Hard constraints● φ● ψ1 ≠ 0

Soft constraints● ψ2 = 1● ψ3 = 0● x1 = 0● x2 = 0● x3 = 0● y0 = 1● y1 = 0● y2 = 0● y3 = 0

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

ψ1 ψ2 ψ3

0 1 0

MAX-SMT ...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

σ1

25

Solution σ

0 0 0 0 1 0 1 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

MAX-SMT

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

ψ1 ψ2 ψ3

0 1 0

1 1 0

...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

σ1

26

Solution σ

0 0 0 0 1 0 1 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

ψ1 ψ2 ψ3

0 1 0

MAX-SMT ...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

σ1

27

Solution σ

0 0 0 0 1 0 1 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

σ1

28

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

σ2

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0

0 0 0

...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

σ1

29

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

σ2

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

σ1

30

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

σ2

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

Coverage Predicates

ψ1

ψ2

ψ3

σ1

31

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

σ2 σ3

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

0 0 1

Coverage Predicates

ψ1

ψ2

ψ3

σ1

32

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

σ2 σ3

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

σ1

33

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

MAX-SMT

= σ ⊕ σ1 = σ ⊕ σ2 = σ ⊕ σ3

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

δ12

σ1

34

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

MAX-SMT

= δ1 ∨ δ2

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

σ12

δ12

σ1

35

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

0 1 0 0 1 0 1 1

MAX-SMT

= σ ⊕ δ12

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

ψ1 ψ2 ψ3

1 0 0

...

Coverage Predicates

ψ1

ψ2

ψ3

σ12

δ12

σ1

36

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

0 1 0 0 1 0 1 1

MAX-SMT

= σ ⊕ δ12

Why does it work?● δ1 and δ2 are a minimal set of bits that can be

flipped and preserve the satisfiability of the formula● It’s likely that the formula has some clauses

establishing a relation between those bits● Those clauses will likely still be satisfied when

flipping both the bits in δ1 and δ2

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

ψ1 ψ2 ψ3

1 0 0

...

Coverage Predicates

ψ1

ψ2

ψ3

σ12

δ12

σ1

37

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

0 1 0 0 1 0 1 1

MAX-SMT

= σ ⊕ δ12

⇐ And new sample σ12 is likelyfrom a new coverage class

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

ψ1 ψ2 ψ3

1 0 0

...

Coverage Predicates

ψ1

ψ2

ψ3

σ12

δ12

σ1

38

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

0 1 0 0 1 0 1 1

MAX-SMT

= σ ⊕ δ12

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

ψ1 ψ2 ψ3

1 0 0

...

Coverage Predicates

ψ1

ψ2

ψ3

σ12

δ12

σ1

39

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

0 1 0 0 1 0 1 1

MAX-SMT

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

δ131 0 0 0 1 1 1 0

= δ1 ∨ δ3

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

σ12

δ12 δ13

σ13

σ1

40

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

1 0 0 0 1 1 1 0

0 0 0 0 0 1 1 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

0 1 0 0 1 0 1 1

MAX-SMT

= σ ⊕ δ13

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

ψ1 ψ2 ψ3

1 0 1

...

Coverage Predicates

ψ1

ψ2

ψ3

σ12

δ12 δ13 δ23

σ13

σ1

41

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

1 0 0 0 1 1 1 0

0 0 0 0 0 1 1 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

0 1 0 0 1 0 1 1

0 1 0 0 1 1 0 1

MAX-SMT

= δ2 ∨ δ3

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

σ12

δ12 δ13 δ23

σ13

σ1

42

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0

1 0 0 0 1 1 1 0

0 0 0 0 0 1 1 0

σ2 σ3

δ31 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

0 1 0 0 1 0 1 1

0 1 0 0 1 1 0 1

MAX-SMT

σ231 1 0 0 0 1 0 1

= σ ⊕ δ23

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

ψ1 ψ2 ψ3

0 0 1

...

Coverage Predicates

ψ1

ψ2

ψ3

σ12 σ13

σ1

43

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 0 0 0 0 1 1 0

σ2 σ3

0 1 0 0 1 0 1 1

MAX-SMT

σ231 1 0 0 0 1 0 1

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

σ12 σ13

σ1

44

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 0 0 0 0 1 1 0

σ2 σ3

0 1 0 0 1 0 1 1

MAX-SMT

σ231 1 0 0 0 1 0 1

17 / 18 valid solutions

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

σ12 σ13

σ1

45

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 0 0 0 0 1 1 0

σ2 σ3

0 1 0 0 1 0 1 1

MAX-SMT

σ231 1 0 0 0 1 0 1

σ1230 1 0 0 0 1 1 1

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

ψ1 ψ2 ψ3

1 0 1

...

Coverage Predicates

ψ1

ψ2

ψ3

σ12 σ13

σ1

46

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 0 0 0 0 1 1 0

σ2 σ3

0 1 0 0 1 0 1 1

MAX-SMT

σ231 1 0 0 0 1 0 1

σ1230 1 0 0 0 1 1 1

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

ψ1 ψ2 ψ3

1 0 1 Repeated class

...

Coverage Predicates

ψ1

ψ2

ψ3

σ12 σ13

σ1

47

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 0 0 0 0 1 1 0

σ2 σ3

0 1 0 0 1 0 1 1

MAX-SMT

σ231 1 0 0 0 1 0 1

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

σ12 σ13

σ1

48

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 0 0 0 0 1 1 0

σ2 σ3

0 1 0 0 1 0 1 1

MAX-SMT

σ231 1 0 0 0 1 0 1

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

At most ngenerate atomic mutations

=O(n6)mutations: NO MAX-SMT ( ) n

6 samples by combining

MAX-SMT calls to

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

σ12 σ13

σ1

49

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 0 0 0 0 1 1 0

σ2 σ3

0 1 0 0 1 0 1 1

MAX-SMT

σ231 1 0 0 0 1 0 1

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

At most 50generate atomic mutations

15 890 700mutations: NO MAX-SMT

samples by combining

MAX-SMT calls to

MAX-SMT

ψ1 ψ2 ψ3

0 1 0......

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

σ12 σ13

σ1

50

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 0 0 0 0 1 1 0

σ2 σ3

0 1 0 0 1 0 1 1

MAX-SMT

σ231 1 0 0 0 1 0 1

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0...

Coverage Predicates

ψ1

ψ2

ψ3

...

mem’[1] < 0

mem’[1] ≥ 4

mem’[0] < 0

σ12 σ13

σ1

51

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0

1 0 0 0 1 0 0 0

1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3

0 0 0 0 0 1 1 0

σ2 σ3

0 1 0 0 1 0 1 1

MAX-SMT

σ231 1 0 0 0 1 0 1

Formula φx = mem[0]y = mem[1]

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])

MAX-SMT

ψ1 ψ2 ψ3

0 1 0...

Coverage Predicates

ψ1

ψ2

ψ3

Random assignment

σ′

52

Key Ideas

σ

Random assignment

Base solution

σ′

53

Key Ideas

● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values

σ2

σ1

σ

Random assignment

Base solution

Closest solutions

σ′

54

σ3σ4

Key Ideas

● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values

● M1: Find neighboring solutions that flip coverage predicates

Random assignment

Base solution

Closest solutions

Generated samples

σ2 σ12

σ1

σσ′

55

σ3σ4

Key Ideas

● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values

● M1: Find neighboring solutions that flip coverage predicates

● M2: Whenever generating a new sample, discard it if it’s from a repeated coverage class

σ2 σ12

σ1

σ

Random assignment

Base solution

Closest solutions

Generated samples

σ′

56

σ3σ4

Key Ideas

● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values

● M1: Find neighboring solutions that flip coverage predicates

● M2: Whenever generating a new sample, discard it if it’s from a repeated coverage class

σ2 σ12

σ1

σ

Random assignment

Base solution

Closest solutions

Generated samples

σ′

57

σ3σ4

Key Ideas

● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values

● M1: Find neighboring solutions that flip coverage predicates

● M2: Whenever generating a new sample, discard it if it’s from a repeated coverage class

Implementation

● Implemented in C++ using Z3 as the constraint solver● https://github.com/RafaelTupynamba/GuidedSampler

58

Experiments on SMT-LIB

We evaluated GuidedSampler on 213 industrial benchmarks from 22 classes.

59

Benchmark Class Average # Nodes Average # Bits

QF_AUFBV/ecc 179 1931

QF_ABV/bmc-arrays 855 53

QF_ABV/stp_samples 1139 192

QF_BV/bmc-bv-svcomp14 7518 7607

QF_BV/tacas07 8812 16620

QF_BV/sage/app8 978 1047

Experiments

We compared 6 approaches for SMT sampling:● BH: Baseline with hard constraints● BS: Baseline with soft constraints● S0: SMTSampler [1]● S1 = S0 + M1 (flipping coverage predicates to generate neighboring solutions)● S2 = S0 + M1 + M2 (discarding solutions from repeated classes)● S3 = S0 + M1 + M2 + M3: GuidedSampler (randomize class of base solution)

[1] Rafael Dutra, Jonathan Bachrach and Koushik Sen. 2018. SMTSampler: Efficient Stimulus Generation from Complex SMT Constraints. In ICCAD 2018. 60

Coverage Predicates

● Internal Predicates○ Look at values of internal nodes○ Analogous to internal wires in a circuit○ General notion of coverage from the

formula itself

● Random Predicates○ Random formulas generated from a

grammar including variables of φ○ Problem-specific notion of coverage

61

Experiments: Unique Coverage Classes

62

Higher is better

GuidedSampler vs. BH Baseline

Number of uniquecoverage classesper time

Experiments: Unique Coverage Classes

63

Higher is better

GuidedSampler vs. BS Baseline

Number of uniquecoverage classesper time

Experiments: Unique Coverage Classes

64

Higher is better

GuidedSampler vs. SMTSampler

Number of uniquecoverage classesper time

Experiments: Unique Coverage Classes

65

Higher is better

S3 = GuidedSamplerS0 = SMTSamplerBS, BH: baselines

Number of uniquecoverage classesper time

Experiments: Uniformity over Coverage Classes

66

S3 = GuidedSamplerS0 = SMTSamplerBS, BH: baselines

→ GuidedSampler generated > 100 000 classes

Discussion

● The most important modification is M1, which allows covering 3.1 times more classes in average

● M1 and M2 are also essential for producing a more uniform distribution over coverage classes.

● Similar results for internal predicates and random predicates

67

Modifications:

● M1: Flipping coverage predicates to compute neighboring solutions

● M2: Discarding new solutions that repeat a previously seen coverage class

● M3: Randomizing coverage class of initial base solution

Conclusion

● Generating lots of solutions efficiently given an SMT formula

● Generate millions of solutions with tens of solver calls

● Achieve better coverage of the constraint space, even for user-defined coverage classes

68

σ12 = σ ⊕ δ12

δ12 = δ1 ∨ δ2

σ1

Solution σ

0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1

1 0 0 0 1 0 0 0

0 1 0 0 0 0 0 1

σ2

1 0 0 0 0 0 1 0 δ2δ1

1 1 0 0 0 0 1 1

0 1 0 0 1 0 1 1

ψ1 ψ2 ψ3

0 1 0

ψ1 ψ2 ψ3

1 0 0

MAX-SMT

x0 x1 x2 x3 y0 y1 y2 y3