An Effective Methodto Control Interrupt Handler

for Data Race Detection

Makoto Higashi †, Tetsuo Yamamoto ‡, Yasuhiro Hayase †, Takashi Ishio † and

Katsuro Inoue †

† Osaka University‡ Ritsumeikan University

Outline

• Motivation– Embedded software and data race conditions

• Approach– Control of an interrupt handler

• Case study– uClinux

• Summary and future work

2010/5/4

AST2010 2

2010/5/4

AST2010 3

Reliability of embedded software

• There are many safety-critical embedded software– Pacemaker: risk of losing human life– Mobile phone: risk of intercepting personal

data

• Embedded system consists of external devices and control software– Embedded software becomes aware of inputs

from external devices through interrupts or I/O memory

2010/5/4

4

Interrupt-driven software

• We focus on interrupt-driven software in embedded software, where processing is initiated when external devices signal the CPU– Interrupts add fine-grained concurrency to the

softwarehandler(void)

op = 0

return

Interrupt handlermain(void)

if (op == 1) …

Main routine

interrupt

return

interrupt

Particular fault to interrupt-driven software

• Main routine shares memory with interrupt handler

• There is a failure at the particular timing– Interrupt occurs at unexpected timing– Interrupt handler changes shared memory

2010/5/4

AST2010 5

It is important to detect

data race conditions

2010/5/4

AST2010 6

Example of data race condition

divide(void)

no

yes

return

x != 0

ret = 100 / x

interrupt_handler(void)

x = 0

return

interrupt

interrupt

interrupt

interrupt

Interrupt handler

Main routinevariable

xaccess

accessaccess

Divide 100 by x in case of x != 0Divide 100 by x

in a state of x == 0

Detection of data race conditions through testing

2010/5/4

AST2010 7

Testing Process

• Testing process of non interrupt-driven software– Input values to module– Check return value from the module

• Testing process of interrupt-driven software– Combination Interrupt handler with other

module– Consider a wide variety of Interrupt timing

Key ideas 1/2

• Condition of data race condition1. Main routine accesses a variable twice

• 1st access is reading or writing

• 2nd access is reading

2. Main routine assumes that the value of the variable is unchanged

3. Interrupt occurs between 1st and 2nd access

4. Interrupt handler changes the value of the variable

2010/5/4

AST2010 8

x = 3

ret = 100 / x

x != 0 Assumes x != 0

no

return

interrupt

ret = 100 / x

interruput

Assumes a == 3

Key ideas 2/2

• Control of execution path of interrupt handler

• Embedded software uses memory-mapped I/O for communicating with external devices– Load instructions read the value of memory– Substitute user specified value for the value of

memory

2010/5/4

AST2010 9

Our work

• Objective: Testing of interrupt-driven software

• Approach: Detection of faults related to interrupts (data race conditions) to cause interrupts automatically

• Result: Detection a fault not to cause interrupts manually

2010/5/4

AST2010 10

2010/5/4

AST2010 11

Mechanism to cause interruptsCPU emulator

Mechanism to cause interrupts

interpreter

A user specifies the kind ofinterrupt

Configuration file

Machine language

instruction

Checkread instruction orwrite instruction

LDR ADDRADDMOV

・・・・・・

Interrupt handler:

・・・・・・

RET

interrupt

2010/5/4

AST2010 12

Prevention of infinite loop

Comparecurrent program counter

with the saved counter

If the saved counter isdifferent from

the current program counter,casuse an interrupt andsave program counter

Mechasin to cause interrupts

Saved counter

Currentprogram counter

interrupt

LDR ADDRADDMOV

・・・・・・

Interrupt handler:

・・・・・・

RET

Result ofcomparison

Currentprogram counter

Access memory

Interrupt handlerPrevent causing interrupts at the same

location

Saved counter

2010/5/4

AST2010 13

Mechanism to substitute values

Mechanism to substitute values

Memory accessLDR ADDRADDMOV

・・・・・・

Return user specified value

Return value

Read instrunctionCPU emulator

Memory access

memory

check if the memory address isuser specified address

If yes,Substitute the value

Memory access

Return value

0100011101010101111000101111

…..…..

2010/5/4

AST2010 14

Configuration file 1/2

• Interrupt– The kind of interrupt– Support only one kind of interrupt in single file

• Memory address– An address which is mapped to external

device– If you know memory address of global

variable, the address can be specified

2010/5/4

AST2010 15

Configuration file 2/2

• Function name– Substitute new value within only specified

function• Because it is very slow to substitute values within

all functions

• New value– Constant– Global variables– Current value

Case study

• We have applied our method to software which contains data race condition– Aim

• Investigate the process to detect the data race condition

– Target software• uClinux

2010/5/4

AST2010 16

Data race condition on uClinux

• When sending characters of queue, the code accesses out of queue– After checking the count of queue, the routine

sends characters– Just after the checking, interrupts occurs

2010/5/4

AST2010 17

・

if (xmit_cnt <= 0 || ……) return;

・・・

xmit_cnt--;

・

if (xmit_cnt <= 0 || ……) return;

・・・

xmit_cnt--;

Main routine Interrupt handler

interrupt

The count of queue is 1

The count of queue is 0Access out of queue

2010/5/4

AST2010 18

Procedure to detect the data race condition

• Assign 5 to the count of queue– Because interrupts occurs 4 times before the

count of queue is checked

• Call main routine

・

Static void rs_flush_chars(struct tty_struct *tty){struct m68k_serial *info = ……;m68328_uart *uart = ……;

・・・

if (xmit_cnt <= 0 || ……) return;

interrupt interrupt

interrupt

interrupt

2010/5/4

AST2010 19

Testing process

1. Doubt the possibility to cause a data race condition within a certain module A

2. Assume an interrupt handler B to cause the data race condition in cooperation with module A

3. Specify the kind of the interrupt handler to configuration file

4. Test module A

Cost

• CPU cycles– Our method: 72,417,488– Normal execution: 4,836,078

About 15 times

• The total number of cycles took in the interrupt handler: 69,952,632

2010/5/4

AST2010 20

2010/5/4

AST2010 21

Data race condition that our mechanisms cannot detect 1: unsigned int len = 0; 2: void str_cpy(char *buf, char *str); 3: { 4: len = strlen(str); 5: if((0 < len) && (len <= strlen(str))) 6: memcpy(buf,str,len+1); 7: } 8: 9: void interrupt_handler(void){10: len++;11: }

interrupt

interrupt

No data race condition

data race condition

2010/5/4

AST2010 22

Summary and future work

• We have implemented 2 mechanisms to a CPU emulator to test for data race conditions in interrupt-driven software– Causes interrupts automatically– Substitute values of memory

• Future work– More appropriate timing of interrupts– Multiple kinds of interrupts