System SoftwareScanning

Part - II

Parse Trees

• A sequence of derivations or reductions reveals the syntactic structure of a string with respect to G.

• This Syntactic structure can be viewed in form of a parse tree.

• Example – the production A ::= α gives the following parse

tree

Parse Trees

A

NT iSequence of Ts and NTs constituting α

A subsequent step in the derivation replaces an NT in α say NTi by a string. To view this derivation the following parse tree can be built.

NTi

The two parse trees can be combined by replacing the node of Nti in the first tree with the second tree. Thus the parse tree has grown in the downward direction due to derivation.

<Sentence>

<Noun Phrase>

<Article>

<Verb Phrase>

<Noun Phrase> <Verb>

<Noun> <Article>

subject an Teachesteacher The

<Noun>

Recursive Specification

• Following is the Grammar 1.2 for an Arithmetic expression containing , * and +

<exp> ::= <exp>+<term>|<term> <term> ::= <term> * <factor> | < factor><factor> ::= <factor> <primary> | <primary><primary> ::= <id>|<constant>|(<exp>)<id> ::= <letter> |<id>[<letter>|<digit>]<const> ::= [+|-]<digit>|<const><digit><letter> ::= a|b|c|d|…|z<digit> ::= 0|1|2|3|…|9

Recursive specification contd

• This Grammar uses recursive specification where the NT being defined in a production itself occurs in a RHS string of the production eg X:= ….X ….

• This RHS alternative employing recursion is called a recursive rule.Example : A non recursive specification for expressions

containing the ‘+’ operator.<exp> ::= <term>|<term> + <term> | <term> + <term> +

<term>| ….Using recursion the <exp> can be specified simply as<exp> ::= <exp> + <term> | < term>

Recursive Specification contd• The Recursive rules are classified to be left recursive or

right recursive depending whether the NT defined appears on the extreme left or extreme right in the recursive rule.

• All the recursive rules of the grammer for the expression are left recursive rules.

• Indirect recursion occurs when two or more NTs are defined in terms of one another. It is useful for specifying nested constructs in a language.

• Direct Recursion is not useful in situations where a limited number of occurrences is required.

• Controlled recurrence can be specified by using { …}150

Classification of Grammer

• Type 0 Grammer– Phrase structure Grammers contains production

of the form – α ::= β where this are both strings of Ts and NTs– Such productions permit arbitrary substitution of

strings during derivation or reduction.

• Type 1 grammar– It is known as Context sensitive grammer because

their productions specify that derivation or reduction of strings be done only in specific contexts.• α A β ::= α π β

– This grammar is not relevant for PL specification, since recognizing PL constructs is not context sensitive in nature.

• Type -2 Grammars– It imposes no context requirements on

derivations or reductions– Type 2 production is of the form

• A ::= π which can be applied independent of its context.

– These grammar are known as context Free Grammar.

– It is suited for PL specification.

• Type 3 Grammar– Type 3 production is of the form

• A ::= t B | t or • A ::= Bt|t

– These productions also satisfy the requirements of type 2 grammars

– The use of type 3 productions is restricted to the specification of lexical units ex : identifiers, constants etc .

– The productions for <constant> and <identifier> in grammar (1.2) are infact Type 3 in nature.

• Operator Grammer– An operator grammar is a grammar in which none

of whose productions contain two or more consecutive NTs in any RHS alternative.

Lexical Analysis

• It’s main task is to read the input characters and produce as output a sequence of tokens that the parser uses for syntax analysis.

Lexical Analyzer Parser

Symbol table

token

get next Token

Source program

Lexical analyzers contd.

• It can perform certain secondary tasks also at the user interface level like– Removing from the source program comments

and white space– Correlating the error messages from the compiler

with the source program.• Some lexical analyzers are divided into two

phases • Scanning • Lexical analysis.

Lexical analysis

Token Sample lexemes Informal description of Patterns

Const Const Const

If If If

Relation <, <= , = , <>, > , >= < or <= or = or <> or > or >=

Id totalCount Letter followed by letters and / or digits.

Num 2.14, 5.46, 10 Any numeric constant

Terms normally used in Lexical analysis. - Token : set of strings in the input for which the same token is produced as output.- Pattern : A Pattern is a rule describing the set of lexemes that can represent a particular token in source programs.- lexeme : It is a sequence of characters in the source program treated as a lexical unit , that is matched by a pattern for a token.

Example : int totalcount = 10;

The substring totalcount is a lexeme for the token identifier.

About Tokens

• Tokens are treated as Terminal symbols in the grammar for the source language.

• Quick review of the term used in lexical analysis.• In Programming Languages the constructs like

keywords , operators , identifiers , constants and punctuation symbols are treated as tokens

• The patterns for more complex tokens like id and num can be described by using regular Expressions.

About the tokens

• Attributes for tokens– When one pattern matches more than one lexemes ,

the lexical analyzer must provide additional information about the particular lexeme that matched.

– The lexical analyzer collects information about tokens into their associated attributes.

– a token usually has only one single attribute a pointer to the symbol table entry in which the information of token is kept. • Information of token Like : lexeme for an identifier and the

line number on which it was first seen.

Token’s Attributes

• Example • Tokens and associated attribute for a C

Statement– totalSalary = basic + da + 100;Tokens<id, pointer to symbol table entry for totalSalary><assign_op, ><id, pointer to symbol table entry for basic><add_op, ><id, pointer to symbol table entry of da><add_op, ><num , integer value 100>

Specification of tokens

• Regular expressions are important notation for specifying patterns. Each pattern matches a set of Strings , thus Regular Expressions will serve as names for sets of Strings.

• Strings– The term alphabet or character class denotes any

finite set of symbols• Example {0,1} is the set of binary alphabet.

About String

• A string over some alphabet is a finite sequence of symbols drawn from that alphabet.

• Terms like sentence and word are often used as synonyms for the term string

• Length of the String is the no of occurrences of symbols in s denoted as |s|.

• Empty string is denoted by a basic symbol є• Concatenation of Strings or string and empty String

String contd.

• Terms for parts of the String• Prefix of s• Suffix of s• Substring of s• Proper prefix , suffix or substring of s• Subsequence of s.

Strings Contd

• Terms for parts of the StringTerm Definition

Prefix of s A string obtained by removing zero or more trailing symbols of String ‘s’

Suffix of s A string obtained by deleting zero or more leading symbols of String ‘s’

Substring of s A string obtained by deleting a prefix and a suffix from ‘s’

Proper prefix , suffix or substring of s

Any nonempty string x that is respectively, a prefix , suffix or substring of s such that s != x

Subsequence of s Any string formed by deleting zero or more not necessarily contiguous symbols from ‘s’

Operations on Languages

Language denotes any set of strings over some fixed alphabet.

Ex ; English language is a set of all grammatically correct English Sentences.

- For lexical analysis there are several operations that can be applied to languages - Operation like union, concatenation and closure

Operation on Languages contd.Operation Definition

Union of L and DWritten L U D

L U D= { s | s is in L or s is in D}

Concatenation of L and DWritten LD

LD = {sd | s is in L and d is in D}

Kleene closure of LWritten as L*

L* denotes “ zero of more concatenations of” L

Positive closure of L Written L+

L+ denotes “one or more concatenations of” L

Let L be the alphabet consisting of the set of upper and lower case letters L = {A,B,…..Z , a,b, …..z} andLet D be the alphabet consisting of the set of ten decimal digits. D= { 0,1,….9}Here the symbol is regarded as a string of length one , the sets L and D are finite languages. Lets Discuss the new Languages created from L and D by applying the above mentioned operators.

Regular Expressions• Regular Expression• - this notation allows to define tokens ex id

– Identifier : is a letter followed by zero or more letters or digits, that is an identifier is a member of the set defined as below• L(LUD)* is the set of all strings of letters and digits beginning with a

letter.– Regular expressions allows us to define precisely sets such as above with

different notation• letter(letter | digit)*

– Regular Expression notationsNotation Meaning

“ | “ Or

() used to group subexpressions

* Means zero or more instances of the parenthesized expressions

Justaposition of letter with the remainder of the expn

Concatenation

Regular Expressions Continued.

• Each regular Expression r denotes a language L(r) • The following are the Rules that define the

regular expressions over alphabet Σ . Associated with each rule is a specification of the language denoted by the regular expression being defined.

1)Є is a regular expression that denotes {є}, that is , the set containing the empty string.

2) If a is a symbol in Σ , then a is a regular expression that denotes {a}. The set containing the string a.

Regular Expression Contd.

3) suppose r and s are regular expressions denoting the languages L(r) and L(s). Thena)(r)|(s) is a regular expression denoting L(r)|L(s)b) (r)(s) is a regular expression denoting L(r)L(s)c) (r)* is a regular expression denoting L(r)*d) (r)+ is a regular expression denoting L(r)+A language denoted by a regular expression is said

to be a regular set.

• Precedence Rule– unary operator * has the highest precedence and

is left associative– Concatenation has the second highest precedence

and is left associative– | has the lowest precedence and is left

associative.Example (a)|((b)*(c)) is equi. To a|b*c

Regular Expressions - Examples• Example • The regular expression (a|b)(a|b) denotes the set {aa, ab,

ba, bb} , the set of all strings of a’s and b’s of length two• Equivalent regular expression for the same set is

aa|ab|ba|bb• Exercise 1) which set of strings does the regular expression a* denotes 2) which set of strings does the regular expression a|a*b

denotes ?

•

Algebraic properties of regular expressions

Axiom Description

r|s = s|r | is commutative

r|(s|t) = (r|s)|t | is associative

(rs)t = r(st) Concatenation is associative

r(s|t) = rs|rt(s|t)r = sr|tr

Concatenation distributes over |

Єr = rrє = r

Є is the identity element for concatenation

r* = (r|є)* Relation between * and єr** = r* * Is idempotent

The algebraic properties are used to manipulate regular expressions into equivalent forms.The following shows the algebraic laws that hold for regular expressions r , s and t.

Regular Definitions

• Regular Expressions are defined using names as if they were symbols.

• If Σ is an alphabet of basic symbols then a regular definition is a sequence of definitions of the form– d1 -> r1– d2 -> r2– …….– dn -> rn

Where each di is a distinct name and each ri is a regular expression over the symbols in Σ U {d1,d2, ….dn}

Example

• Example– Regular Definition for the IdentifierLetter -> A|B|….|Z|a|b|…|zDigit -> 0|1|…|9Id -> letter(letter|digit)*

Regular Expressions• Notational Shorthands Certain constructs occur so frequently in regular expression

that it is convenient to introduce notational shorthands as shown below1. One or more instances The unary postfix operator + - Precedence and associativity like *2. Zero or one instance The unary postfix operator ? 3. Character Classes : The notation [abc] where a, b and c are

alphabet symbols denotes the regular expression a|b|c. - using Character Classes we can describe identifiers as being strings

generated by the regular expression [A-Za-z][A-Za-z0-9]*

Example for using Notational shorthands

• Example• Unsigned numbers are strings such as 100 ,

100.15 , 7.453E3 The following regular definition can be written as per the notational shorthands.– Digit - > 0|1|….|9– Digits -> digit digit*– Optional_fraction -> .digits | є– Optional_exponent -> (E (+|-| є) digits ) | є– Num -> digits optional_fraction optional_exponent.

Recognition of Tokens

• Regular Expressions allow to specify the tokens

• How to recognize the tokens ?

Recognition of Tokens• Consider the following grammar

– Stmt -> if expr then stmt | if expr then stmt else stmt | є

- Expr -> term relop term | term- Term - > id | numIf , then ,else, relop, id , num are terminals that generate set of strings

given by following regular definitions if -> if then -> then else -> else relop -> < | <= | = | <> | > | >= id -> letter( letter | digit)* num -> digit+(. digit+ )?(E(+|-)? digit+ )?

- The goal of the lexical analyzer is to isolate the lexeme for the next token in the input buffer and produce as output a pair consisting of the appropriate token and attribute-value, using the translation table

Translation TableRegular Expression Token Attribute-Value

Ws - -

If If -

Then Then -

Else Else -

Id Id Pointer to symbol table entry

Num Num Pointer to symbol table entry

< Relop LT

<= Relop LE

= Relop EQ

<> Relop NE

> Relop GT

Transition Diagrams

• To Recognize the tokens it is required to view the actions that take place when a lexical analyzer is called by the parser to get the next token. To depict this actions a transition diagram is used.

• Example • TotalSal = basic + da;

t o t a l s a l = b a

A : Lexeme beginning pointerB: Forward pointer

A B

Input Buffer

About Transition Diagrams• A transition diagram keeps track of information of

characters that are seen as the forward pointer scans the input.

• This is done by moving from position to position in the diagram as characters are read.

• Positions in a transition diagram are called states and represented as circles.

• The states are connected by arrows called edges • Edges leaving state s have labels indicating the input

characters that can next appear after the transition diagram has reached state s

• The label “other” refers to any character that is not indicated by any of the other edges leaving s

Transition diagram Example• Transition diagram for the identifier

0 1 2letter

Letter or digit

other *

Exercise : draw transition diagram for unsigned numbers

Finite automata

• Finite automata– A recognizer for a language is a program that

takes as input a string x and answers yes, if x is a sentence of the language or no otherwise

– A regular expression is compiled into a recognizer by constructing a generalized transition diagram called a finite automaton.

– A finite automaton can be deterministic or nondeterministic.

NFA and DFA – A introduction

• Nondeterministic finite automaton means that more than one transition out of a state may be possible on the same input symbol

• Whereas in Deterministic finite automaton does not have this.

• Both can recognize exactly what regular expressions can denote.

About NFA

• NFA – It is a mathematical model that consists of

• A set of states• A set of symbols Σ (input symbol alphabet)• A transition function move that maps state-symbol

pairs to sets of states.• A state s0 that is distinguished as the start state• A set of states distinguished as final states (accepting

states)

• How to represent NFA- It can be represented diagrammatically by a

labeled directed graph called a transition graph in which nodes are states and the labeled edges represent the transition function.

- It looks like the transition diagram seem before.- In this diagram the same character can label two

or more transitions out of one state.- Edges can also be labeled by the special symbol є

as well as by the input symbols

Example - NFA

• Example – A transition graph for an NFA that recognizes the

language – (a|b)*ab1)Set of states : {0,1,2}2)Input symbols are {a,b}3)State 0 is the start state4)State 2 is the accepting state 5)In computer to describe the transition function of NFA

a transition table is used.

Transition tableState a b

0 {0,1} {0}

1 - {2}

-NFA accepts an input string x if and only if there is some path in the transition graph from the start state to some accepting state such that -edge labels along the path spell out x- A path can be represented by a sequence of state transitions called moves -Example : The following shows the moves made in accepting the input string aaab

-0 0 0 1 2- The language defined by an NFA is the set of strings it accepts.

aa a b

How to construct the NFA from the regular Expression?

• The algorithm is syntax directed in that it uses the syntactic structure of the regular expression to guide the NFA construction.

• Simple Steps of the algorithm• 1 : Construct the automata to recognize є and any symbol in the

alphabet.• 2: then construct the automata for expressions containing an

alternation , concatenation or kleene closure operation.• Example : for expression r|s ,we construct an NFA inductively from

the NFA’s for r and s.• For each step , two new states are introduces as the construction

proceeds. Thus NFA constructed for a regular expression has at most twice as many states as there are symbols and operators in the regular expression.

Algorithm -NFA from a regular Expression

• Algorithm - Thompsons Construction• Input : A regular expression r over an alphabet

Σ• Output ; an NFA accepting L(r).• Rules :Construct NFA for each basic symbol inr• 1) for є, construct the NFA. This NFA

recognizes {є }i f

start є

• 2) for a in Σ , construct the NFA.

This NFA recognizes {a}.- Then guided by the syntactic structure of r we combing

these NFA’s inductively using rule (3) till the NFA for the entire expression is obtd.

3) suppose N(s) and N(t) are NFA’s for the regular expressions s and t

a) Then for the regular expression s|t. construct the following composite NFA N(s|t).

fstart a

i

• This composite NFA recognizes L(s) U L(t)

i f

N(s)

N(t)

є

є є

є

• For the regular Expression st, construct the composite NFA N(st).

• This composite NFA recognizes L(s)L(t)

fN(s)start

iN(t)

• C) For the regular expression s*, construct the composite NFA N(s*)

• This composite NFA recognizes (L(s))*.• D) for the parenthesized regular expression (s)

use N(s) itself as the NFA.

i fN(s)start є

є

є

є

• Exercise : construct N(r) for the regular expression r = (a|b)*abb

• 1) For the constituent r1, the first a, we construct the NFA

Example

3start a

2

About Constructing NFA

• A new distinct name is given to every new state.

• Even if the same symbol appears several times in r, we create for each instance of that symbol a separate NFA with its own states

NFA properties

• During the construction of NFA , the NFA N(r) is produced with following properties

• 1) N(r) has at most twice as man states as the number of symbols and operators in r

• 2) N(r) has exactly one start state and one accepting state.

• 3) Each state of N(r) has either one outgoing transition on a symbol in Σ or at most two outgoing є-transitions.