xuan guo chapter 3: utilities for power users graham glass and king ables, unix for programmers and...
TRANSCRIPT
Xuan Guo CSc 3320 1
Chapter 3: Utilities for Power Users
Graham Glass and King Ables, UNIX for Programmers and Users,
Third Edition, Pearson Prentice Hall, 2003.
Xuan Guo CSc 3320 2
Regular Expression
Suppose we have a 10,000 lines text file, and we want to search words from the text file.
Query 1: Words in forms of “aa _ _ cc”Query 2: Words “atlanta” or “Atlanta” Query 3: Words consisting of more than three “yyy”
Xuan Guo CSc 3320 3
Regular Expression
Query1
Query 2
Query 3
Regular Expression
Engine
Regular Expression
Regular Expression
Regular Expression
Application
Xuan Guo CSc 3320 4
Regular Expression
1. Vi2. Sed, Awk, Grep3. Java, C#
http://www.zytrax.com/tech/web/regex.htm
Xuan Guo CSc 3320 5
Regular Expression
• Query 1: Words in forms of “aa _ _ cc”
• aa..cc
• Query 2: Words “atlanta” or “Atlanta” • [aA]tlanta (atlanta|Atlanta)
• Query 3: Words consisting of more than three “yyy”
• (y){3,}
Xuan Guo CSc 3320 6
More Example
[ab][a-z][A-Z][0-9]\d[^0-9][a-z 0-9](ae|bd)
a?a+a*(ab){3,5}(ab){3,}(ab){3}
Xuan Guo CSc 3320 7
Other Issues
1. Anchors ^, $2. Metacharacters[, ], {, }, \, ^, $, ?, *, +, ., (, )
Xuan Guo CSc 3320 8
Exercise
1 Which of the following matches regexp a(ab)*a 1) abababa
2) aaba
3) aabbaa
4) aba
5) aabababa
Xuan Guo CSc 3320 10
Exercise
3 Which of the following matches regexp a.[bc]+ 1) abc2) abbbbbbbb3) azc4) abcbcbcbc5) ac6) asccbbbbcbcccc
Xuan Guo CSc 3320 11
Exercise
4 Which of the following matches regexp (abc|xyz) 1) abc2) xyz3) abc|xyz
Xuan Guo CSc 3320 12
Exercise
5 Which of the following matches regexp [a-z]+[\.\?!] 1) battle!2) Hot3) green4) swamping.5) jump up.6) undulate?7) is.?
Xuan Guo CSc 3320 13
Exercise
6 Which of the following matches regexp [a-zA-Z]*[^,]= 1) Butt=2) BotHEr,=3) Ample4) FIdDlE7h=5) Brittle =6) Other.=
Xuan Guo CSc 3320 14
Exercise
7 Which of the following matches regexp [a-z][\.\?!]\s+[A-Z]
(\s matches any space character)1) A. B2) c! d3) e f4) g. H5) i? J6) k L
Xuan Guo CSc 3320 15
Exercise
8 Which of the following matches regexp
(very )+(fat )?(tall|ugly) man 1) very fat man2) fat tall man3) very very fat ugly man4) very very very tall man
Xuan Guo CSc 3320 16
Exercise
9 Which of the following matches regexp
<[^>]+> 1) <an xml tag>2) <opentag> <closetag>3) </closetag>4) <>5) <with attribute=”77”>
Xuan Guo CSc 3320 17
Answer
(1) 2, 5 (2) 1 (3) 1, 2, 3, 4, 6 (4) 1, 2 (5) 1, 4, 6 (6) 1, 5, 6 (7) 4, 5 (8) 3, 4 (9) 1, 3, 5
Xuan Guo CSc 3320 18
Basic Regular Expression & Extended Regular Expression
Meta-characters in Basic Regular Expression^ $ . * \( \) [ ] \{ \} \
vi, grep, sed accept basic regular expression.
Meta-characters in Extended Regular Expression
| ^ $ . * + ? ( ) [ ] { } \
egrep, grep –E, sed –E accept extended regular expression
Xuan Guo CSc 3320 19
Grep(Global or Get Regular Expression and Print)
• Filtering patterns: egrep, fgrep, grep– grep -hilnvw pattern {fileName}*– displays lines from files that match the pattern– pattern : regular expression
• -h : do not list file names if many files are specified• -i : ignore case• -l : displays list of files containing pattern• -n : display line numbers• -v : displays lines that do not match the pattern• -w : matches only whole words only
Xuan Guo CSc 3320 20
Grep variations
– fgrep : pattern must be fixed string– egrep : pattern can be extended regular expression
• -x option in fgrep: displays only lines that are exactly equal to string
– extended regular expressions:• + matches one or more of the single preceding
character• ? matches zero or one of the single preceding
character• | either or (ex. a* | b*)• () *, +, ? operate on entire subexpression not just on
preceding character; ex. (ab | ba)*
Xuan Guo CSc 3320 21
Differences
• grep Search a Pattern from current directory.
egrep (grep -E in linux) is extended grep where additional regular expression metacharacters have been added like +, ?, | and ().
fgrep (grep -F in linux) is fixed or fast grep and behaves as grep but does not recognize any regular expression metacharacters as being special.
Xuan Guo CSc 3320 22
48 Dec 3BC1997 LPSX 68.00 LVX2A 138 //line 1483 Sept 5AP1996 USP 65.00 LVX2C 189 //line 247 Oct 3ZL1998 LPSX 43.00 KVM9D 512 //line 3219 dec 2CC1999 CAD 23.00 PLV2C 68 //line 4484 nov 7PL1996 CAD 49.00 PLV2C 234 //line 5487 may 5PA1998 USP 37.00 KVM9D 644 //line 6471 May 7Zh1999 UDP 37.00 KV30D 643 // line 7
grep ”38$" exam1.dat
grep "^[^48]" exam1.dat
grep "[Mm]ay" exam1.dat
grep "K...D" exam1.dat
grep "[A-Z][A-Z][A-Z][9]D" exam1.dat
grep "9\{2,3\}" exam1.dat
Xuan Guo CSc 3320 23
Examples
grep “38$" exam1.datgrep "^[^48]" exam1.datgrep "[Mm]ay" exam1.datgrep "K...D" exam1.datgrep "[A-Z][A-Z][A-Z][9]D" exam1.datgrep "9\{2,3\}" exam1.dat
Xuan Guo CSc 3320 24
CSV file
• A CSV file consists of any number of record, separated by line breaks of some kind;
• each record consists of fields, separated by some other character or string, most commonly a literal comma or tab.
Xuan Guo CSc 3320 25
CSV files
Invent.dat
1. Pen 5 20.002. Pencil 10 2.003. Rubber 3 3.504. Cock 2 45.50
Xuan Guo CSc 3320 26
Pattern Scanning and Processing
• awk: utility that scans one or more files and performs an action on all lines that match a particular condition
• The conditions and actions are specified in an awk program.
• awk reads a line– breaks it into fields separated by tabs/spaces – or other separators specified by -F option
Xuan Guo CSc 3320 27
awk Command
• awk program has one or more commands:• awk [condition] [ \{ action \} ]• where condition is one of the following:
– special tokens BEGIN or END– an expression involving logical operators, relational
operators, and/or regular expressions
Xuan Guo CSc 3320 28
awk Command
• awk [condition] [ \{ action \} ]• action is one of the following kinds of C-like
statements– if-else; while; for; break; continue– assignment statement: var=expression– print; printf;– next (skip remaining patterns on current line)– exit (skips the rest of the current line)– list of statements
Xuan Guo CSc 3320 29
awk Command
• accessing individual fields: – $1, ..., $n refer to fields 1 thru n– $0 refers to entire line
• built-in variable NF means number of fields• % awk -F: '{ print NF, $1 }' /etc/passwd• prints the number of fields and the first field in
the /etc/passwd file• -F: means to use : as the field separator
Xuan Guo CSc 3320 30
awk Command
• BEGIN condition triggered before first line read • END condition triggered after last line read• FILENAME: built-in variable for name of file
being processed• We will use this data in following examples:
Xuan Guo CSc 3320 31
awk Example
Serial NO Product Quantity Unit Price
1 Pen 5 20.00
2 Rubber 10 2.00
3 Pencil 3 3.50
4 Cock 2 45.50
$1 $2 $3 $4
“invent.dat”
Xuan Guo CSc 3320 32
awk Example
1.Print the name of each product
awk ‘{print $2}’ invent.dat
Pen
Pencil
Rubber
Cock
Xuan Guo CSc 3320 33
awk Example
2.Print the name of each product and its unit price
awk ‘{print $2”>>”$4}’ invent.dat
Pen>>20.00
Pencil>>2.00
Rubber>>3.50
Cock>>45.50
Xuan Guo CSc 3320 34
awk Example
3.Print each line
awk ‘{print $0}’ invent.dat
1. Pen 5 20.00
2. Pencil 10 2.00
3. Rubber 3 3.50
4. Cock 2 45.50
Xuan Guo CSc 3320 35
awk Example
4. Print the name and unit price of the products whose quantity are greater than 5
awk ‘ $3>=5 {print $2 “>>” $4}’ invent.dat
Pen>>20.00
Pencil>>2.00
Xuan Guo CSc 3320 36
awk Example
5. Print the name and unit price of the products which contain the word “Pen”
awk ‘ /Pen/ {print $2 “>>” $4}’ invent.dat
Pen>>20.00
Pencil>>2.00
Xuan Guo CSc 3320 37
awk predefined variables
Variable Example
FILENAME name of file being processed
Invent.dat
RS New line
FS whitespace
NF number of fields 4
NR current line #
Xuan Guo CSc 3320 38
awk Example
awk '{print FILENAME;print NR}' invent.dat
invent.dat
1
invent.dat
2
invent.dat
3
invent.dat
4
Xuan Guo CSc 3320 39
awk Example
Compute the overall value of these product
1. Pen 5 20.002. Pencil 10 2.003. Rubber 3 3.504. Cock 2 45.50
Xuan Guo CSc 3320 40
awk Example
BEGIN {
print "---------------------------"
print "BEGIN section is only printed once.“
print "==========================="
}
Xuan Guo CSc 3320 41
awk Example
{
total = $3 * $4
recno = $1
item = $2
gtotal += total
printf "%d %s Rs.%f\n", recno, item, total
}
Xuan Guo CSc 3320 42
awk Example
END {
print "---------------------------"
printf "Total Rs. %f\n" ,gtotal
print "END section is only printed once."
print "==========================="
}
Xuan Guo CSc 3320 44
---------------------------
BEGIN section is only printed once.
===========================
1 Pen Rs.100.000000
2 Pencil Rs.20.000000
3 Rubber Rs.10.500000
4 Cock Rs.91.000000
---------------------------
Total Rs. 221.500000
END section is only printed once.
===========================
Xuan Guo CSc 3320 45
awk actions
Built-in functions: exp(), log(), sqrt(), substr() etc.
If condition, for loop, while loop
Xuan Guo CSc 3320 46
awk another example
% cat /etc/passwdnobody:*:-2:-2:Unprivileged User:/:/usr/bin/falseroot:*:0:0:System Administrator:/var/root:/bin/sh...lp:*:26:26:Printing Services:/var/spool/cups:/usr/bin/false
Xuan Guo CSc 3320 47
awk Example
% cat p2.awk BEGIN { print "Start of file: "} { print $1 " " $6 " " $7 } END { print "End of file", FILENAME }
% awk -F: -f p2.awk /etc/passwdStart of file: nobody / /usr/bin/falseroot /var/root /bin/sh...lp /var/spool/cups /usr/bin/falseEnd of file /etc/passwd
Xuan Guo CSc 3320 48
awk Operators
• built-in variable NR contains current line #• remember, “-F:” uses colon as separator
% cat p3.awk NR > 1 && NR < 4 { print NR, $1, $6, NF }
% awk -F: -f p3.awk /etc/passwd 2 root /var/root /bin/sh 73 daemon /var/root /usr/bin/false 7
Xuan Guo CSc 3320 49
awk Variables
% cat p4.awk BEGIN {print "Scanning file"}{ printf "line %d: %s\n", NR, $0 lineCount++; wordCount += NF; }END { printf "lines = %d, words = %d\n", lineCount, wordCount }
% awk -f p4.awk /etc/passwdScanning fileline 1: nobody:*:-2:-2:Unprivileged User:/:/usr/bin/falseline 2: root:*:0:0:System Administrator:/var/root:/bin/sh...line 37: lp:*:26:26:Printing Services:/var/spool/cups:/usr/bin/falselines = 37, words = 141
Xuan Guo CSc 3320 50
awk Control Structures
% cat p5.awk
{ for (i = NF; i >= 1; i--) printf "%s ", $i; printf "\n";}
% awk -f p5.awk /etc/passwdUser:/:/usr/bin/false nobody:*:-2:-2:Unprivileged Administrator:/var/root:/bin/sh root:*:0:0:System ... Services:/var/spool/cups:/usr/bin/false lp:*:26:26:Printing
Xuan Guo CSc 3320 51
awk Condition Ranges
• Condition ranges: – two expressions separated by comma
• awk performs action on every line – from the first line that matches first expression – until line that matches second condition
% awk -F: ' /nobody/,/root/ {print $0}' /etc/passwdnobody:*:-2:-2:Unprivileged User:/:/usr/bin/falseroot:*:0:0:System Administrator:/var/root:/bin/sh
Xuan Guo CSc 3320 52
awk Built-in Functions
• Built-in functions: – exp()– log()– sqrt()– substr() etc.
% awk -F: '{print substr($1,1,2)}' /etc/passwdnoro...lp
Xuan Guo CSc 3320 53
Stream Editor (sed)
• sed – scans one or more text files– performs an edit on all lines that match a condition– actions and conditions may be stored in a file– may be specified at command line in single quotes– commands begin with an address or an
addressRange or a Regular expression– does not modify the input file– writes modified file to standard output
Xuan Guo CSc 3320 54
Sed syntax
• sed -option 'general expression' [data-file]Replace words action:s/old pattern/new pattern/
Delete lines action:/pattern/d
Xuan Guo CSc 3320 55
Sed syntax
• sed -option 'general expression' [data-file]Search action:-n /pattern/p
Xuan Guo CSc 3320 56
Paris PS1 Charles Chin 01/20/86 30Ind PS2 Susan Green 04/05/86 32SUST PS2 Lewis SUST 08/11/85 23JUST IS1 Xiao Ming 11/30/84 9HEBUT IS1 John Main 12/03/84 8SUST PS2 Da Ming 06/01/86 35Paris IS3 Peter Webor 07/05/82 32Paris PS2 Ann Sreph 09/28/85 10Paris IS3 Margot Strong 02/29/82 9
Xuan Guo CSc 3320 57
Examples
Search lines that starts with HEBUTsed -n ’/^HEBUT/p' studentssed ’/^HEBUT/p' students // NOT GOOD
HEBUT IS1 John Main 12/03/84 8
Xuan Guo CSc 3320 59
Paris PS1 Charles Chin 01/20/86 30Ind PS2 Susan Green 04/05/86 32SDUST PS2 Lewis SUST 08/11/85 23JUST IS1 Xiao Ming 11/30/84 9HEBUT IS1 John Main 12/03/84 8SDUST PS2 Da Ming 06/01/86 35Paris IS3 Peter Webor 07/05/82 32Paris PS2 Ann Sreph 09/28/85 10Paris IS3 Margot Strong 02/29/82 9
Xuan Guo CSc 3320 61
Examples
Delete lines that contain “../../86”
sed ‘/..\/..\/86/d’ students
• % sed 's/^/ /' file > file.new– indents each line in the file by 2 spaces
• % sed 's/^ *//' file > file.new– removes all leading spaces from each line of the file
• % sed '/a/d' file > file.new– deletes all lines containing 'a'
Xuan Guo CSc 3320 62
Ranges by patterns
You can specify two regular expressions as the range.
Assuming a "#" starts a comment, you can search for a keyword, remove all comments until you see the second keyword.
In this case the two keywords are "start" and "stop:" sed '/start/,/stop/ s/#.*//'
The first pattern turns on a flag that tells sed to perform the substitute command on every line. The second pattern turns off the flag.
If the "start" and "stop" pattern occurs twice, the substitution is done both times.
If the "stop" pattern is missing, the flag is never turned off, and the substitution will be performed on every line until the end of the file.