LINUX and UNIX Shell Programming

Acknowledgements

xviii

Introduction

xix

The Bourne shell

Shell portability

Organization of the book

The shell

xxi

Text file filtering

xxi

Basic shell programming

xxi

Better scripting skills

xxii

Assumptions

xxii

Conventions used in this book

xxii

PART 1 The shell

(68)

File security and permissions

(18)

Files

(1)

Types of files

(1)

Permissions

(2)

Changing permission bits

(4)

Symbolic mode

(1)

chmod examples

(1)

Absolute mode

(1)

More chmod examples

(1)

Absolute or symbolic, your choice

(1)

Directories

(1)

Set-uid

(2)

Why use set-uid?

(1)

Adding set-uid permission examples

(1)

chown and chgrp

(2)

chown examples

(1)

chgrp examples

(1)

Finding out what groups you belong to

(1)

Finding out what groups other users belong to

(1)

umask

(2)

How to work out umask values

(1)

Common umask values

(1)

Symbolic (soft) links

(2)

Using soft links saves multiples copies of files

(1)

Example of using a soft link

(1)

Conclusion

(1)

Using find and xargs

(11)

find options

(8)

Find files by name

(1)

Find files by perm mode

(1)

Ignoring directories

(1)

Find files by user and nouser

(1)

Find files by group and nogroup

(1)

Find files by modification times

(1)

Finding files that are newer or older in days or minutes

(1)

Find files by type

(1)

Find files by size

(1)

Find files by depth

(1)

Find files by mount

(1)

Find files using cpio

(1)

Using exec or OK to run shell commands

(1)

Find examples

(1)

Xargs

(1)

Conclusion

(1)

Running commands unattended

(12)

Cron and crontab

(4)

crontab fields

(1)

crontab entry examples

(1)

crontab options

(1)

Creating a new crontab entry

(1)

Listing your crontab

(1)

Editing your crontab

(1)

Removing your crontab

(1)

Restoring a lost crontab

(1)

The at command

(3)

Submitting commands and scripts using at

(1)

Listing jobs submitted

(1)

Deleting a submitted job

(1)

The & command

(2)

Submitting a command to the background

(1)

Checking the process using ps

(1)

Killing a background job

(1)

The nohup command

(1)

Submitting a job using nohup

(1)

Submitting several jobs at once

(1)

Conclusion

(1)

Filename substitution

(4)

Using the *

(1)

Using the ?

(1)

Using [ . . . ] and [! . . .]

(1)

Conclusion

(1)

Shell input and output

(16)

echo

(3)

read

(1)

cat

(2)

Pipes

(1)

tee

(1)

Standard input, output and errors

(1)

Standard input

(1)

Standard output

(1)

Standard error

(1)

File redirection

(4)

Redirecting standard output

(1)

Redirecting standard input

(1)

Redirecting standard error

(1)

Combining standard output and error

(1)

Merging standard output and standard error

(1)

exec

(1)

Using file descriptors

(1)

Conclusion

(1)

Command execution order

(5)

Using & &

(1)

Using | |

(1)

Grouping commands using ( ) and { }

(1)

Conclusion

(2)

PART 2 Text filtering

(116)

Introducing regular expressions

(11)

Matching a single character with a period

(1)

Matching a string or character at the beginning of a line with a ∧

(1)

Matching a string or character at the end of a line with $

(1)

Matching a single or string of characters with *

(1)

Escaping the meaning of a special character using \

(1)

Matching ranges or sets using [ ]

(2)

Matching a specific number of occurrences with \{ \}

(3)

Conclusion

(1)

The grep family

(14)

grep

(3)

Please quote me

(1)

grep options

(1)

Searching more than one file

(1)

Line matches

(1)

Line numbers

(1)

Do not match

(1)

Getting an exact match

(1)

Being case-sensitive

(1)

grep and regular expressions

(4)

Pattern ranges

(1)

Don't match at the beginning of a line

(1)

Trapping upper and lower cases

(1)

Matching any characters

(1)

Date searching

(1)

Combining ranges

(1)

Occurrences in a pattern

(1)

Using grep to match either or all patterns

(1)

Blank lines

(1)

Matching special characters

(1)

Searching for formatted filenames

(1)

Searching for ip addresses

(1)

Class names

(2)

Pattern matching with wildcards

(1)

grep on the system

(2)

Directories

(1)

passwd file

(1)

Using the ps command

(1)

Using grep on a string

(1)

egrep

(1)

Conclusion

(1)

Introducing awk

(35)

Calling awk

(1)

awk script

(32)

Patterns and actions

(1)

Fields and records

(4)

Regular expressions and operators in awk

103

(1)

Metacharacters

103

(1)

Conditional operators

103

(5)

awk built-in variables

108

(1)

NF, NR and FILENAME

109

(1)

awk operators

110

(4)

Built-in string functions

114

(4)

String escape sequences

118

(1)

awk output functions (printf)

119

(1)

print modifiers

119

(7)

awk arrays

126

(4)

Conclusion

130

(1)

Using sed

131

(23)

How sed reads data

132

(1)

Calling sed

132

(2)

Saving sed output

133

(1)

Ways to find text in a file using sed

133

(1)

Basic sed editing commands

133

(1)

sed and regular expressions

134

(1)

Basic sed editing examples

134

(9)

Displaying lines using p(ring)

134

(1)

Range printing

135

(1)

Pattern printing

135

(1)

Searching using a pattern and line number

135

(1)

Matching a metacharacter

136

(1)

Displaying the whole file

136

(1)

Any characters

136

(1)

First line

136

(1)

Last line

137

(1)

Printing line numbers

137

(1)

Appending text

137

(1)

Creating a sed script file

138

(1)

Inserting text

139

(1)

Changing text

140

(1)

Deleting text

141

(1)

Substituting text

142

(1)

Modifying strings with substitute (&)

143

(1)

Writing to a file within sed

144

(1)

Reading text from a file

144

(1)

Quitting after a match

145

(1)

Displaying control characters in a file

146

(1)

Using sed on the system

147

(4)

Dealing with control characters

147

(2)

Dealing with report output

149

(1)

Getting rid of numbers at the beginning of a line

149

(1)

Appending text

150

(1)

Passing values from the shell to sed

150

(1)

Assigning a shell variable from sed output

151

(1)

Quick one-liners

151

(2)

Conclusion

153

(1)

Merge and divide

154

(22)

Using sort

154

(9)

A word to the wise

155

(1)

sort options

155

(1)

Saving output

156

(1)

How sort starts

156

(1)

How sort references fields

156

(1)

Is the file already sorted?

157

(1)

Basic sort

157

(1)

Reverse sort

157

(1)

Sorting on a specific field

158

(1)

Numeric field sorting

158

(1)

Unique sorting

159

(1)

Other sort methods using - k

159

(1)

Order of sort keys using - k

160

(1)

Specifying sort sequences

160

(1)

Using pos

160

(1)

Sort output using head and tail

161

(1)

Using sort output with awk

162

(1)

Merging two sorted files

162

(1)

sort on the system

163

(1)

Using uniq

164

(2)

Occurrences

165

(1)

Using join

166

(3)

Joining two files

167

(2)

Using cut

169

(2)

Using the field separator

170

(1)

Cutting specific fields

170

(1)

Using paste

171

(2)

Specifying columns

172

(1)

Using a different field separator

172

(1)

Piping into a paste command

173

(1)

Using split

173

(2)

Conclusion

175

(1)

Using tr

176

(9)

About tr

176

(7)

Character ranges

177

(1)

Saving output

178

(1)

Getting rid of repeated characters

178

(1)

Deleting blank lines

178

(1)

Upper to lower case

179

(1)

Lower to upper case

180

(1)

Deleting certain characters

180

(1)

Translating control characters

181

(1)

Quick conversions

182

(1)

Matching more than one character

183

(1)

Conclusion

183

(2)

PART 3 The login environment

185

(38)

The login environment

187

(10)

The /etc/ profile

188

(3)

Your $HOME .profile

191

(2)

Using stty

193

(2)

Creating a .logout file

195

(1)

Conclusion

196

(1)

Environment and shell variables

197

(20)

What is a shell variable?

197

(1)

Local variables

198

(5)

Displaying a variable

199

(1)

Clearing a variable

199

(1)

Displaying all local shell variables

200

(1)

Variable values together

200

(1)

Testing if a variable is set (substitution)

200

(2)

Using variables to hold arguments for system commands

202

(1)

Making a variable read-only

202

(1)

Environment variables

203

(8)

To assign an environment variable

204

(1)

Displaying environment variables

204

(1)

Clearing environment variables

204

(1)

Built-in shell variables

205

(3)

Other environment variables

208

(1)

Using the set command

209

(1)

Exporting variables to child processes

210

(1)

Positional variable parameters

211

(5)

Using positional parameters in scripts

212

(1)

Passing parameters to system commands

213

(1)

Special variable parameters

213

(2)

Last exit status

215

(1)

Conclusion

216

(1)

Quoting

217

(6)

The need for quoting

217

(1)

Double quotes

218

(1)

Single quotes

219

(1)

Back quote

219

(1)

Backslash

220

(2)

Conclusion

222

(1)

PART 4 Basic shell programming

223

(170)

Introduction to shell scripts

225

(4)

Reasons for using a shell script

225

(1)

Try out new ideas

226

(1)

What a script contains

226

(1)

Running a script

226

(2)

Conclusion

228

(1)

Conditional testing

229

(8)

Testing for a file status

229

(2)

Using logical operators with tests

231

(1)

Testing strings

232

(1)

Testing numbers

233

(1)

Using expr

234

(2)

Incrementing counters

235

(1)

Testing for a number

235

(1)

Pattern matching

236

(1)

Conclusion

236

(1)

Control flow structures

237

(54)

Exit statuses

237

(1)

Control structures

238

(2)

Flow Control

239

(1)

Iteration

239

(1)

if then else statements

240

(17)

Simple if statements

240

(1)

Testing values of variables

241

(1)

Checking the output of grep

242

(1)

Testing the output of grep on a variable

242

(1)

Checking the outcome of a file copy

243

(1)

Testing for current directory

244

(1)

Testing for file permissions

244

(1)

Testing for parameters passed to a script

244

(1)

Determining if a script is in interactive mode

245

(1)

Simple if else statements

246

(1)

Testing for set variables

246

(1)

Checking the user who is running a script

246

(1)

Using script parameters to pass down to a system command

247

(1)

Using the null: command

248

(1)

Testing the outcome of a directory creation

248

(1)

Another copy

249

(1)

More than one if statement

250

(1)

Testing and setting environment variables

250

(2)

Checking the last command status

252

(1)

Adding and checking integer values

252

(2)

Simple security login script

254

(1)

Using elif

255

(1)

Multiple checks using elif

255

(1)

Checking multiple file locations

256

(1)

Case statement

257

(7)

Simple case statement

258

(1)

Using | with pattern matching

259

(1)

Prompting for a y or n

260

(1)

Case and passing command parameters

260

(1)

Trapping input with no pattern commands

261

(2)

Default variable values

263

(1)

for loop

264

(7)

Simple for loop

264

(1)

Printing out a string list

265

(1)

Using Is with the for loop

265

(1)

Using parameters with a for loop

266

(1)

Pinging servers with a for loop

267

(1)

Backing up files using a for loop

268

(1)

Multiple translation

268

(1)

Multiple sed deletes

268

(1)

Counting iterations

269

(1)

for loops and here documents

269

(1)

Nested for loops

270

(1)

until loop

271

(2)

Simple until loop

272

(1)

Monitoring for a file presence

272

(1)

Monitoring disk space

273

(1)

while loop

273

(12)

Simple while loop

274

(1)

Using the while loop to read from the keyboard

274

(1)

Using the while loop to read data from files

275

(1)

Reading files using IFS

275

(1)

Processing a file with conditional tests

276

(2)

Running totals

278

(1)

Reading a pair of records at a time

279

(2)

Ignoring the # character

281

(1)

Processing formatted reports

282

(2)

The while loop and file descriptors

284

(1)

Controlling loops using break and continue

285

(2)

break

285

(1)

Breaking out of a case statement

285

(1)

Continue

286

(1)

Skipping lines of files

286

(1)

Menus

287

(3)

Conclusion

290

(1)

Shell functions

291

(29)

Declaring functions in a script

292

(1)

Using functions in a script

292

(1)

Passing parameters to a function

293

(1)

Returning from a called function

293

(1)

Testing the values returned by a function

294

(1)

Using functions in the shell

295

(1)

Creating a function file

295

(1)

Sourcing the file

296

(1)

Checking the loaded functions

296

(1)

Executing shell functions

296

(18)

Deleting shell functions

297

(1)

Editing shell functions

297

(1)

Function examples

298

(16)

Putting it all together

314

(1)

Calling functions

314

(3)

Calling functions inside a script

315

(1)

Calling functions from a function file

316

(1)

Sourcing files is not only for functions

317

(2)

Conclusion

319

(1)

Passing parameters to scripts

320

(15)

The shift command

321

(6)

Simple use of shift

322

(1)

Last parameter supplied on the command line

322

(1)

Using shift to process file conversions

322

(5)

getopts

327

(7)

An example getopts script

327

(2)

How getopts works

329

(1)

Specifying values using getopts

329

(2)

How values can be accessed

331

(1)

Using getopts to process file conversions

332

(2)

Conclusion

334

(1)

Creating screen output

335

(21)

tput

335

(2)

String output

336

(1)

Numeric output

336

(1)

Boolean output

336

(1)

Using tput

337

(18)

Assigning tput commands

337

(1)

Using Boolean output

337

(1)

Using tput in your scripts

337

(1)

Generating escape sequences

338

(2)

Cursor position

340

(1)

Centring text on the screen

341

(1)

Finding out your terminal attributes

341

(2)

Using function keys with your scripts

343

(1)

Using colours

344

(1)

Generating colours

344

(4)

Creating better menus

348

(7)

Conclusion

355

(1)

Creating screen input

356

(27)

Adding records

357

(11)

Deleting records

368

(6)

Amending records

374

(4)

Viewing records

378

(4)

Conclusion

382

(1)

Debugging scripts

383

(4)

Common errors

383

(2)

The classic missing quote

384

(1)

The test error

384

(1)

Character case

384

(1)

for loops

384

(1)

echo

385

(1)

The set command

385

(1)

Conclusion

386

(1)

Shell built-in commands

387

(6)

Complete list of shell built-in commands

387

(4)

pwd

388

(1)

set

388

(1)

times

389

(1)

type

389

(1)

ulimit

390

(1)

wait

391

(1)

Conclusion

391

(2)

PART 5 Better scripting skills

393

(90)

Going further with here documents

395

(10)

Creating a quick file

396

(1)

Creating a quick print document

396

(1)

Automating menus

396

(2)

Automating ftp transfers

398

(4)

Accessing databases

402

(2)

Conclusion

404

(1)

Shell utilities

405

(21)

Creating hold files

405

(3)

Using date to create log files

405

(2)

Creating unique temporary files

407

(1)

Signals

408

(3)

Killing a process

409

(2)

Detecting a signal

411

(1)

trap

411

(8)

Trapping signals and taking action

412

(1)

Catching a signal and taking action

413

(2)

Locking up your terminal

415

(2)

Ignoring signals

417

(2)

eval

419

(3)

Executing commands held in a string

419

(2)

Making a value a variable name

421

(1)

logger command

422

(3)

Using the logger command

423

(1)

Using logger in your scripts

423

(2)

Conclusion

425

(1)

A small collection of scripts

426

(15)

pingall

426

(1)

backup-gen

427

(6)

del.lines

433

(2)

access.deny

435

(3)

logroll

438

(1)

nfsdown

439

(1)

Conclusion

440

(1)

Run level scripts

441

(10)

How can you tell if you have run level directories?

442

(1)

Finding out your current run level

443

(1)

Bringing you up to speed on inittab

443

(1)

Now for the run levels

444

(5)

The different run levels

445

(1)

Format of a run level script

446

(1)

Installing a run level script

447

(2)

Using inittab to launch applications

449

(1)

Other methods of starting and shutting down services

449

(1)

Conclusion

450

(1)

cgi scripts

451

(32)

What is a Web page?

452

(1)

cgi

452

(1)

Connecting to a Web server

452

(1)

cgi and HTM script

453

(11)

Basic cgi script

453

(3)

Displaying the output of a shell command

456

(1)

Using SSI

457

(1)

Access counter

458

(2)

Printing out current Web environment settings using a link

460

(1)

Other common environment variables

461

(3)

Introducing the get and post methods

464

(18)

get method

464

(7)

post method

471

(8)

Populating a list

479

(1)

Automatically refreshing a Web page

480

(2)

Conclusion

482

(1)

Appendix A ASCII chart

483

(4)

Appendix B Useful shell commands

487

(14)

Index

501

Introduction This book is about shell programming or to be more precise shell and Bourne shell programming. With LINUX now firmly taking its rightful place in the market as a viable and robust operating system, shell programming has become even more popular. No-one can say with any confidence how many LINUX users there are out there because LINUX is free, though there is a growing number of third parties selling their own LINUX variants. UNIX is still as popular as ever and growing despite a forecasted demise by IT consultants a few years ago. If you want to learn shell programming, then this is the book for you; even if you are an intermediate shell user you will find the book a good learning and reference tool as there are some handy administrative tips and one-liners for you. This book has been written from the outset with six main objectives in mind. These are: Getting the reader up to speed quickly learning about shell tools and shell programming. Using the book not only as a learning tool but also as a reference book. Using shell scripts to use your system more productively. Showing scripts that are clear and easily understandable. Making the chapters self-contained where possible for the subject covered, for ease of use. Showing the reader not just shell scripting but also some administrative tasks like rc.scripts and cgi scripts. One of the annoying things about certain books on this topic was how some scripting examples were made to look complicated just to save a few lines of extra code. You won't find that sort of thing happening in this book: all the code in this book is simple but effective. If you're reading this, then you have probably already got your own reasons for learning shell programming; great. If you want to know the main reasons for learning shell programming they are: The shell is a programming language all on its own; it has iteration, conditions and testing constructs and it's easy to use. You can create scripts quickly. You can automate boring manual tasks using scripts. The Bourne shell The Bourne shell is the standard shell shipped with all UNIX systems and is linked into the LINUX bash shell. A book about shell programming that covers all leading systems must cover the Bourne shell. There are other shells, mind you, like bash, Korn, and the C shell. If you are familiar with the bash shell, the scripts presented in this book will run on your system, because bash is backward-compatible with Bourne. If you have the Korn shell, the syntax of the shell is very close. If you look at installation scripts you will find that over 95 per cent of them are Bourne shell scripts. This is simply because the person who wrote the script knows it will run on any UNIX or LINUX system. Shell portability When you write a script that you want to run on any system it has to be what is called portable. Portability of scripts has two major issues: script syntax of the shell you are using; shell commands. The first issue is practically already solved if you are going to program with the Bourne shell: it has few or no portability problems. Most shell scripts spend at least 20 per cent (or probably more) of their time using shell commands such as cp, mv, mkdir etc. This is where the problems of portability come in. Different system vendors have different command options on their machines; coupled with this if you have UNIX then you are either a Systemiacute;V or BSD user. This book uses generic scripts and command examples. I have only used options and commands that are common to both Systemiacute;V and BSD and in my