Shells and Shell Scripting
U. C. L. A. Extension - X417.31
see syllabus for email address
This Website (http://uclaextension.dmorgan.us/morgan_david/unex.html) will be used extensively to communicate with you. Announcements, grade reports, and assignments will be posted here. The site can be viewed from an internet-connected browser anywhere. You are responsible for awareness of the information posted here.
Course outline - with approximate weekly topic coverage. All corresponing readings, , homework assignments, and in-class slides I will use are here. You can preview or review the slide lessons at any time or view them while I am presenting them in class.
Syllabus - containing my email
address, specification of required textbook, recommendation of other
useful books, and some policy information.
Thank you - for taking this course and giving me Tuesday evenings to look forward to. (11/27)
Final exam - no test (11/27)
Related linux/unix classes - that may interest you.
Schedule: the courses tend to be offered every other quarter. There may be variations but this is the general expectation. If you are interested please "stay tuned" to the class schedule as UCLA Extension publishes it. Or if you email me I can let you know what we are putting in the pipeline. None of these is offered for Winter 2019 however.
Homework - textbook awk labs 3 and
4 (on pages 187-88 and 202, respectively).
Three good awk resources - in class we talk about awk. We spend time on course outline section 9's exercises at the links entitled "extracting classic AWK book's sample scripts" and "sort multi-line blocks with AWK." These are based on the classic book The AWK Programming Language, which is rich in small examples. Another good source of instructive small examples is Eric Pement's awk one-liners document. Also in section 9 is a related link titled "AWK one-liners." Our textbook is a third great source on awk, note the link below to the book's practice files that enable you to perform the many examples in the awk chapter (chap. 6) as you read through them. (11/20)
Grades - published at link entitled "Grade reports" at left. (11/20)
Upcoming topics - mostly awk
-- programming features of bash
(textbook ch 14)
-- awk (textbook ch 6)
lab 3, on p. 187-88
Labs 3 and 4 challenge you to produce 1-liners that do specified things with given data. Labs 5 and 6 require you to write short but full programs that produce formatted reports from given data. Try them all. Doing so requires reading the material preceding each lab, and takes some time and effort. We will work on and discuss them in the next 2 class meetings. I will decide which to ask you to formally turn in as homework. (11/20)
Practice files from Quigley chapter 6 (awk). (11/20)
awk one-liners - from the same gentleman, Eric Pement, who brought us the sed one-liners. (11/20)
an interesting awk exercise in class - next class. It's the one at the course outline link "example scripts from classic AWK book." It gives you a file named awkcode.txt that contains about 200 sample awk scripts. The interesting thing is the way it delivers all these little sample scripts. Normally this sort of thing is done with zip or tar files. Here, it's all done using awk itself. Take a look at the "awkcode.txt" file, its structure, and its first 4 lines. I want you to "extract" the 200 files from it in class. I will show you the model for doing so in lecture, and the elements of the awk language involved. (11/20)
Quoting text - a couple reasons to do it
1a) make it a single token
[root@unexgate sed]# cat show-args
1b) make it a single token
[root@unexgate ~]# cat holidays
[root@unexgate sed]# VAR=something
Many small specialists - according to the 2nd slide we viewed in this course programs in Unix are supposed to "do one thing well." In that vein I noted this:
"One of the great contributions of Unix to the computing world is that process creation is cheap and easy. This encourages the practice of writing small programs that each do a part of a larger job, and then combining them to collaborate on the completion of that task. Because programming complexity grows much faster than linearly with program size, small programs are much easier to write, debug, and understand than large ones." Shell Scripting, Robbins & Beebe, p353
It can be argued, in relative terms, whether process creation is cheap. Famously for example, threads are useful because they are so much cheaper than processes. But compared to historical and competing operating systems Unix's fork/exec is cheap and, to the point, "cheap enough." That removes the barrier to hitching (perhaps pipelining) a team of small programs together in place of a monolithic one. (11/6)
Grades - published at link entitled "Grade reports" at left. (10/30)
Grades - published at link entitled "Grade reports" at left. (10/23)
Command line fu - have fu. (just for fun) (10/23)
Regular expression humor - you think you got problems? (10/19)
A comment about interpreter scripts' interpreters and how they handle their file argument
What happens in an interpreter script to the lines after the initial shebang (first), if the shebang is " #!/bin/mv " as opposed to " #!/bin/bash " ? Suppose the script file's name is myscript, and it is called as:
Then an exec( ) function call will be constructed that is the same as would be constructed if you gave the following commands:
/bin/mv myscript argument
(as opposed to, /bin/bash myscript argument )
Then, what happens to the lines in the myscript file depends on how the commands (mv and bash) treat their first argument. mv takes it as the name of a file to rename. In doing so it does not delve into the file's content at all. bash takes it as the name of a file over whose contents, viewed as lines, it must iterate in a loop. When called, bash is content-conscious and mv is content-oblivious when a file's name shows up as their first argument. That's why mv is not hindered by a file with multiple lines and just renames it same as if the file held just the single shebang line only. (10/15)
a bash "exec" redirection use case
read them out:
read them out, stopping after each line for user go-ahead:
Does not work. Instead:
Arranges for the 2 consumers of input (the two "read"s) to get it from different places.
Another: you want the output of entire portions of a script (but not of the whole script) captured into a file. The script has many individual lines within it that generate output. You know how to put file redirection ( >> ) on each line, but there are too many lines. At the top of the portion(s) whose output you want diverted, divert it. At the bottom, undivert it:
echo put \"blue suede shoes\" in a file
Different "echo"s - I distinguished between the built-in, shell-resident code for "echo" and the binary-file-resident one. They are two different things, though they both do largely the same thing. This note appears in the echo man page, which pertains to the binary-file echo:
"NOTE: your shell may have its own version of echo, which usually
This echo was written by Brian Fox and Chet Ramey, the ones who wrote bash itself and hence presumably its built-in echo. No doubt therefore they are highly similar. The echo's that may be built in to the korn shell or c shell will be different, and those found as binary executables in other environments may be different too, written by somebody else yet again. echo is one of the commands that is least standardized, hence least portable. Many people suggest avoiding echo and relying on the printf command instead, whose operational behavior and syntax is more uniformly standardized. (10/9)
Warning - the "datebook" file on the textbook's CD may be corrupted. Some editions deliver a file that uses Microsoft line termination convention. That is, they separate lines with hex 0D 0A character pairs, instead of the single character 0A that unix utilities expect.
You need to fix the file, or you will get wrong results when doing the homework. There is a utility for this purpose called dos2unix. You could fix the file by:
If you don't have dos2unix, another way to do it is with tr (the "translate" command). Something like:
cat datebook | tr -d "\r" >
And a third way uses sed:
sed -i 's/.$//' datebook
which says, replace the character before the end of the line with nothing. The 0A being interpreted as "the end of the line," that character is its preceding 0D. Replacing it with nothing is getting rid of it. The -i option applies the changes to the original file, in-place.
Applying the correction to an uncorrupted file is harmless, it leaves it uncorrupted. So you might, conservatively, do it on your "datebook" file just in case. Alternatively, download it from the link entitled "Practice files" below. It delivers a zip file, and the files inside the zip are good. (10/9)
Clarification about file descriptor representation - A former student and I talked about file descriptors. It led me to realize that my slides and many sources represent the descriptor table in a potentially confusing way (below, uppper left). The student thought that stdin represents what descriptor 0 connects to, just as /somefile represents what descriptor 3 connects to. They are presented in exactly the same way, after all. No, descriptor 0 doesn't connect to stdin. Nothing connects to stdin. There is no stdin to connect to, since it's nothing but a nickname or synonym for 0. As well, stdout and stderr are synonyms for 1 and 2. Whatever you choose to call them, what they connect to must be some material entity, like a device or file. It makes no sense that they would connect to their name! Better, my revised table below upper right. Princeton Professor Rexford's slides (below) used a good representation in that she shows only the descriptors and their devices, omitting names altogether. The names are an unneeded distraction.
The definitive list of regular expression
metacharacters - is in your textbook.
When you talk about regular experssions, you have to ask "whose?" or "as implemented in which product?" (10/9)
Hotshot sytle or legibility? - it's a debate. I think I would argue that the latter has more value in shell scripting. Here are two functionally equivalent code snippets. The first one is:
grep -q "$TARGET" "$FILE" && echo "$TARGET" is in "$FILE"
Can you read it? Here is the second one:
if grep -q "$TARGET" "$FILE"
Easier or harder to read? (10/9)
Primitive shell capability comparison results
Steven Bourne - originator of the sh, gives a talk about the shell in 2015. More historical than technical. Listen if interested. (10/1)
How to access your individual account on the class Remote Unix system.
Handout - explaining use of the computers in our classroom.
Patches to bash source code - applying these will be a step in the bash compilation exercise we will complete next week. There were 11, but now there are 17 patches. Examining release dates, numbers 12-17 are all from September 24, 2014, or later. That's the day information about the shellshock bash bug went public. These latter patches no doubt address shellshock. Bash users on older distribution versions may not have the option of an online update using yum or apt-get, as the maintainers of the repositories have end-of-life'd those versions. For those folks, applying these patches and compiling bash afresh will be their only fix. It sounds complicated but you will do it next week. The procedure is also well documented in a blog post by Steve Jenkins (see the link, in section 1 of the course outline, entitled "bash from source to address the shellshock bash bug"). (9/25)
Practice files from Quigley chapter 3. (9/25)
Someday you too can be heroic.
What is a shell? - "A Unix shell is both a command interpreter and a programming language. As a command interpreter, the shell provides the user interface to the rich set of GNU utilities. The programming language features allow these utilities to be combined."
Detailed evolution of bash and bash features version-by-version, among other information in this FAQ. (9/25)
bash maintainer Chet Ramey interviewed.
``The lyf so short, the craft so long to lerne.'' - Chaucer (from Ramey's email signature)
Cheat sheet/reference guide - for the bash shell, compiled by a student from the bash man page (help file). (9/25)
Exercise - about diff and patch. (9/25)
GNU bash homepage (9/25)
Request - please don't change the passwords on the "root" or "student" accounts of the classroom workstations.
Using ssh (secure shell). ssh is an important tool you will use for interacting with remote computers. For that you will need an ssh client. There are a number of ssh client alternatives.
Running linux at home.
Bibliography - of shell scripting information
Classic Shell Scripting Arnold Robbins and Nelson A. F. Beebe, O'Reilly & Associates, 2005
UNIX Shells by Example Ellie Quigley; Prentice Hall, 4th edition, 2005 (our textbook)
Learning the bash Shell Cameron Newham and Bill Rosenblatt, O'Reily & Associates, 3rd edition, 2005
Introducing Regular Expressions Michael Fitzgerald, O'Reilly & Associates, 2012
Mastering Regular Expressions Jeffrey E. F. Friedl, O'Reilly & Associates, 3rd edition, 2006
Sed and Awk Dale Dougherty, O'Reilly & Associates, 1997
The AWK Programming Language, Alfred Aho, Brian Kernighan, Peter Weinberger, Addison-Wesly Publishing Company, 1988
Milestones in the history of computation
Colossus - 1944