David Morgan
U. C. L. A. Extension - X417.31
see syllabus for email address
Informational links
Remote Unix
access with ssh
BASH programming Introduction HOW-TO
FALL
2018
Tuesdays 6:30pm-10:00pm
UCLA Extension Lindbrook Center
10920 Lindbrook Drive
This Website (http://uclaextension.dmorgan.us/morgan_david/unex.html) will be used extensively to communicate with you. Announcements, grade reports, and assignments will be posted here. The site can be viewed from an internet-connected browser anywhere. You are responsible for awareness of the information posted here.
Course outline - with approximate weekly topic coverage. All corresponing readings, , homework assignments, and in-class slides I will use are here. You can preview or review the slide lessons at any time or view them while I am presenting them in class.
Syllabus - containing my email
address, specification of required textbook, recommendation of other
useful books, and some policy information.
-------------------------------------------------------------------------------------------
Thank you - for taking this course and giving me Tuesday evenings to look forward to. (11/27)
Final exam - no test (11/27)
Related linux/unix classes - that may interest you.
Introduction to
Linux/Unix -
System
Administration - continuation of my introductory course with system administration focus
Advanced Linux/Unix: Networking - networking, emphasizing the linux
platform commands to "do" it
Advanced Linux/Unix: Security - theory and practice
Schedule: the courses tend to be offered every other quarter. There may be variations but this is the general expectation. If you are interested please "stay tuned" to the class schedule as UCLA Extension publishes it. Or if you email me I can let you know what we are putting in the pipeline. None of these is offered for Winter 2019 however.
Content:
Introduction to Linux/Unix see website from most
recent quarter and course outline
System administration - see website from most
recent quarter and course outline
Networking - see website from
most recent quarter and course
outline
Security - website from
most recent quarter. and course
outline This course was derived from extensive materials I have compiled from teaching security elsewhere,
is lab-heavy. (11/27)
Homework - textbook awk labs 3 and
4 (on pages 187-88 and 202, respectively).
The needed datafiles lab3.data and lab4.data are in /home/public on our
class server, or at the "Practice files" link below. For each
assignment please produce bash (not awk) script that I can run. Don't do
lab 3's question 10. In your bash scripts, write an awk 1-liner for each
question, that does what that questions specifies. Between 1-liners,
interperse bash "read" commands to halt your script after the
output of each 1-liner. Name the scripts lab3.sh and lab4.sh and place
them in your assignments directory. -
due on server end-of-day Tuesday December 4 (11/27)
Three good awk resources - in class we talk about awk. We spend time on course outline section 9's exercises at the links entitled "extracting classic AWK book's sample scripts" and "sort multi-line blocks with AWK." These are based on the classic book The AWK Programming Language, which is rich in small examples. Another good source of instructive small examples is Eric Pement's awk one-liners document. Also in section 9 is a related link titled "AWK one-liners." Our textbook is a third great source on awk, note the link below to the book's practice files that enable you to perform the many examples in the awk chapter (chap. 6) as you read through them. (11/20)
Grades - published at link entitled "Grade reports" at left. (11/20)
Upcoming topics - mostly awk
-- programming features of bash
(textbook ch 14)
Many of the bash features are (assumed to be) review for you. We will
discuss them informally in class.
-- awk (textbook ch 6)
The Quigley textbook has good coverage of awk in chapter 6. I like it
because it is more comprehensive than most and full of examples we can
try. (They do no good unless we actually take time to try them however!)
Please read these 4 labs distributed throughout chapter 6:
lab 3, on p. 187-88
lab4, on p. 202
lab 5, on pp. 226-27
lab 6, on pp. 255-56
Labs 3 and 4 challenge you to produce 1-liners that do specified things with given data. Labs 5 and 6 require you to write short but full programs that produce formatted reports from given data. Try them all. Doing so requires reading the material preceding each lab, and takes some time and effort. We will work on and discuss them in the next 2 class meetings. I will decide which to ask you to formally turn in as homework. (11/20)
Practice files from Quigley chapter 6 (awk). (11/20)
awk one-liners - from the same gentleman, Eric Pement, who brought us the sed one-liners. (11/20)
an interesting awk exercise in class - next class. It's the one at the course outline link "example scripts from classic AWK book." It gives you a file named awkcode.txt that contains about 200 sample awk scripts. The interesting thing is the way it delivers all these little sample scripts. Normally this sort of thing is done with zip or tar files. Here, it's all done using awk itself. Take a look at the "awkcode.txt" file, its structure, and its first 4 lines. I want you to "extract" the 200 files from it in class. I will show you the model for doing so in lecture, and the elements of the awk language involved. (11/20)
sed one-liners - and the one-liners' deeper explanation. (11/20)
Homework -
do - the modification to the Advanced Bash Scripting Guide's word
frequency script, course
outline section 5 - - due in your "assignments"
subdirectory by end-of-day Wednesday November 14
read - per course outline, section 5, Quigley chapter 5 (sed).
read - per course outline, section 6, Quigley chapter 14 (programming
bash)
anticipate/upcoming - assignment as described in course outline
section 6, applying sed to Quigley's "datebook" file. -
due on server by end-of-day Wednesday November 21
(11/13)
Quoting text - a couple reasons to do it
1a) make it a single token
[root@unexgate sed]# cat show-args
echo hello
echo "1st arg is: $1"
echo "2nd arg is: $2"
echo goodbye
[root@unexgate sed]#
[root@unexgate sed]# ./show-args aaa bbb ccc
hello
1st arg is: aaa
2nd arg is: bbb
goodbye
[root@unexgate sed]# ./show-args "aaa bbb" ccc
hello
1st arg is: aaa bbb
2nd arg is: ccc
goodbye
[root@unexgate sed]#
1b) make it a single token
[root@unexgate ~]# cat holidays
Thanksgiving day is for turkeys
July fourth is for fireworks
Christmas eve is for gifts
[root@unexgate ~]# cat holidays | grep July fourth
grep: fourth: No such file or directory
[root@unexgate ~]#
[root@unexgate ~]# cat holidays | grep "July fourth"
July fourth is for fireworks
[root@unexgate ~]#
2) make sure it'll be a string (even a null one)
[root@unexgate sed]# VAR=something
[root@unexgate sed]#
[root@unexgate sed]# test "$VAR" = "something"; echo
$?
0
[root@unexgate sed]# test $VAR = "something"; echo $?
0
[root@unexgate sed]#
[root@unexgate sed]# VAR=
[root@unexgate sed]# test "$VAR" = "something"; echo
$?
1
[root@unexgate sed]# test $VAR = "something"; echo $?
-bash: test: =: unary operator expected
2
[root@unexgate sed]#
Many small specialists - according to the 2nd slide we viewed in this course programs in Unix are supposed to "do one thing well." In that vein I noted this:
"One of the great contributions of Unix to the computing world is that process creation is cheap and easy. This encourages the practice of writing small programs that each do a part of a larger job, and then combining them to collaborate on the completion of that task. Because programming complexity grows much faster than linearly with program size, small programs are much easier to write, debug, and understand than large ones." Shell Scripting, Robbins & Beebe, p353
It can be argued, in relative terms, whether process creation is cheap. Famously for example, threads are useful because they are so much cheaper than processes. But compared to historical and competing operating systems Unix's fork/exec is cheap and, to the point, "cheap enough." That removes the barrier to hitching (perhaps pipelining) a team of small programs together in place of a monolithic one. (11/6)
Homework -
do - the arrays for bubble sort programming exercise, course
outline section 5 - - due in your "assignments"
subdirectory by end-of-day Wednesday November 7
(10/30)
Grades - published at link entitled "Grade reports" at left. (10/30)
Homework -
finish - the bash version of the "party program" if you
have not yet done so. See homework posting below dated 10/15. - due in your "assignments"
subdirectory by end-of-day Sunday October 28
do - the safe-remote-poweroff programming exercise, course outline
section 5; we will discuss in class next week, there are different ways to
do it, you can tell us about challenges solved and unsolved - due in your "assignments"
subdirectory by end-of-day Tuesday October 30 (10/23)
Grades - published at link entitled "Grade reports" at left. (10/23)
Command line fu - have fu. (just for fun) (10/23)
$-
a special shell variable. Its
contents are several lower and/or uppercase letters. These are the option
letters of the set command that are currently "on", those bash
options that are in effect. For documentation of these cryptic letters,
try "help set | less" and additionally find the section in
the main bash man page headed "OPTIONS". (10/23)
Regular expression humor - you think you got problems? (10/19)
Homework -
please see the Homework posting below dated 10/9
note that a due date has been formally assigned there for the grep
exercise
do - the "party program" exercise at the link in course
outline section 2 entitled "comparing shells". You can do it on
unexgate. The file you're supposed to obtain is local there. It's
/home/public/the-party-program-csh-ksh.zip. It's readable so you could cp
it to your own directory. Leave the bash version of the program (the
objective of the exercise) in your "assignments" directory.
read forward - take a look at the material indicated in course outline
section 4 (10/15)
A comment about interpreter scripts' interpreters and how they handle their file argument
What happens in an interpreter script to the lines after the initial shebang (first), if the shebang is " #!/bin/mv " as opposed to " #!/bin/bash " ? Suppose the script file's name is myscript, and it is called as:
myscript argument
Then an exec( ) function call will be constructed that is the same as would be constructed if you gave the following commands:
/bin/mv myscript argument
(as opposed to, /bin/bash myscript argument )
Then, what happens to the lines in the myscript file depends on how the commands (mv and bash) treat their first argument. mv takes it as the name of a file to rename. In doing so it does not delve into the file's content at all. bash takes it as the name of a file over whose contents, viewed as lines, it must iterate in a loop. When called, bash is content-conscious and mv is content-oblivious when a file's name shows up as their first argument. That's why mv is not hindered by a file with multiple lines and just renames it same as if the file held just the single shebang line only. (10/15)
a bash "exec" redirection use case
-
prepare a file with 10 lines in it for reading:
seq 1 10 | xargs -I + echo "line +" > file
read them out:
while read line;do echo "$line";done < file
read them out, stopping after each line for user go-ahead:
while read line;do echo "$line"; read -p "Press any
key" -n 1;done < file
Does not work. Instead:
exec 3<file
while read -u 3 line;do echo "$line"; read -p "Press any key" -n 1;done
Arranges for the 2 consumers of input (the two "read"s) to get it from different places.
from http://wiki.bash-hackers.org/howto/redirection_tutorial
Another: you want the output of entire portions of a script (but not of the whole script) captured into a file. The script has many individual lines within it that generate output. You know how to put file redirection ( >> ) on each line, but there are too many lines. At the top of the portion(s) whose output you want diverted, divert it. At the bottom, undivert it:
echo put \"blue suede shoes\" in a file
exec 9>&1 1> somefile
echo One for the money
echo Two for the show
echo Three to get ready
echo Now go, cat go.
exec 1>&9 9>&-
echo enjoy your song it's in the file
(10/15)
Homework -
read - the material indicated in course outline sections 2 and 3
do - the I/O redirection programming exercise, course outline
section 3. Please name the script you create "redirection" and
put it in the "assignments" subdirectory of your home directory.-
due in your "assignments"
subdirectory by end-of-day Wednesday October 17 (provided we are able to
cover it in class tonight)
do - the grep exercise from the text book, course outline section 4 due in your "assignments"
subdirectory by end-of-day Wednesday October 24 (10/9)
Different "echo"s - I distinguished between the built-in, shell-resident code for "echo" and the binary-file-resident one. They are two different things, though they both do largely the same thing. This note appears in the echo man page, which pertains to the binary-file echo:
"NOTE: your shell may have its own version of echo, which usually
supersedes the version described here. Please refer to your shell's
documentation for details about the options it supports."
This echo was written by Brian Fox and Chet Ramey, the ones who wrote bash itself and hence presumably its built-in echo. No doubt therefore they are highly similar. The echo's that may be built in to the korn shell or c shell will be different, and those found as binary executables in other environments may be different too, written by somebody else yet again. echo is one of the commands that is least standardized, hence least portable. Many people suggest avoiding echo and relying on the printf command instead, whose operational behavior and syntax is more uniformly standardized. (10/9)
Warning - the "datebook" file on the textbook's CD may be corrupted. Some editions deliver a file that uses Microsoft line termination convention. That is, they separate lines with hex 0D 0A character pairs, instead of the single character 0A that unix utilities expect.
You need to fix the file, or you will get wrong results when doing the homework. There is a utility for this purpose called dos2unix. You could fix the file by:
dos2unix datebook
If you don't have dos2unix, another way to do it is with tr (the "translate" command). Something like:
cat datebook | tr -d "\r" >
datebook.temp
mv -f datebook.temp datebook
And a third way uses sed:
sed -i 's/.$//' datebook
which says, replace the character before the end of the line with nothing. The 0A being interpreted as "the end of the line," that character is its preceding 0D. Replacing it with nothing is getting rid of it. The -i option applies the changes to the original file, in-place.
Applying the correction to an uncorrupted file is harmless, it leaves it uncorrupted. So you might, conservatively, do it on your "datebook" file just in case. Alternatively, download it from the link entitled "Practice files" below. It delivers a zip file, and the files inside the zip are good. (10/9)
Clarification about file descriptor representation - A former student and I talked about file descriptors. It led me to realize that my slides and many sources represent the descriptor table in a potentially confusing way (below, uppper left). The student thought that stdin represents what descriptor 0 connects to, just as /somefile represents what descriptor 3 connects to. They are presented in exactly the same way, after all. No, descriptor 0 doesn't connect to stdin. Nothing connects to stdin. There is no stdin to connect to, since it's nothing but a nickname or synonym for 0. As well, stdout and stderr are synonyms for 1 and 2. Whatever you choose to call them, what they connect to must be some material entity, like a device or file. It makes no sense that they would connect to their name! Better, my revised table below upper right. Princeton Professor Rexford's slides (below) used a good representation in that she shows only the descriptors and their devices, omitting names altogether. The names are an unneeded distraction.
(10/9)
The definitive list of regular expression
metacharacters - is in your textbook.
it's on page 69
it's on page 83
it's on page 95
it's on page 101
the only question is, which one is definitive? The only answer is, none.
One is vanilla, the next is grep, another is egrep, and also GNU grep.
That's grep. Not vi. Not sed. Not perl.
When you talk about regular experssions, you have to ask "whose?" or "as implemented in which product?" (10/9)
Hotshot sytle or legibility? - it's a debate. I think I would argue that the latter has more value in shell scripting. Here are two functionally equivalent code snippets. The first one is:
grep -q "$TARGET" "$FILE" && echo "$TARGET" is in "$FILE"
Can you read it? Here is the second one:
if grep -q "$TARGET" "$FILE"
then
echo "$TARGET" is in "$FILE"
fi
Easier or harder to read? (10/9)
Homework -
read - the material indicated in course outline section 1.
anticipate - upcoming actvities
1 - 2nd part of lab exercise at course outline section 1 link
entitled "primitive shells"
where we compile bash from source code
2 - exercise at course outline section 2 link entitled
"comparing shells" where we
write functionally equivalent scripts under 3 different shells; this
is textbook author's
"party program" from chapter 2.
(10/2)
Primitive shell capability comparison results
matrix
-
Here are the results of trying different features in each of the 4
primitive shells tested in the hands-on in-class lab exercise performed in
class.
The point is that shells are merely programs, different ones behave
somewhat differently, in accord with the features their authors have
chosen to program into them. (10/2)
Steven Bourne - originator of the sh, gives a talk about the shell in 2015. More historical than technical. Listen if interested. (10/1)
How to access your individual account on the class Remote Unix system.
Handout - explaining use of the computers in our classroom.
Structure of a unix process - an anchor reference for much of the discussion about operating system and shell. Keep it in mind throughout the course.
Patches to bash source code - applying these will be a step in the bash compilation exercise we will complete next week. There were 11, but now there are 17 patches. Examining release dates, numbers 12-17 are all from September 24, 2014, or later. That's the day information about the shellshock bash bug went public. These latter patches no doubt address shellshock. Bash users on older distribution versions may not have the option of an online update using yum or apt-get, as the maintainers of the repositories have end-of-life'd those versions. For those folks, applying these patches and compiling bash afresh will be their only fix. It sounds complicated but you will do it next week. The procedure is also well documented in a blog post by Steve Jenkins (see the link, in section 1 of the course outline, entitled "bash from source to address the shellshock bash bug"). (9/25)
Practice files from Quigley chapter 3. (9/25)
Someday you too can be heroic.
What is a shell? - "A Unix shell is both a command interpreter and a programming language. As a command interpreter, the shell provides the user interface to the rich set of GNU utilities. The programming language features allow these utilities to be combined."
--Bash
Reference Manual
Here is a tutorial website that covers the basic
functionality of a shell nicely. (9/25)
Detailed evolution of bash and bash features version-by-version, among other information in this FAQ. (9/25)
bash maintainer Chet Ramey interviewed.
Q: "What prompted the language's name: why was a pun created on the Bourne Shell?"
A: "The FSF has a penchant for puns, and this one seemed appropriate, I suppose. The name predates my involvement."
bash got started in the late '80s. Ramey became primary maintainer in
1990.
``The lyf so short, the craft so long to lerne.'' - Chaucer (from Ramey's email signature)
(9/25)
bash authors queried about shellshock (9/25)
Cheat sheet/reference guide - for the bash shell, compiled by a student from the bash man page (help file). (9/25)
Exercise - about diff and patch. (9/25)
GNU bash homepage (9/25)
Request - please don't change the passwords on the "root" or "student" accounts of the classroom workstations.
Using ssh (secure shell). ssh is an important tool you will use for interacting with remote computers. For that you will need an ssh client. There are a number of ssh client alternatives.
Running linux at home.
Bootable Fedora 23 on USB for you to burn
- I made a suitable, persistent image recently. If you bring an 8GB or
larger USB flash drive to class (empty of anything you care about) we
can burn my image to your drive for you to boot from in the future.
Bibliography - of shell scripting information
resources
These are not random. They are curated. All of them are good. I
particularly recommend and call your attention to Cooper, Robins/Beebe,
and Friedl.
Advanced Bash-Scripting Guide, An in-depth exploration of the art of shell scripting; Mendel Cooper
Classic Shell Scripting Arnold Robbins and Nelson A. F. Beebe, O'Reilly & Associates, 2005
UNIX Shells by Example Ellie Quigley; Prentice Hall, 4th edition, 2005 (our textbook)
Learning the bash Shell Cameron Newham and Bill Rosenblatt, O'Reily & Associates, 3rd edition, 2005
Introducing Regular Expressions Michael Fitzgerald, O'Reilly & Associates, 2012
Mastering Regular Expressions Jeffrey E. F. Friedl, O'Reilly & Associates, 3rd edition, 2006
Sed and Awk Dale Dougherty, O'Reilly & Associates, 1997
The AWK Programming Language, Alfred Aho, Brian Kernighan, Peter Weinberger, Addison-Wesly Publishing Company, 1988
GNU manuals:
bash: https://www.gnu.org/software/bash/manual/bashref.html
awk: https://www.gnu.org/software/gawk/manual/
bc: https://www.gnu.org/software/bc/manual/bc.html
find: https://www.gnu.org/software/findutils/manual/find.html
grep: https://www.gnu.org/software/grep/manual/
sed: https://www.gnu.org/software/sed/manual/sed.html
Milestones in the history of computation
Colossus - 1944
Eniac - 1946