Bash - Core Skills

• This page is meant to be a quick and practical guide to Bash scriping and the CLI.
• This page is also serves as my personal notes and reference.
• This page is meant to be concise not overly verbose.

Environment and Profiles

Sourcing Scripts

Sourcing a script will execute the script but it will not be executed in a separate shell. It is executed in the same shell. This allows it to be used to setup the environment ( environment variables, shell settings, etc. ). Shared settings are often sourced by scripts. Profiles are also typically sourced when a shell is launched.

Two different ways to source an example script:

source setup.sh
. setup.sh

Profiles

Your milage may vary. Your system may be different.

Interactive login shell:

• This is read first: /etc/profile
• Next, the first of these files that is readable is read: ~/.bash_profile, ~/.bash_login or ~/.profile
• This is read upon logout: ~/.bash_logout

Interactive non-login shell:

• This is read: ~/.bashrc

When invoked with the sh command:

• These two are read: /etc/profile, ~/.profile

Non-interactive / shell script:

• Supposed to source stuff defined here: $BASH_ENV

• Mine scripts just source ~/.bashrc

• NOTE - ~/.bash_profile usually calls ~/.bashrc

These may also be used:

• /etc/inputrc
• /etc/profile.d
• /etc/bashrc

Extra Info

Bourne Shell built-ins:

:, ., break, cd, continue, eval, exec, exit, export, getopts, hash, pwd, readonly, return, set, shift, test, [, times, trap, umask and unset.

Bash built-in commands:

alias, bind, builtin, command, declare, echo, enable, help, let, local, logout, printf, read, shopt, type, typeset, ulimit and unalias.

Executing Scripts

bash -c "echo test"    # run command in string directly
bash test1.sh          # run script
bash -x script1.sh     # debug
./test1.sh             # run script

The very first line in every script is called the shebang and it specifies which interpreter to use. It looks like this. This bassically lets you know that this is a bash script and that it is run with bash.

#!/bin/bash

You can add a dir to your PATH variable like this. Scripts in one of the directories on your path can be run without specifying the full path, just the name of the script. You can run this line by itself for a single shell session or add it to your bashrc file if you want it to be persistent.

~/.bashrc


export PATH="$PATH:~/scripts"

Shell Options

Setting or unsetting shell options

set -x			# activate debugging from here
w               # run a command
set +x			# stop debugging from here

A few useful shell options:

set -f	set -o noglob	Disable file name generation using metacharacters (globbing).
set -v	set -o verbose	Prints shell input lines as they are read.
set -x	set -o xtrace	Print command traces before executing command.

Enable shell options at the beginning of a script:

#!/bin/bash -xv

Comments

Multi-line bash comment:

: '
This is a
very neat comment
in bash
'


: <<'END_COMMENT'
comment1
comment2
comment3
END_COMMENT

Variables

• variables are upper case by convention but sometimes local vars are lower case ( not always followed and I’m not sure if this is a real convention )

Scope:

• global variables
• local variables

Types:

• String variables
• Integer variables
• Constant variables
• Array variables

View environment vars or normal vars:

env or printenv    # show environment variables
set                # show all variables and functions ( local and global )

Basic variable usage:

x="abc"
echo $x
unset x     # remove variable

export x="abc"   # export making it an environment variable so child processes can use it

Notable special variables:

$-	# contains an i when in interactive mode
$PS1	# defines prompt, only set for interactive shells
$PS2	# secondary prompt
$IFS	# A list of characters that separate fields; used when the shell splits words as part of expansion.
$PPID	The process ID of the shell’s parent process
$PWD	The current working directory as set by the cd built-in command.
$UID	The numeric, real user ID of the current user.
$EUID	The numeric effective user ID of the current user.
$HOME	The current user’s home directory;
$PATH	A colon-separated list of directories in which the shell looks for commands.
$*	Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, it expands to a single word with the value of each parameter separated by the first character of the IFS special variable.
$@	Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word.
$#	Expands to the number of positional parameters in decimal.
$?	Expands to the exit status of the most recently executed foreground pipeline.
$-	A hyphen expands to the current option flags as specified upon invocation, by the set built-in command, or those set by the shell itself (such as the -i).
$$	Expands to the process ID of the shell.
$!	Expands to the process ID of the most recently executed background (asynchronous) command.
$0	Expands to the name of the shell or shell script.
$_	The underscore variable is set at shell startup and contains the absolute file name of the shell or script being executed as passed in the argument list. Subsequently, it expands to the last argument to the previous command, after expansion. It is also set to the full pathname of each command executed and placed in the environment exported to that command. When checking mail, this parameter holds the name of the mail file.

• NOTE - $* is bad, don’t use it. Use $@ instead.

Positional Parameters

These refer to the arguments passed to a script:

$0	name of the script
$1	first parameter
$2	second paramter
$3	Third parameter
etc.	and so on …

Call a script like this:

./test1.sh a b c

test1.sh


#!/bin/bash

echo $0
echo $1
echo $2
echo $3

Output:

test1.sh
a
b
c

Quotes

• Single quotes - perserve literal value of each char, including $
• Double quotes - perserve literal value of each char, except $, `, \

echo \$date        # use escape char
echo "\$date"      # use escape char
echo '$date'       # print literal value, not variable

Expansions

Brace expansion

echo start{a,b,c}end

Output:

startaend
startbend
startcend

Nested brace expansion

echo start{a,b{1,2,3},c}end   #

Tilde expansion:

~/testdir	$HOME/testdir
~+/testdir	$PWD/testdir
~-/testdir	$OLDPWD/testdir
~user2/testdir	/home/user2/testdir

Parameter expansion:

${VAR1}	avoid confusion when next char isn’t part of var
${!VAR1*}	indirect expansion?????

Arithmetic expansion:

$(( 5 + 5 ))
$[ 5 + 5 ]

0x123 # hex 0123 # octal, leading zero

Process substitution:

• uses a FIFO, NEED MORE INFO !!!!!!!!!!

<(LIST)

(LIST)

Command Substitution, Subshell:

• can be nested

x=$(ls)      # newer, easier to nest, POSIX
x=`ls`       # older, can have some escape chars?

Word splitting

• when not using double quotes

• IFS -value is “‘<space><tab><newline>’” when unset, will split on any of these, when null no splitting occurs

echo $( ps )        # split on all delimiters defined by $IFS
echo "$( ps )"      # don't split

IFS=$'\n'      # set IFS to just new lines
echo $( ps )   # now it

File name expansion ( globbing ):

• multiple unknown chars
  ?	single unknown char
  [1-5]	range of matchable chars
  [a-ew-z]	range of chars

ls -l *
ls -l test?.txt
ls -l test[1-5].txt

Character classes:

ls -ld [[:digit:]]*
ls -ld [[:upper:]]*

Character classes:

"alnum", "alpha", "ascii", "blank", "cntrl", "digit", "graph", "lower", "print", "punct", "space", "upper", "word" or "xdigit"

Aliases

alias ll='ls -l'    # create alias
unalias ll          # remove alias
unalias -a          # remove all aliases

Shell Options:

set -o                 # show all shell options
set -o noclobber       # turn on existing files won't be overwritten by redirects
set +o noclobber       # turn off
set -u                 # unset vars cause error and exit non-interactive shell
set -o noglob           # turn off globbing

Grep / Sed / Awk

grep

grep abc test.txt
grep -i abc test.txt
grep -v abc test.txt
ps -ef | grep -i nginx
grep -E "abc|xyz" test.txt

sed

sed 's/abc/xyz/g' test.txt       # global swap
sed 's/abc/xyz/gI' test.txt      # case insensitive
sed -i 's/abc/xyz/g' test.txt    # edit file in place
sed -E 's/a|b/x/g' test.txt      # extended regex
ps -ef | sed 's/root/abcd/g'     # piping

awk

awk '{print $3, $5, $7}'  test1.txt                  # print columns
awk '{print "Fields: "$3" == "$5" "$7}'  test1.txt   # custom string

awk -F/ '{print $3, $5, $7}'  test1.txt       # field separator
awk -F: '{print $3, $5, $7}'  test2.txt       # field separator
awk -F, '{print $3, $5, $7}'  test3.txt       # field separator

ps -ef | awk '{print output $3, $5, $7}'      # piping
ps -ef | awk ' /tty/ {print $3, $5}'          # split and match

Conditionals - if/else

Primary expressions

Primary	Meaning
[ -a FILE ]	True if FILE exists.
[ -b FILE ]	True if FILE exists and is a block-special file.
[ -c FILE ]	True if FILE exists and is a character-special file.
[ -d FILE ]	True if FILE exists and is a directory.
[ -e FILE ]	True if FILE exists.
[ -f FILE ]	True if FILE exists and is a regular file.
[ -g FILE ]	True if FILE exists and its SGID bit is set.
[ -h FILE ]	True if FILE exists and is a symbolic link.
[ -k FILE ]	True if FILE exists and its sticky bit is set.
[ -p FILE ]	True if FILE exists and is a named pipe (FIFO).
[ -r FILE ]	True if FILE exists and is readable.
[ -s FILE ]	True if FILE exists and has a size greater than zero.
[ -t FD ]	True if file descriptor FD is open and refers to a terminal.
[ -u FILE ]	True if FILE exists and its SUID (set user ID) bit is set.
[ -w FILE ]	True if FILE exists and is writable.
[ -x FILE ]	True if FILE exists and is executable.
[ -O FILE ]	True if FILE exists and is owned by the effective user ID.
[ -G FILE ]	True if FILE exists and is owned by the effective group ID.
[ -L FILE ]	True if FILE exists and is a symbolic link.
[ -N FILE ]	True if FILE exists and has been modified since it was last read.
[ -S FILE ]	True if FILE exists and is a socket.
[ FILE1 -nt FILE2 ]	True if FILE1 has been changed more recently than FILE2, or if FILE1 exists and FILE2 does not.
[ FILE1 -ot FILE2 ]	True if FILE1 is older than FILE2, or is FILE2 exists and FILE1 does not.
[ FILE1 -ef FILE2 ]	True if FILE1 and FILE2 refer to the same device and inode numbers.
[ -o OPTIONNAME ]	True if shell option “OPTIONNAME” is enabled.
[ -z STRING ]	True of the length if “STRING” is zero.
[ -n STRING ] or [ STRING ]	True if the length of “STRING” is non-zero.
[ STRING1 == STRING2 ]	True if the strings are equal. “=” may be used instead of “==” for strict POSIX compliance.
[ STRING1 != STRING2 ]	True if the strings are not equal.
[ STRING1 < STRING2 ]	True if “STRING1” sorts before “STRING2” lexicographically in the current locale.
[ STRING1 > STRING2 ]	True if “STRING1” sorts after “STRING2” lexicographically in the current locale.
[ ARG1 OP ARG2 ]	“OP” is one of -eq, -ne, -lt, -le, -gt or -ge. These arithmetic binary operators return true if “ARG1” is equal to, not equal to, less than, less than or equal to, greater than, or greater than or equal to “ARG2”, respectively. “ARG1” and “ARG2” are integers.

Combining expressions

Operation

Effect

[ ! EXPR ]	True if EXPR is false.
[ ( EXPR ) ]	Returns the value of EXPR. Can use to override operator precedence.
[ EXPR1 -a EXPR2 ]	True if both EXPR1 and EXPR2 are true.
[ EXPR1 -o EXPR2 ]	True if either EXPR1 or EXPR2 is true.

if [ -f /var/log/messages ]
  then                                # no semi colon if then is on next line
    echo "existsand regular"
fi

if [ -f /var/log/messages ]; then echo "exists and regular"; fi
if [ $a == "test" ]; then echo "strings equal"; fi
if [ $a != "test" ]; then echo "strings  not equal"; fi
if [ -z STR1 ]; then echo "zero len string"; fi
if [ $x -eq 5 ]; then echo "equal"; fi    # -eq, -ne, -lt, le, -gt, ge


if [ $? -eq 0 ]; then echo "exit status OK"; fi


if ! grep $extra_user /etc/passwd; then echo "not found"; fi

if [ "$(whoami)" != 'root' ]; then echo "not root"; fi

[ "$(whoami)" != 'root' ] && ( echo "not root"; exit 1)
[ "$(whoami)" == 'root' ] || ( echo "not root"; exit 1)
[ "$(whoami)" == 'root' ] || { echo "not root"; exit 1}  # don't invoke subshell

test "$(whoami)" != 'root' && (echo "not root"; exit 1)


if [[ "$x" == t* ]]; then echo "match"; fi       # use regex

single square bracket vs double square bracket?????????????????????????????

if [ ! $# == 2 ]; then            # check number of args
  echo "Usage: $0 abc xyz"
  exit
fi



if [ $x -eq "abc" ] ; then
  echo "matched"
else
  echo "nothing"
fi


if [ -f "$FILENAME" ]; then        # double quotes so filename with space won't be split
....
if [[ -f "$FILENAME" ]]; then      # or use [[
....

if [ $x -eq "abc" ]; then
  echo "match 1"
elif [ $x -eq "xyz ]; then
    echo "match 2"

        if [ z -ne 0 ]; then
          echo "non zero z"
        else
          echo "z was zero"
        fi
else
  echo "no match"
fi

if [ $[$year % 400] -eq "0" ]; then        !!!!! why a $[  ???????

if (( ($a % 4) == “0” ))

(( ($a % 4) == “0” )) # boolean logic &&

???? parens with if statement, differnt test operators?????????????????????????????????????????????????????????????

============================

((a++))
((x = 42))
for ((i=0; i<10; i++))
echo $((a + b + (14 * c)))

let x=5+5
let x=5


$(     # return value instead of command
$((    # needed to assign to var or echo, not needed for test
$[     # needed to assign, like arithmetic expansion but obsolete
$[[    # DOESN'T work

let     basically same as ((, but no spaces

(       subshell or array definition
((      arithmetic ops, can add spaces around ops, ommit $ on int and array vars
        bare parens return 0 (true) value is non zero, 1 (false) if zero

test    shell built in   test condition
[       shell built in   test condition allows -a, -o, spacing matters!!
[[      shell keyword    test condition allows &&, ||, =~, spacing matters!!
{       unambiguously identify vars, parameter expansion, code block


echo f{oo,ee,a}d                 # brace expansion
echo {000..2}                    # new in Bash 4
for num in {000..2}; do echo "$num"; done
echo {00..8..2}
echo {D..T..4}

 { date; top -b -n1 | head ; } >logfile  # code block, requires spaces around { and semicolon after last command

• NOTE - need $(( to return value but not to assign within the expression ex: ((x = 1 + 2)); echo $x

(ls -l)         # sub shell
x=$(ls -l)
x=`ls -l`         # sub shell, older

Command directly in if statement, no test needed:

if ls; then echo asdf; else echo xxxxx; fi
if ls asdfasdfasdfasdf; then echo asdf; else echo xxxxx; fi

if ((5 == 5)); ...            # arrithmetic
if [ 5 -eq 5 ]; ...           # arrithmetic
if [ "5" -eq "5" ]; ...       # arrithmetic
if [ "abc" == "xyz" ]; ...    # string compare

= vs == for ((, [, [[

if (( x = "5" )); ...         # test assignment
if (( "5" == "5" )); ...      # test arith equality
if [ "abc" == "abc" ]; ...    # test string compare
if [ "abc" = "abc" ]; ...     # test string compare
if [[ "abc" == "abc" ]]; ...  # test string compare
if [[ "abc" = "abc" ]]; ...   # test string compare

if [ 5 -eq 5 -o 1 -eq 3 ]; ...     # logical or
if [ 5 -eq 5 -a 1 -eq 1 ]; ...     # logical and

if [[ 5 -eq 5 || 1 -eq 3 ]]  ...   # logical or
if [[ 5 -eq 5 && 1 -eq 1 ]]; ...   # logical and


if [[ (5 -eq 5 ) || (1 -eq 1 && 2 -eq 2) ]];        # group for logic
if [ \( 5 -eq 5 \) -o \( 1 -eq 1 -a 2 -eq 2 \) ];   # same but need to escape ( and picky about spaces


if [ $x == abc ];    # works
if [ $x == "abc" ];  # works
if [ $x == ab* ];    # fail
if [ $x == "ab*" ];  # works but NOT as wildcard

if [[ $x == abc ]];    # works
if [[ $x == "abc" ]];  # works
if [[ $x == ab* ]];    # glob
if [[ $x =~ ab* ]];    # regex ( * still matches multiple chars ????)
if [[ $x == a.* ]];    # glob
if [[ $x =~ a.* ]];    # regex

if [ -e $filename ];       # fail on file name with spaces because of word splitting
if [ -e "$filename" ];     # justt put it in double quotes to fix
if [[ -e $filename ]];     # works on file name with spaces because NO word splitting

-eq, =, ==	work in [ and [[
-a, -o	only works in [
		, &&	only works in [[
<	works in [[
<	need to escape for [

============================

exit 0   # exit normal / OK
exit 1   # exit with error code
exit 2   # exit with error code
exit 3   # exit with error code

Case Statements

case $x in
    [1-6]*)                 # using a range and wildcard
        echo "lower"
        ;;
    12|15|23)               # or
        echo "more"
        ;;
    100|200)                # or
        echo "higher"
        ;;
    *)
        echo "anything else"
        ;;
esac

Common case statement example:

case "$1" in
        start)
            start
            ;;

        stop)
            stop
            ;;

        status)
            status
            ;;

        restart)
            stop
            start
            ;;

        *)
            echo $"Usage: $0 {start|stop|restart|condrestart|status}"
            exit 1

esac

Input / Output

echo "test"      # echo out some text
echo -e "test"   # interpret backslash escape chars
echo -n "test"   # suppress trailing newline


printf "First: %s\nSecond: %s\n" "12" "34"


read         # read inpupt, save to var REPLY by default
read x       # promt for input, save in variable
read a b c   # read multiple words and save each word in corresponding var,
             # remaining vars all assigned to last variable
             # word split by $IFS

input "this is a test"

echo $a   # prints "this"
echo $b   # prints "is"
echo $c   # prints "a test"

Redirection

Standard file handles / streams:

| 0 | STDIN  | input |
| 1 | STDOUT | output |
| 2 | STDERR | error output |

Redirection operators:

>	overwrite
>>	append
2>&1	redirect STDERR to STDOUT

ls > test1.txt            # redirect command output ( STDOUT ) to a file and **OVERWRITE** file
ls >> test1               # redirect command output ( STDOUT ) to a file and **APPEND** to file

ls asdf 2> test1.txt      # redirect command output ( STDERR ) to a file and **OVERWRITE** file
ls asdf 2>> test1.txt     # redirect command output ( STDERR ) to a file and **APPEND** to file

ls asdf 2> test1.txt      # redirect command output ( STDOUT and STDERR ) to a file ( since Bash 4 )

ls asdf >> output.log 2>&1   # append STDOUT to file, also redirect STDERR to STDOUT ( both go to file )

Background

You can run a command in the background by appending an ampersand to it:

./server1 &

./process_data.sh &

Pipes

You can pipe output from one command to another like this:

cat data1.csv | sort

ps -ef | grep -i nginx

Here documents

cat << MYLIST
tree
rock
grass
MYLIST

cat << MYLIST > /var/log/output
car
water
sand
MYLIST

Pass input to something:

yum install $1 << CONFIRM
y
CONFIRM

Loops

• NOTE - Loops can be nested.

For Loops

x="a b c"

for i in `cat list`; do echo $i ; done
for i in `ls`; do echo $i ; done
for i in a b c; do echo $i; done
for i in {1..10}; do echo $i; done
for i in `seq 1 10`; do echo $i; done
for i in $x; do echo $i; done            # loop over string ( word splitting )
for i in /opt/*; do echo $i; done        # loop over globbed file list

for (( i=0; i<10; i++)); do echo $i; done


IFS=$'\n'                             # split on new line, not space or tab
for i in `ps -ef`; do echo $i; done

While Loops

i=0; while [ $i -lt 4 ]; do echo $i; i=$[$i+1];done

i=0
while [ $i -lt 4 ]; do
    echo $i
    i=$[$i+1]
done


while true; do      # infinite loop
    echo "."
    sleep 5
done

Until Loops

i=0
until [ $i -gt 4 ]; do
    echo $i
    i=$[$i+1]
done

Piping and Redirection into Loops

ls | while read i; do
    echo "file: " $i
done


while read i; do
    echo "file: " $i
done < test1.txt


c=0
while true; do
    ((c++))
    read x
    [[ $x == "done" ]] && break         # break out of loop
    [[ $x == "skip" ]] && continue      # cut this iteration short and start next iteration
    echo $c
done

Select Loops

Keeps reading lines from input. If line is blank will re-print menu.

select x in frog rock tree;
do
     echo "You selected: $x item number: $REPLY"
done

Using a glob:

select x in *;
do
     echo "You selected: $x item number: $REPLY"
done

Using a command:

select x in $(ls);
do
     echo "You selected: $x item number: $REPLY"
done

Shift

Useful for unknown number of args if you want to process them one at a time.

shift 3   # shift positional args left by 3
          # drop 1,2,3, rename what is left

$cat test.sh
echo $1 $2 $3 $4 $5 $6 $7 $8 $9
shift 3
echo  $1 $2 $3 $4 $5 $6 $7 $8 $9
$

$bash test.sh a b c d e f g h i
a b c d e f g h i
d e f g h i
$

Variables - More

-a	Variable is an array.
-f	Use function names only.
-i	The variable is to be treated as an integer; arithmetic evaluation is performed when the variable is assigned a value (see Section 3.4.6).
-p	Display the attributes and values of each variable. When -p is used, additional options are ignored.
-r	Make variables read-only. These variables cannot then be assigned values by subsequent assignment statements, nor can they be unset.
-t	Give each variable the trace attribute.

declare -p x      # show type
declare -i x=12   # declare as integer ( restrict to only integer )

readonly x=abc    # create constant ( read only )
readonly -p       # show all constants

Arrays

declare -a ARRAYNAME

a[3]=5   # assign element, will declare even if not first element

a=(1,2,3)   # declare, initialize
a=(a,b,c)   # declare, initialize


echo ${a[3]}       # specific element
echo ${a[*]}       # entire array
echo ${a[@]}       # entire array

unset a[3]         # remove element
unset a            # remove array

Length of variable in chars or elements in array:

${#VAR}

echo ${TEST:-ALT}  # default value - if TEST is null or undefined, use alt instead
echo ${TEST:=abc}  # default value - if TEST is null or undefined, use alt instead, also assign value
echo ${TEST:?"asdf"}   # if not set, print string and exit script ( non-interactive shell )


echo ${STRING:4}    # everything after char 4

echo ${STRING:6:5}  # 5 chars after char 6

Starting at beginning:

echo ${x#a}     # match and remove this char from var
echo ${x#a*}    # match and remove shortest possible match from this char
echo ${x##a*}   # match and remove longest possible match from this char

Starting at end:

echo ${x%a}     # match and remove this char from var
echo ${x%a*}    # match and remove shortest possible match from this char
echo ${x%%a*}   # match and remove longest possible match from this char

echo ${x/abc/xyz}   # replaced abc with xyz in var x

Functions

function test1 { echo "test"; }
test1 () { echo "test"; }

• “The curly braces must be separated from the body by spaces.”
• “The body of a function should end in a semicolon or a newline.”

test1 () {                         # totally valid, just need that space before {
    echo "Values: " $1 $2 $3 $#
    return 0
}

test1 ()
{
    local x='abc'                      # local variable
    y='xyz'                            # not local, survives outside the funciton even if first used here
    echo "Values: " $1 $2 $3 $#
    return 123                         # between 0-255
}


test1 a b c
echo $?

Signals

Signal name |Signal value |Effect SIGHUP |1 |Hangup SIGINT |2 |Interrupt from keyboard ctrl-c SIGKILL 9 Kill signal SIGTERM |15 |Termination signal SIGSTOP |17,19,23 |Stop the process, 19 is ctrl-z

• SIGKILL and SIGSTOP can not be caught, blocked or ignored.

kill    5607    #  SIGTERM 15
kill -9 5607    #  SIGKILL 9

trap "echo will not stop" SIGINT SIGTERM           # when these signals are trapped
trap "load_configs" SIGHUP                         # load config files on SIGHUP
trap "{ rm -f $LOCKFILE ; exit 255; }" EXIT        # executed when shell exits EXIT or 0

Detect when a variable is used:

declare -t VARIABLE=value
trap "echo VARIABLE is being used here." DEBUG

xargs

!!!!!!!!!! xargs !!!!!!!!!!!!!

More

Directory stack ????

DIRSTACK pushd popd dirs

References

• TLDP - Bash Beginners Guide
• TLDP - Advanced Bash Scripting