Bash Scripting Tricks

Special variables

• Difference between $* and $@

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
for arg in $*; do
  echo "$arg "
done

for arg in $@; do
  echo "$arg "
done

for arg in "$*"; do
  echo "$arg "
done

for arg in "$@"; do
  echo "$arg "
done

Test with arguments "a 1" "b 2" "c 3"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
a
1
b
2
c
3
a
1
b
2
c
3
a 1 b 2 c 3
a 1
b 2
c 3

When unquoted, $* and $@ are identical, both breaking up arguments whether quoted or not. However, when quoted, $* is all arguments joined together with spaces; $@ is an array of arguments matching exactly what was given to shell.

Parameter expansion

${#var} expands to length in char of var

Conditional

Example script

1
2
3
4
5
6
7
8
9
10
#!/bin/bash
in="${1:?Please provide input file.}"
out="${2:-out.gif}"
png="$(mktemp --suffix '.png')"
[[ ! -f "$in" ]] && {
  echo "File not found."
  exit 1
}

ffmpeg -y -i ...

Substring

Pattern is same as globbing

• PATH priority

1
2
3
4
echo "Lowest priority in PATH: ${PATH##*:}"
echo "Everything except lowest priority: ${PATH%:*}"
echo "Highest priority in PATH: ${PATH%%:*}"
echo "Everything except highest priority: ${PATH#*:}"

• Batch convert img

1
for f in *.png; do ffmpeg -i "$f" "${f%.png}.jpg"; done

Substitution

Pattern matching is always greedy

Bash array

Similar to $* and $@, ${arr[*]} and ${arr[@]} behaves differently when quoted.

1
2
3
4
5
6
7
8
9
10
11
12
for el in ${arr[*]}; do
  echo $el
done
for el in ${arr[@]}; do
  echo $el
done
for el in "${arr[*]}"; do
  echo $el
done
for el in "${arr[@]}"; do
  echo $el
done

Test with arr=("a 1" "b 2" "c 3")

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
a
1
b
2
c
3
a
1
b
2
c
3
a 1 b 2 c 3
a 1
b 2
c 3

This answer gives another example where quoted array parameter expansion is used along with printf.

Redirection

File descriptors for stdin, stdout, and stderr are 0, 1, and 2, respectively.

Multiple instances of input and output redirection and/or pipes can be combined in a single command line.

1
<command1> | <command2> | <command3> < <input_file> > <output_file>

This answer explains why <command> >> <output_file> 2>&1 works as expected and <command> 2>&1 >> <output_file> does not.

• Suppress stderr

1
<command> 2> /dev/null

• Suppress both stdout and stderr

1
<command> &> /dev/null

1
<command> > /dev/null 2>&1

Here document

Here document is a special form of redirection, which uses a pair of limit string to feed a multiline string into the stdin of a command.

• Print help message

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash

print_help() {
  cat <<EOF
Usage: ...

Options: ...
  -h, --help               Show help message
EOF
  exit 1
}

print_help

• Write to file

1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash

config="$HOME/myconf"
write_config() {
  cat <<EOF >"$config"
opt1 = 1
opt2 = 2
EOF
}

write_config

• Pipe to another command

1
2
3
4
5
6
7
8
9
#!/bin/bash

pipe_command() {
  cat <<EOF | mysql
SELECT * FROM users;
EOF
}

pipe_command

Here string is another form of here document.

• Prepend a line to a file

1
cat - $file <<< $line > $newfile

Pattern matching

Bash itself does not recognize regular expressions, but it does carry out filename expansion, also known as globbing. The re-matching operator =~ accepts regular expressions, as of Bash version 3.

Regular expression

Certain commands and utilities, such as grep, sed and awk, interpret regular expressions.

Any special symbol may become literal by preceding it with a \.

• Basic re

• Extended re

Globbing

Wildcards should be escaped to match their literals

Bash performs filename expansion on unquoted command-line arguments like echo *.

1
2
3
4
str="a * b"
for w in $str; do
  echo $w
done

1
2
3
a
<every_file_in_current_folder>
b

Wildcards will not expand to a dot in globbing, it has to be explicitly included. For example, to match .bashrc and .zshrc, .*shrc instead of *shrc should be used.

Word splitting

• When does word splitting happen?

The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did not occur within double quotes for word splitting.

• Some examples

1
2
3
4
5
6
7
str="a b c"
for w in $str; do
  echo $w
done
for w in "$str"; do
  echo $w
done

1
2
3
4
a
b
c
a b c

1
2
3
str="a b c"
printf "%s\n" $str
printf "%s\n" "$str"

1
2
3
4
a
b
c
a b c

Parse command-line arguments

Here is a good practice of parsing command-line arguments.

1
2
3
4
5
6
./parse_cmdline_args.sh --help
./parse_cmdline_args.sh --output file
./parse_cmdline_args.sh --output=file
./parse_cmdline_args.sh -o file
./parse_cmdline_args.sh -ofile
./parse_cmdline_args.sh -o file --verbose foo bar

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
#!/bin/bash

print_help() {
  cat <<EOF
Usage: ...

Options: ...
  -h, --help               Show help message
EOF
  exit 1
}

args=$(getopt --options "ho:v" --longoptions "help,output:,verbose" -- "$@" 2>/dev/null)
if [[ $? -ne 0 ]]; then
  print_help
  exit 1
fi

eval set -- "$args"

OUTPUT="default_name"
VERBOSE=0

while [ : ]; do
  case "$1" in
    -h | --help)
      print_help
      exit 1
      ;;
    -o | --output)
      OUTPUT="$2"
      shift
      ;;
    -v | --verbose)
      VERBOSE=1
      ;;
    --)
      shift
      break
      ;;
    *)
      echo "Invalid option: $1" >&2
      exit 1
      ;;
  esac
  shift
done

# handle positional arguments
printf "%s\n" "$@"

Quotes, Brackets, Ampersand, Bar

Remember to always use double quotes around parameter expansions because it has fewer surprises and most usually does what we want. This answer is a simple illustration.

Remember to always use double brackets in test constructs because it is generally safer to use. This answer explains the pros.

Ampersands & on the command line

Bash command chaining

1
[[ -d "$name" ]] || mkdir -p "$name"

1
selected=$(cat "$file1" "$file2" | fzf)

1
newstr=$(echo "$str" | tr ' ' '_')

Command groups

• Parentheses

1
(<command1>; <command2>; <command3>)

This command group creates a subshell and executes in a separate environment.

• Braces

1
{ <command1>; <command2>; <command3>; }

This command group executes in the current context and allows redirection.

Notice the trailing semicolon and whitespaces on both sides

Here is a good practice of command grouping.

Error handling & Debugging

• Stop on error

1
command || { echo "command error" >&2; exit 1; }

1
if ! command; then echo "command error" >&2; exit 1; fi

1
2
command
if [[ $? -ne 0 ]]; then echo "command error" >&2; exit 1; fi

• Useful when debugging

1
2
set -Eeux
set -o pipefail

Exit status

• Reserved codes

Exit statuses fall between 0 and 255. A 0 status means a command has executed successfully. A 1 status means a general error during execution.

$? gives the exit status of previous command.

References

• Bash Test Operators
• Brace expansion
• Character classes and bracket expression
• Cheatsheet
• Exit Status
• Filename expansion
• Globbing
• Parameter expansion
• Regular expression
• Special Characters
• The Set Builtin
• The Shopt Builtin
• Word splitting

Great sites

• Advanced Bash-Scripting Guide
• Dash Cheat Sheets
• Find command-line tools
• TLDR

• Previous
  FFmpeg Tricks
• Next
  The power of grep, sed, and awk