0%
May 20, 2022

Shell Script Fundamentals

coding

Contents

Iterables

Types of Iterable Objects

There are several common objects that are iterable in shell scripts:

  • String separated by \n. For example, consider the string iterable:

    iterable=$'file1\nfile2'

# or

iterable="
  file1
  file2
"

for char in $iterable; do
  echo $char
done

    outputs

    file1
file2

  • glob (click to see detail) :

    for image in some/folder/*; do
  echo "$image"
done

    Due to character-encoding problem, it is never a good idea to loop the array generated by files=$(ls some/dir), as always we should use glob.

  • Array object like files=("file1" "file2"). Common source of array can be built using glob, for example, arr=($some/folder/*.jpg).

    There is special syntax to loop array elements:

    for file in ${files[@]}; do
  echo $file
done
Example of Iterables

Some real example I created in my work:

EXPER_IN=some/folder/INPUT
EXPER_OUT=some/folder/OUTPUT
# stock of images
STOCK=some/folder/stock
RESULT=some/folder/result

CONFIG_PATH=some/folder/config.json

images_arr=($STOCK/*.jpg)

# can choose either one of the below:
# for image in $images_arr; do
# for image in $STOCK/*; do
for image in $STOCK/*; do
  image_name=$(basename -- $image)
  image_name_with_jpg=$(echo $image_name | sed "s/.jpg//g")
  cp $STOCK/$image_name $EXPER_IN

  python dsds_main.py debug-defect --config $CONFIG_PATH --tag dev

  target_dir=$RESULT/$image_name_with_jpg
  mkdir $target_dir
  mv $EXPER_OUT/* $target_dir

  rm $EXPER_IN/$image_name
done

Remarks.

  • We use basename -- "a/b/c/d.jpg to get d.jpg.
  • There is no mv -r command, we just have mv.

If Statements and Booleans

No Booleans in Bash

From this post:

  • There are no Booleans in Bash
  • Always compare against strings or numbers Therefore asking how to construct boolean in bash is the same thing as asking how to write comparison in bash.

We usally wrap our comparison inside [[ comparison ]].

List of All Comparisons

Copied from here.

The [[ ]] form is generally safer than [ ] and should be used in all new code.

|

Operator | Purpose | | --------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------- | | | | | == | String equality | | != | String inequality | | < | String lexiographic | comparison (before) | | > | String lexiographic comparison (after) | | =~ | String regular expression match (bash 3 only, not currently allowed in ebuilds) | | -z "string" | String has zero length | | -n "string" | String has non-zero length | | -eq | Integer equality | | -ne | Integer inequality | | -lt | Integer less than | | -le | Integer less than or equal to | | -gt | Integer greater than | | -ge | Integer greater than or equal to | | -a file | Exists (use -e instead) | | -b file | Exists and is a block special file | | -c file | Exists and is a character special file | | -d file | Exists and is a directory | | -e file | Exists | | -f file | Exists and is a regular file | | -g file | Exists and is set-group-id | | -h file | Exists and is a symbolic link | | -k file | Exists and its sticky bit is set | | -p file | Exists and is a named pipe (FIFO) | | -r file | Exists and is readable | | -s file | Exists and has a size greater than zero | | -t fd | Descriptor fd is open and refers to a terminal | | -u file | Exists and its set-user-id bit is set | | -w file | Exists and is writable | | -x file | Exists and is executable | | -O file | Exists and is owned by the effective user id | | -G file | Exists and is owned by the effective group id | | -L file | Exists and is a symbolic link | | -S file | Exists and is a socket | | -N file | Exists and has been modified since it was last read | | file1 -nt file2 | file1 is newer (according to modification date) than file2, or if file1 exists and file2 does not. | | file1 -ot file2 | file1 is older than file2, or if file2 exists and file1 does not. | | file1 -ef file2 | file1 and file2 refer to the same device and inode numbers. | | first \|\| second | first or second (short circuit) | | first && second | first and second (short circuit) | | ! condition | not condition |

Example of If Statement
for image in echo $STOCK/*; do
  if [[ "$image" =~ \.jpg || "$image" =~ \.png ]]; then
      echo $image
  fi
done

Point to Values in an Array

Take images_arr=($STOCK/*.jpg) as in the example of iterables above, we can get specific element by:

  • ${images_arr[k]} the -th element
  • ${images_arr[-1]} the last element
  • ${images_arr[@]:0:k} the first elements

Read First few/Last few Lines or Skip Lines in std-out

Suppose I am going read a file by cat main.py, then:

  • Read the first 3 lines: 
    cat main.py | head -3
  • Read the last 3 lines: 
    cat main.py | tail -3
  • Skip 3 lines from the top: 
    cat main.py | tail -n +3

Try-Catch in Bash

We mix iterable and try-catch in this example. The principles are

  • command_1 && command_2 if command_1 succeeds, run command_2.
  • command_1 || command_2 if command_1 fails, run command_2.
trip01=some/other/folder/w0m8fnyb
trip02=some/other/folder/i8y6v_yn
trip03=some/other/folder/cght2jy4

IMAGES="
    $trip02/P202203252_732_4395.jpg
    $trip02/P202203252_743_4456.jpg
    $trip03/P202203253_299_1789.jpg
"
TARGET_DIR="some/folder/stock"
LOG=$(date +%Y-%m-%d--%Hh%Mm%Ss).log
# output 2022-05-20--10h24m52s.log:

for img in $IMAGES; do
  {
      {
          # try
          cp $img $TARGET_DIR
      } && {
          echo "copied from $img to $TARGET_DIR"
      }
  } || {
      # catch, save the log etc
      echo "fails to copy" > $LOG
  }
done

Remarks. The || will not surpress the exception handling of bash itself, it will keep reporting the root cause. The command after || helps us do extra work when problem occurs.

Reference