This article is the first in a series of posts about automating everyday actions. We’ll start with Bash shell scripting, which allows you to write scripts to automate dull, repetitive tasks. You can also find it on my blog.
The great advantage of Bash shell scripting compared to writing a full CLI tool to do what we need is that it is very easy to do, especially for those familiar with the Bash shell already, but it is only feasible to build Bash scripts that can be used for a very limited range of applications.
Let’s identify a problem anyone can have at some point in life: editing some files according to the folder they’re stored in.
Let’s say you spend your summer vacation in Italy because you love when the weather is extremely hot, have heard there are beautiful cities, ancient churches, Roman Empire ruins and you want to taste typical Italian food in the place where it all started. However much time you decide to spend, Italy isn’t too big, there are fast trains, and you decide to visit several cities. Just like any tourist, you take pictures everywhere and, by the time you’re back home, you organize your pictures in folders.
At this point you can at least figure out where you were when each picture was taken from the date on which it was taken from the hotel/rail/bus/domestic plane reservations you must have because rental cars with automatic transmissions are very expensive and less than 20% of Americans can drive a manual. You arrange your pictures into subfolders: you have folders for each place you visited and you put those folders in three folders called North, Center and South because it’s nice to be able to know that when you’ll look at the pictures in the future.
That in particular is a problem: looking at the pictures will now require you to browse and select pictures in each subfolder manually, and that’s especially painful on some less advanced I/O devices like TV remotes.
It would be ideal to be able to take all of those neatly organized pictures you have, put them all in one folder with a watermark telling you where you’ve taken them so it won’t look like you would have been better off just downloading some pictures from the Web when you show them to your friends (or kids) 10 years after you’ve been there.
Doing that by hand will require another 10 years and your friends or kids will already have visited all of the places you’ve visited fifty times by the time you are done. Fortunately, there is a way to make that quick and easy, and that is by embracing Bash shell scripting.
Solving this problem requires two different levels of difficulty depending on what operating system you’re running. If you’re running most GNU/Linux distributions or macOS, Bash is your default shell, so it’s already installed and you can go on with the next section without having to install anything else. If you’re running any other Unix or Unix-like operating system that doesn’t install Bash by default, installing Bash is generally very easy and you can find specific instructions online in the very unlikely case you don’t already know how to install a package or port called bash on your OS of choice.
If you’re running Windows, you’ll have to install the WSL (Windows Subsystem for Linux) by installing one of the packages available in the Windows Store that include Bash. You just need to open the Windows Store, search for Ubuntu (for example) and install it. When starting for the first time, it will prompt you to enter a username and password you’ll need to remember. The password won’t even be shown to you in the form of asterisks, in case you’re confused by the fact it seems like you’re not actually typing anything in.
After that, you’ll be able to access Bash in any directory on your PC by running the
bash
command. You can exit the Bash shell by running the
exit
command.
More details about how to use it will be provided in the rest of the article.
At its simplest, a Bash script is just a list of shell commands separated by newlines or concatenated together using pipes or some of the many script-oriented constructs Bash includes.
This section will be a very quick introduction to the usage of the Bash shell and the Unix command line in general, given that most shells are very similar when it comes to the most basic tasks. There are plenty of books available online that will teach you how to use it, many of which are aimed directly at Linux users, but they also apply to other Unix-like operating systems and to the Windows Subsystem for Linux.
The first thing to understand about any command line interface is that it’s like using a file manager: at any point you’re operating in a specific directory, called the working directory. Running the command (by typing it in and then pressing enter)
pwd
will return the current working directory.
The directory structure of Unix-like operating systems is a tree that branches out from the single root directory, the path of which is simply the character /. Other directories are chained after that separated by forward slashes. For example, the home directory at the root of the tree is found at path
/home
and a hypothetical user directory inside that would be at path
/home/user
Paths can be also expressed as relative paths, based on the current CLI working directory. The current working directory is expressed as ./ and the parent directory (the directory that contains the working directory) is expressed as ../.
You can change the working directory using the cd command followed by the path of the directory, expressed either as a relative path or an absolute path. For example, if you want to change the working directory to the parent directory, you’d write
cd ../
The trailing slash can be omitted and, if you’re moving into a subdirectory you can omit the ./ at the start, making the commands
cd ./Pictures/italy_pics/
and
cd Pictures/italy_pics
equivalent.
Files can be copied using the cp command, which takes two arguments: the path to the file to be copied and the path where you want the copy to be created, including the file name if you want the copy to have a different name: if you have a file called pic001.png in the italy_pics subdirectory and you want to copy it to the current working directory retaining the original file name, you’d run one of the following three commands (in decreasing order of command length)
cp italy_pics/pic001.png ./pic001.png
cp italy_pics/pic001.png ./
cp italy_pics/pic001.png .
The command to move files is mv and you use it just like cp, except for the fact that it can be used to rename files by trying to move a file into the same directory it came from but with a different file name:
mv italy_pics/pic001.png italy_pics/pic1.png
While using the interactive shell, you can use the Tab keyboard button to get automatic completion of commands and arguments when there is only one choice or get a list of possible option. This is not relevant for Bash scripting, but will be more relevant in the coming articles.
Bash is an interpreted language and Bash scripts are ran mostly just like .py files.
To run a bash script saved in a file called script.sh, open a terminal window in the same directory as the script and run
bash script.sh
But there is actually a better way: just like with Python scripts, you can add a line at the top of the file, called the shebang line.
The shebang line consists of the two characters #! followed by the path to the interpreter to be used to run the script. In the case of bash, it is found at (or symlinked to) /bin/bash in pretty much every environment in which Bash installed, so you can add
#!/bin/bash
at the very top of your script so that the shell knows what interpreter to run.
This is useful because you can make the script executable with
chmod +x script.sh
and then run it just like any executable with
./script.sh
If it’s a script you think you’ll need to use often, you can either add it to the systemwide binary file paths (where the packages you download are installed) or create a bin folder in your home directory and copy the script there. For example, rename the script file to the command to the name you want to give to the command, for example myfirstscript using mv and then create the ~/bin directory and copy the file there with the following three commands (in an interactive Unix shell or Bash shell on Windows):
mv script.sh myfirstscript
mkdir ~/bin
cp myfirstscript ~/bin/
and you can run the script simply by running
myfirstscript
from any working directory as long as you’re using the same user account you used to copy the file.
Let’s start writing a Bash script by making a script that copies all of our organized pictures into a single directory and renameames them according to the place where they were taken. This is not quite as good as the watermark we wanted, but let’s do one thing at a time
Open any text editor and create a file called picorganizer in the ~/bin directory. The first thing you’ll need to add is the shebang line
#!/bin/bash
Make it executable right away by opening a terminal and running
chmod +x ~/bin/picorganizer
To solve the problem we have, we need to:
List the files in the directories we need to copy the files from
For each file we need to take the following three actions
The example directory tree we’ll be working with (that you can get by running the tree command) will be the following
italy_pics/
├── Center
│ ├── Assisi
│ ├── Florence
│ ├── Marche
│ ├── Pisa
│ ├── Rome_Lazio
│ └── Siena
├── North
│ ├── EmiliaRomagna
│ ├── Genoa_CinqueTerre
│ ├── Milan_Lombardy
│ ├── Trentino
│ ├── Turin
│ └── Venice
└── South
├── Bari_Apulia
├── Basilicata
├── Calabria
├── Campobasso
├── Naples_Campania
└── Sicily
where each city/region name is a directory containing the pictures taken in that place. These are example places in Italy and do not necessarily represent places I would recommend going to, don’t judge me for a semi-pseudo-random selection of places.
To make the script aware of what we’re working with, we need to get a list of files and directories and store them in a Bash variable. Let’s start by learning the command to list files and directories.
In the Bash interactive shell, we can use the
ls
command to simply list the files and directories contained in the working directory, or you can run it with a path argument, like this:
ls /path/to/dir
to list the files and directories contained in /path/to/dir.
We can save the output of the ls command to a variable called list by writing, in our Bash script, the following:
list=$(ls)
where $(command) means whatever command prints to standard output. You can then use that variable just by prefixing the variable name with $ or by prefixing it with $ and enclosing the variable name in square brackets: just
ls
is equivalent to
files=$(ls)
echo "$files"
and to
files=$(ls)
echo "${files}"
This is not actually what we need right now, though: Bash’s for in loop is able to iterate over files in the working directory very easily, and we can just nest them and get to the pictures very quickly:
for area in *; do
for city in ${area}/*; do
for picture in ${area}/${city}/*; do
# do something with ${area}/${city}/${picture}
done
done
done
We can simply copy them all to another directory renamed to reflect where they were taken by adding the cp command to the innermost for loop:
mkdir ../italy_pics_organized
i=1
for area in *; do
for city in ${area}/*; do
for picture in ${city}/*; do
extension=${picture##*.}
cityname="${city##*/}"
cp ${picture} "../italy_pics_organized/${i}-${cityname}(${area}).${extension}"
((i++))
done
done
done
There are a few things I haven’t yet explained and used here. Now I’ll explain them.
First of all, Bash doesn’t have variable types: every variable is a string and it doesn’t implement any mathematical operators or commands directly, so we need to use arithmetic expansion, which supports some specific Shell Arithmetic operators.
The ${parameter##word} expression used to get the city name and extension in the following way: it looks inside the parameter for the pattern (word in this notation) we specify after ## (in our case it’s *. for the extension and */ for the city name) and only returns the rest of the parameter, deleting the pattern (but keeping it in the original variable). You can find more information about this and the rest of what can be done with parameter expansion using the $ sign here.
We are doing something, and the script isn’t going to get much more complicated than that, but we aren’t adding a watermark yet. That’s because there isn’t a built-in tool to do that. No worries, though: the shell is expandable in the easiest way possible: by installing some software that provides a CLI interface.
The tool for the job when it comes to image manipulation is ImageMagick, which you can install by following the instructions on its own official download page.
On Linux, what I actually recommend you to do is to install the ImageMagick package on Fedora/RHEL/CentOS by running
sudo dnf install ImageMagick
on Fedora or RHEL/CentOS 8 or by running
sudo yum install ImageMagick
on RHEL/CentOS 7 or earlier.
On Ubuntu, you can install the imagemagick package using APT by running
sudo apt install imagemagick
When using WSL with Ubuntu installed on top of Windows, you need to follow instructions for installation on Ubuntu while inside the Bash shell interface.
ImageMagick provides, among other things, a command called convert, which can be used, in conjunction with the annotate functionality, to add watermarks to images by running a command that looks like the following:
convert input.png -fill "textcolor" -pointsize textsize -gravity WhereTheTextWillBe -annotate +offsetHorizontal+offsetVertical "watermark text" output.png
where you need to replace textcolor with either a color name or an RGB hexadecimal color code (e.g. green or #76ff03), textsize with a number specifying the size of the font (e.g. 10 for a small font, 100 for a big font), WhereTheText will have to be replaced with something along the lines of NorthEast or SouthWest according to where you want the text to be, and paddingHorizontal and paddingVertical are offsets that can be used to move the text around or, more often, away from the edges. input.png and output.png have to be replaced with paths to the input and output pictures.
For our example, the command I chose, with ${picture}, ${watermarktext} and ${saveto} being variables, is:
convert ./${picture} -fill "white" -pointsize 90 -gravity SouthEast -annotate +30+30 "${watermarktext}" "${saveto}"
So the final script is:
#!/bin/bash
mkdir ../italy_pics_organized
i=1
for area in *; do
for city in ${area}/*; do
for picture in ${city}/*.jpg; do
cityname="${city##*/}"
extension="${picture##*.}"
saveto="../italy_pics_organized/${i}-${cityname}(${area}).${extension}"
watermark="${cityname} (${area})"
convert ./${picture} -fill "white" -pointsize 90 -gravity SouthEast -annotate +30+30 "${watermark}" "${saveto}"
((i++))
done
done
done
After Ben Sinclair in the comments to this post on dev.to noticed that this wouldn’t handle spaces in the path properly, I need to point out that you need to change the character used by Bash to separate items to loop through in the for loop by adding two lines at the top like the following:
IFS='
'
which sets the separator to the newline character (\n, aka the LF character in character encoding specifications), so that the script ends up being this:
#!/bin/bash
mkdir ../italy_pics_organized
i=1
IFS='
'
for area in *; do
for city in ${area}/*; do
for picture in ${city}/*.jpg; do
cityname="${city##*/}"
extension="${picture##*.}"
saveto="../italy_pics_organized/${i}-${cityname}(${area}).${extension}"
watermark="${cityname} (${area})"
convert ./${picture} -fill "white" -pointsize 90 -gravity SouthEast -annotate +30+30 "${watermark}" "${saveto}"
((i++))
done
done
done
Here are a few things I didn’tell you you might need to know in the future when working with Bash or browsing Bash-related documentation.
Usually, when reading documentation about Unix command line usage (including the sections of my book about cross-platform mobile app development that concern Linux installation or CLI usage, for example) you might find that the commands are prefixed with the character $, like in the following example:
$ ls -alh
or are prefixed with #, like in the following example:
# vim /etc/fstab
Those prefixed with $ are meant to be executed as an unprivileged, regular user. Those prefixed with # are meant to be executed by the root account or by using sudo.
You might want to clean up and use the command
rm -rf *
if you know the working directory is going to be each of the directories in which you have pictures organized the old way, for example. Only use such a destructive command if you’re 100% sure there is no way for it to get executed in the wrong folder. Make sure to at least turn it into something that only deletes files with the extension you want to delete like this
rm -rf *.jpg
rm -rf *.jpeg
If you’re thinking nobody would be so dumb not to think of it, there is at least one exception in the world. Some years ago I was a bit too confident and wrote a shell script that did some cleaning up afterwards. At some point during the execution of the script, it executed rm -rf * in my home folder. That’s not great.
I only figured it out when it was halfway through deleting the Documents folder, and it had already deleted the ~/bin folder containing, ironically, most of my commonly used (and harmless) bash scripts (some of which I already was using on some remote servers and that I was able to recover) and itself in the process. I had recent backups of most of the important stuff, so it wasn’t the end of the world for me, but I can’t say it wasn’t annoying.
Bash has an if clause, I just didn’t feel like adding more complication to the script (even though it would have been better for it) by adding functionality that requires its use, its basic syntax is
if [[ condition ]]; then
# do something
fi
You can find more information about it online, and online you’ll also find a lot more information on Bash than what it’s made sense to include in this post.
Stay in touch with me on Twitter @carminezacc or follow my blog to know when the next post (about making a full-featured CLI tool with Python) comes out. Also, if you’re interested in mobile development, check out my book on Flutter.
Thanks to Ben Sinclair on dev.to for finding out I had accidentally left spaces around the assignment operator in the two files=$(ls) code snippets, that I should have added ./ before ${picture} so that there’s no chance the directory name will be interpreted as an option if it starts with an hyphen and for noticing that you might have spaces in one of the directory names or in the name of one of the picture, which would have broken the script.