GNU Parallel

Written: 25/10/2023

In multiple occasions i'd have to run a some shell command multiple times but with different inputs, each command not having a dependency on the other. Examples on this can include:

Cloning a list of repositories.
"cat"ing a list of files.

The Problem

You have a command you want to run with different inputs in parallel to speed up execution like:

        $ git clone https://github.com/repository1.git
$ git clone https://github.com/repository2.git
$ git clone https://github.com/repository3.git
$ git clone https://github.com/repository4.git
# Or ..
$ cat ./file1
$ cat ./file2
$ cat ./file3
$ cat ./file4
$ cat ./file5

The solution

GNU Parallel command can help us acheive this by supplying it the command that we want with placeholders. Example for the above problems can be:

      # Will clone all of them in parallel
$ parallel git clone "https://github.com/{}" ::: repository1 repository2 repository3 ...

# Or from a file by redirecting stdin..
$ cat {} < list_of_files

In my humble experience, using parallel will allow you to elimniate most of the loops that you write in your shell scripts.