The shell interacts with a set of Unix tools in /bin and so forth. However, in many cases, those tools have grown functionality that overlaps with shell.

Unix Tools ...

Related: Ad Hoc Protocols in Unix

That Start Processes (in parallel)

• make and other build tools. make -j for parallel builds.
• xargs, -P for parallel execution, -I {} for substitution
  • Also GNU Parallel, which is mentioned in the bash manual.
• find -exec and -exec +

That Have Expression Languages

Expression languages must be fully recursive to count here.

With no lexer:

• find -- -a -o ! ( )
• test -- -a -o ! ( )
• expr -- arithmetic, subsumed by $(())

Languages with lexers:

• awk
• dtrace -- modelled after awk.

Honorable mention:

• strace also has a little expression language, but it's not fully recursive.

That Use Regexes

• grep, grep -E
• sed, sed --regexp-extended in GNU sed
• awk (extended only)
• expr
• find -regex
• bash itself.

That Receive Code Snippets (Remote Evaluation)

• tar has a --sed option.

That Have Printf-Style Formatting

See Appendix A: How to Quickly and Correctly* Generate a Git Log in HTML|http://www.oilshell.org/blog/2017/09/19.html

• find -printf (arbitrary filenames)
• stat -c (arbitrary filenames)
• curl --write-out %{response_code} -- URLs can't have arbitrary characters?
• printf itself (coreutils)
• time (/usr/bin/time) -- mostly numbers
• date -- mostly numbers
• bash
  • the printf builtin
  • the time builtin and the TIMEFORMAT string -- mostly numbers
  • the prompt string: \h \W
• ps --format

That Have Backslash Escaping

• awk -F '\t' -- same as awk -F $'\t'
• xargs -d '\t'
• (GNU cut -d doesn't understand tabs)
• find -printf

Non-standard tools:

• git log --pretty=format: (arbitrary descriptions)
• hg log --template -- http://hgbook.red-bean.com/read/customizing-the-output-of-mercurial.html (doesn't have \0 as far as I can tell.) Mercurial has its own template language like date: {date|isodate}\n\n (no $).

NOTE: grep should have a syntax for captures, like $1 $2 name: $name age: $age. sed just has & for the matched group.

With Quoting/Escaping Algorithms

• ls -q -b for unprintable chars in filenames
• printf %q for spaces in args
• ${var@Q} which is different than printf %q!!! See help-bash@ thread.

Arg Substitution

These are like "$@" in shell.

• xargs -I {} -- echo {}
• find -exec {} +

Could be replaced with $_ or @_ ("it").

With Tabular Output

• find / ls
• ps
• df (has -h and -H human-readable option, --output[=FIELD_LIST] but no format string)
• du -- has -0 for NUL output
• TODO: look at netstat, iostat, lsof, etc. Brendan Gregg's pages.

With File System Path Matching

• du --exclude
• rsync --include --exclude
• find -name, -regex, -wholename, etc.

That Format Binary Data

• od
• xxd
• hexdump -- has a % format language.

Honestly I don't understand the difference between these!

Misc Expression Languages

• getopts builtin spec, and /usr/bin/getopt
  • leading : means to do different error handling! Instead of the arg. Gah.

The Worst Offender

find starts processes (with -exec), has a recursive boolean expression language, regexes, globs, has % and backslash escapes (in -printf), and arg substitution ({} is like "$@"). It should be part of the shell!

It also doesn't give good parse error messages. Sometimes it just says "find: invalid expression" with no location information.

Wow this is crazy too:

The regular expressions understood by find are by default Emacs Regular Expressions, but this can be changed with the -regextype option.

$ find -regextype -help
find: Unknown regular expression type ‘-help’; valid types are ‘findutils-default’, ‘awk’, ‘egrep’, ‘ed’, ‘emacs’, ‘gnu-awk’, ‘grep’, ‘posix-awk’, ‘posix-basic’, ‘posix-egrep’, ‘posix-extended’, ‘posix-minimal-basic’, ‘sed’.

I didn't know there were that many regex types! And emacs is a really bad default!

Families of Unix Tools

• CSV, JSON, HTML, XML, recfile, 1991 paper!
  • https://www.reddit.com/r/oilshell/comments/719p6c/how_to_quickly_and_correctly_generate_a_git_log/dndp9uq/

Misc Problems

• coreutils time doesn't have millisecond resolution! https://stackoverflow.com/questions/16959337/usr-bin-time-format-output-elapsed-time-in-milliseconds. It's hard-coded right in time.c --- usecs/10000. very arbitrary.
• bash TIMEFORMAT has precision like %3R, but it doesn't have the exit code! Annoying.