Links to Unix Shells, Languages, Libraries, Implementations, Research, and More

You probably came here for one of these pages:

• THE BIG LIST of Alternative Shells: fish, Elvish, NGS, nushell, etc.
• Another big list of Internal DSLs for Shell: libraries in languages like Python, Perl, Ruby, JavaScript, Scheme, Racket, Common Lisp, Scala, OCaml, Haskell, etc.
• Unifying Make and Shell: Several projects that combine shell and Make.

POSIX Shells

See the list starting at https://www.oilshell.org/cross-ref.html#bash

• GNU bash -- most popular shell in the world, on Linux, Mac, Windows
• ksh / pdksh / mksh -- ksh was a proprietary extension of the original Bourne shell; pdksh was an open source clone of ksh; mksh is a fork of pdksh and used on Android.
• dash / busybox ash (same lineage)
• busybox hush -- shell in one file.
• yash -- Yash, yet another shell, is a POSIX-compliant command line shell written in C99 (ISO/IEC 9899:1999). Yash is intended to be the most POSIX-compliant shell in the world while supporting features for daily interactive and scripting use. -- Has Debian and Fedora packages.
• zsh (not POSIX compatible by default)

Shell Parsers

• ShellCheck|https://github.com/koalaman/shellcheck -- Written in Haskell, using the parser combinator style. (No seperate lexer).
• shfmt|https://github.com/mvdan/sh -- Shell auto-formatter like gofmt, written in Go.
• sh-parser|https://github.com/jirutka/sh-parser -- parsing POSIX shell with Lua's LPEG
• morbig|https://github.com/colis-anr/morbig -- from Colis project, parsing POSIX shell, FOSDEM Talk|https://fosdem.org/2018/schedule/event/code_parsing_posix_s_hell/. alias makes parsing undecidable too!
• https://github.com/idank/bashlex -- bashlex is a Python port of the parser used internally by GNU bash. For the most part it's transliterated from C ... I wrote this library for another project of mine, explainshell which needed a new parsing backend to support complex constructs such as process/command substitutions.
• tree-sitter-bash -- grammar.js is 512 lines? There's also C++ code?
  • for bash-language-server

Academic Projects

• The Colis/Morbig project above
• Smoosh: The Symbolic, Mechanized, Observable, Operational Shell - Formalizing POSIX shell. Implementation in OCaml.

TODO: Add links here.

• Thompson shell papers
• Bourne Shell
• ksh paper (Usenix)
• bash papers
• A Pipe Has Two Ends. Using APL in a multiprocess/multiprocessor environment.|https://kar.kent.ac.uk/13447/ -- 1989. A proposal for a flexible but easy to use syntax. ... I offer for general consideration a device that allows data to be piped out of APL, through (a series of) shell commands, and back into APL.
• dgsh

Find-like Languages

• https://github.com/leahneukirchen/lr -- lr is a new tool for generating file listings, which includes the best features of ls(1), find(1), stat(1) and du(1).

Awk- and Sed-like Languages

Tab Language|https://tkatchev.bitbucket.io/tab/ -- An interesting statically-typed, non Turing complete language that apparently fills the niche of Awk. Written in C++.

Miller|http://johnkerl.org/miller/doc/index.html (Language Reference|http://johnkerl.org/miller/doc/reference.html) -- Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON. Written in C.

TXR|http://nongnu.org/txr/ -- TXR is a pragmatic, convenient tool ready to take on your daily hacking challenges with its dual personality: its whole-document pattern matching and extraction language for scraping information from arbitrary text sources, and its powerful data-processing language to slice through problems like a hot knife through butter. Many tasks can be accomplished with TXR "one liners" directly from your system prompt. There is a TXR Lisp and then it is embedded in a pattern language. Sort of like the reverse of a template language?

jq|https://stedolan.github.io/jq/ -- jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text. Written in C.

These projects are dormant:

r17|http://www.rseventeen.com/ (Github|https://github.com/matthewnourse/r17, written in C++) -- A flexible, scalable, relational data mining language. No releases since 2013.

• r17's syntax is a cross between UNIX shell and SQL.
• Built-in concurrency, including cross-machine concurrency.
• Strong type checking at stream-header-read time.

streem|https://github.com/matz/streem (by Matz, creator of Ruby, written in C with yacc grammar) -- Streem is a stream based concurrent scripting language. It is based on a programming model similar to the shell, with influences from Ruby, Erlang, and other functional programming languages.

Scientific Workflow Languages

Scientific Workflow System on Wikipedia has a huge list.

• Cuneiform -- Cuneiform combines the strong points of functional programming languages, distributed databases, and workflow management systems.
• Nextflow -- an external DSL, e.g. process { }. Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages.
• Common Workflow Language -- The Common Workflow Language (CWL) is a specification for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high performance computing (HPC) environments. CWL is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, Physics, and Chemistry.
• SciPipe -- SciPipe is a library for writing scientific workflows (sometimes also called "pipelines") of shell commands that depend on each other, in the Go programming language
• HN thread on dgsh mentions many similar systems.

Shell Complements

• ShellJs|https://github.com/shelljs/shelljs -- This is the opposite of a shell in JavaScript -- it's all the Unix utilities in JavaScript, and you use JS as your shell language.
• OCamp|https://github.com/let-def/ocamp -- OCamp extends unix shells with constructions to express memoization, sharing of computations and reactive programming. Written in OCaml.
• TODO: There was one about calling C functions from shared libraries.
• asdf -- not really a library, but a language-independent version manager

Pages with Collections of Links

• Unix Shell on Wikipedia|https://en.wikipedia.org/wiki/Unix_shell -- Some useful links, most of which are on this page.
• List of build automation software on Wikipedia|https://en.wikipedia.org/wiki/List_of_build_automation_software
• UNIX Shell Implementations on rosettacode.org|https://rosettacode.org/wiki/Category:UNIX_Shell_Implementations
• comp.unix.shell thread|http://unix.derkeiler.com/Newsgroups/comp.unix.shell/2009-10/msg00324.html -- A huge list of links to shell implementations.

Notes on Shell Implementation

• libc manual: Implementing a Job Control Shell|https://www.gnu.org/software/libc/manual/html_node/Implementing-a-Shell.html
• Notes on coprocesses|http://www.ict.griffith.edu.au/anthony/info/co-processing/general_hints.txt