Back to Contributing / Oils Dev Cheat Sheet

Spec Tests are written with the sh_spec.py|https://github.com/oilshell/oil/blob/master/test/sh_spec.py framework. There are some comments at the top of that file.

They live in files like spec/smoke.test.sh, and can be run with shell functions test/spec.sh smoke (sorry they're in different dirs)

They are published with each release, on the quality page: https://oils.pub/release/latest/quality.html

They're also published at every commit, with our Soil CI.

Quick Start

Follow these steps in Contributing:

$ build/deps.sh fetch 
$ build/deps.sh install-wedges-fast

Then

$ test/spec.sh smoke          # run a single file, which lives in spec/smoke.test.sh

$ test/spec-py.sh ysh-all     # all YSH spec tests in parallel
$ test/spec-py.sh osh-all     # all OSH spec tests in parallel - this may be flaky

Useful shortcuts:

$ test/spec.sh smoke --range 11-12  # just run two tests
$ test/spec.sh smoke --range 11-12 --verbose  # show detailed error messages

$ test/spec.sh smoke -r 11-12 -v  # short flags

In general, run test/spec.sh FOO to run the cases in spec/FOO.test.sh. Example:

$ test/spec.sh array       # run spec/array.test.sh

Run C++ tests

$ test/spec-cpp.sh run-file smoke  # one file
$ test/spec-cpp.sh osh-all

Also see our README.md: https://github.com/oilshell/oil/blob/master/spec/README.md

Test-Driven Development

The idea behind the spec tests to figure out how OSH should behave (the spec) by taking an automated survey of the behavior of other shells. I follow a test-driven process like this:

  1. Write spec tests for a new feature.
  2. Make the spec tests pass on every shell except OSH. If shells differ in behavior, this may require annotations on the expected results.
    1. A given shell may not implement a feature. For example, bash and zsh both implement the dirs builtin, but mksh and dash don't.
    2. Shells may implement the same feature differently. For example, pushd in bash prints the stack to stdout, but pushd in zsh doesn't.
  3. Write code in OSH to make the tests pass.

After step 2, all columns should be green or yellow, except OSH. After step 3, the OSH column should be green or yellow as well.

Format of the Tests

The spec/foo.test.sh files are designed to be syntax-highlighted like normal shell scripts.

Each line is a "token". Lines with four hashes #### begin a test case. Lines with two hashes ## add metadata to the test case, e.g. assertions to make on status/stdout/stderr.

Basic Test:

#### test for echo
echo 1
## status: 0
## stdout: 1

Test with multiline assertion:

#### multiline test
echo 1
echo 2
## status: 0
## STDOUT:
1
2
## END

You can also add qualifiers to tests, to account for the different behavior of different shells. Example:

#### multiline test with qualifier
echo 1
echo 2
if test $SH = dash; then
  echo 3
fi
## status: 0
## STDOUT:
1
2
## OK dash STDOUT:
1
2
3
## END

You can also write assertions as JSON:

#### test for echo with tab
echo -e '1\t2'
## status: 0
## stdout-json: "1\t2\n"

Creating Temporary Files

The current directory in the spec tests is created for each pair of test and shell. For example, when the third test in builtin-cd is run by osh, it's current directory will be _tmp/spec-tmp/builtin-cd.test.sh/02-osh.

Notes