The TestML language specification.

02Lexical structure

The grammar fits on one page.

TestML files use indented blocks. Two spaces per level. No tabs. Comments start with a hash. Strings use double quotes. The rest is data. This block is the canonical reference grammar.

# TestML v0.4 — abstract syntax (EBNF subset)
file        ::= block { block } ;
block       ::= key ":" ( value | indent_block ) ;
key         ::= /[a-z][a-z0-9_]*/ ;
value       ::= string | number | bool | null | regex ;
string      ::= '"' { utf8_char } '"' ;
regex       ::= "/" { regex_char } "/" [ flags ] ;
indent_block ::= NEWLINE INDENT block { block } DEDENT ;

# Whitespace and comment rules
comment     ::= "#" { any_char } NEWLINE ;
indent      ::= two spaces (tabs forbidden) ;

03Core constructs

Six keywords. The whole language.

Every test file is built from these six top-level keys. A runner that reads them all and emits the right output is already conformant. Each card lists the shape and a short sample.

test: "Login returns a valid session"

when:
  POST: "/api/login"

then:
  status: 200
  json.session: /sess_[a-z0-9]{16}/

expects:
  - { in: 2, out: 4 }
  - { in: 3, out: 9 }

fixtures:
  admin:
    role: "admin"

import:
  - "./shared/auth.tml"

04Type system

Small set of types. No surprises.

TestML keeps the value set small on purpose. Scalars match JSON. Patterns add regex and ranges. Structures stay flat. Any runner that handles these eight types covers the full surface.

05Runtime contract

Five steps every runner must follow.

The runtime contract is the part that keeps results portable. A run in Python and a run in Java must produce the same pass and fail lines. The five steps below define that path.

1. STEP 01
   Parse the file

   Read the .tml file as UTF-8. Build an AST. Reject unknown top-level keys with a clear line number.

2. STEP 02
   Expand matrices

   Walk every expects: block. Stamp out one test case per row. Inherit fixtures from the parent file.

3. STEP 03
   Resolve fixtures

   Merge named fixtures into each test scope. Later keys win. Cycles raise a clean error, not a stack trace.

4. STEP 04
   Run the suite

   Execute every test in source order by default. Parallel mode is allowed but must keep the same report format.

5. STEP 05
   Emit results

   Print one line per assertion. Pass lines start with a dot. Fail lines show the expected and the actual side by side.

06Conformance

Three tiers. One badge per runner.

We grade runners against the public conformance suite. Bronze is the minimum bar. Silver covers fixtures and matrix expansion. Gold runners also support parallel runs and JUnit output.

Current runners and tiers

Each row lists the runner, the install command and the tier it last passed. We re-run the suite on every spec release.