test:
Test header
Names a single test case. The runner prints this name in pass and fail output. Keep it short and human.
test: "Login returns a valid session"
RFCTestML / 0.4Draft
The TestML language specification.
One file. Five runtimes. Same result. This spec defines the grammar, the type rules and the runner contract for TestML. Implementations that match it can claim conformance. The text below is normative.
Lexical structure
The grammar fits on one page.
TestML files use indented blocks. Two spaces per level. No tabs. Comments start with a hash. Strings use double quotes. The rest is data. This block is the canonical reference grammar.
# TestML v0.4 — abstract syntax (EBNF subset) file ::= block { block } ; block ::= key ":" ( value | indent_block ) ; key ::= /[a-z][a-z0-9_]*/ ; value ::= string | number | bool | null | regex ; string ::= '"' { utf8_char } '"' ; regex ::= "/" { regex_char } "/" [ flags ] ; indent_block ::= NEWLINE INDENT block { block } DEDENT ; # Whitespace and comment rules comment ::= "#" { any_char } NEWLINE ; indent ::= two spaces (tabs forbidden) ;
Core constructs
Six keywords. The whole language.
Every test file is built from these six top-level keys. A runner that reads them all and emits the right output is already conformant. Each card lists the shape and a short sample.
when:
Given a request
Describes the action under test. A method, a path, and an optional body. No setup hidden in code.
when: POST: "/api/login"
then:
Assert the result
Lists every assertion the runner must check. Status codes, JSON shapes, regex matches and ranges.
then:
status: 200
json.session: /sess_[a-z0-9]{16}/expects:
Data-driven matrix
Pairs inputs with outputs. The runner expands each row into its own test case. One file, many runs.
expects:
- { in: 2, out: 4 }
- { in: 3, out: 9 }fixtures:
Shared setup
Names reusable data blocks. Any test in the file can pull a fixture in by name. No copy and paste.
fixtures:
admin:
role: "admin"import:
Module references
Links to other TestML files. The runner resolves the path against the suite root. No language hooks.
import: - "./shared/auth.tml"
Type system
Small set of types. No surprises.
TestML keeps the value set small on purpose. Scalars match JSON. Patterns add regex and ranges. Structures stay flat. Any runner that handles these eight types covers the full surface.
scalar.stringQuoted UTF-8 text. Escape rules match JSON.
scalar.numberSigned integer or decimal. No octal, no hex.
scalar.boolLowercase true or false. Yes and no are rejected.
scalar.nullThe bare word null. Empty strings stay strings.
pattern.regexSlash-delimited like /\d{3}/. Flags follow the closing slash.
pattern.rangeInclusive ranges like 200..299 for status codes.
struct.mapKey and value pairs. Keys must be unique inside a block.
struct.listOrdered values. Mixed types allowed but rarely useful.
Runtime contract
Five steps every runner must follow.
The runtime contract is the part that keeps results portable. A run in Python and a run in Java must produce the same pass and fail lines. The five steps below define that path.
STEP 01
Parse the file
Read the .tml file as UTF-8. Build an AST. Reject unknown top-level keys with a clear line number.
STEP 02
Expand matrices
Walk every expects: block. Stamp out one test case per row. Inherit fixtures from the parent file.
STEP 03
Resolve fixtures
Merge named fixtures into each test scope. Later keys win. Cycles raise a clean error, not a stack trace.
STEP 04
Run the suite
Execute every test in source order by default. Parallel mode is allowed but must keep the same report format.
STEP 05
Emit results
Print one line per assertion. Pass lines start with a dot. Fail lines show the expected and the actual side by side.
Conformance
Three tiers. One badge per runner.
We grade runners against the public conformance suite. Bronze is the minimum bar. Silver covers fixtures and matrix expansion. Gold runners also support parallel runs and JUnit output.
Bronze
Parse the grammar and run the core constructs. Pass the public conformance suite at the base tier.
Silver
All Bronze rules plus fixtures, imports and matrix expansion. Report timings per test in milliseconds.
Gold
All Silver rules plus parallel runs, partial reruns and JUnit-compatible output. The badge ships with the runner.
Current runners and tiers
Each row lists the runner, the install command and the tier it last passed. We re-run the suite on every spec release.
Pythonpip install testmlGold
JavaScriptnpm i testmlGold
Rubygem install testmlSilver
Perlcpanm TestMLSilver
Javamaven: org.testml:coreBronze
Contribute
Read the source. File a clarification. Ship a runner.
The spec lives in the open. Send a pull request when you spot a gap. Open an issue when wording is unclear. New language runners are welcome at any tier.