TestML Quick Learners Guide

This guide is intended to teach TestML to experienced programmers as quickly as possible. It is simply a list of assertions and short explanations about all the things currently known about TestML. Enjoy.


TestML is Logical Awesome

http://testml.org/is/logical/awesome/


A TestML test suite consists of TestML documents.

Typically a software project that uses TestML will have a group of TestML files. The files are called ``documents''.

Here is an example TestML document:

    # Meta
    %TestML: 1.0
    %Plan: 3      # 2 pod + 1 wiki test
    # Test
    pod.to_html() == html;
    wiki.to_html() == html;
    # Data
    === Bold
    --- pod
    I like B<pie>.
    --- wiki
    I like *pie*.
    --- html
    I like <b>pie/b>.
    === Italic
    --- pod
    I I<like> pie.
    --- html
    I <i>like</i> pie.


TestML documents are implementation independent.

One should never put implementation or programming language specific commands into a TestML document. These things go in other places, like the bridge class or runner scripts.


A TestML document consists of 3 distinct parts.

A document has a Meta section, a Test section and a Data section.

The Meta section is for meta information.

The Test section contains the program statements that controls the test.

The Data section is contains blocks of named data points the the tests run against.


A single TestML document can be made up of more than one file.

Using meta section directives, sections can be pulled in from other files. This allows for the reuse of common parts.

    %Data: file1.tml
    %Data: file2.tml
    %Data: _         # '_' is the special name for the inline data section


The Meta section contains parsing and runtime directives.

Some statements in the meta section tell TestML how to parse the rest of the document. Some of them say (in extremely generic forms) how to run the tests.


Every TestML document requires a TestML version directive.

The first (non-comment) statement in any TestML document must be:

    %TestML: 1.0

where '1.0' is the version of the spec that the document conforms to. This way the parser knows exactly what to expect, if it can parse the document at all.


Words that begin with a capital letter are defined by TestML.

Keywords in any section of a TestML that begin with a capital letter are defined by the TestML specifications.

Words beginning with lower case are user defined.

This namespacing allows the standard library to grow with breaking old tests.


Test statements come in several forms.

    # Text starting with '#' are comments.
    function();         # A function is also called a "transform".
    point;              # Without parens, a word is a "point".
    point.function();   # Chained together they are called "expressions".
    point; function();  # The chained parts are called "subexpressions".
    point.function().function();   # Chain as many as you want.
    function('string', "string", expression);   # Functions take "arguments".
    expression1 == expression2;   # An assertion.
    expression.EQ(expression2);   # Same thing as previous.


Test statements only come in one form.

Every statement is really just ``an expression that can optionally end with an assertion function call''. From the spec:

    test-statement := test-expression assertion-expression? ';'


A data point is just a special transform.

These two statements are the same:

    point1;
    Point('point1');

The Point transform sets the current value of the expression to the point value from the block.


A bareword point name selects the blocks for a statement.

There is an implicit selection of the blocks in a document that apply to a given statement. A document might have 12 data blocks, but a given test statement might only apply to 3 of them, and the next statement might apply to 5 of them, while yet another statement might apply to all 12.

Consider the statement:

    foo.bar(baz) == gorch.quux();

There are 3 bareword points: 'foo', 'baz' and 'gorch'. Only the data blocks that contain at least these 3 points will be used. Then the statement will be run one time against each of the selected blocks. If no blocks are selected, then the statement will not be run.


I lied.

These two statements are NOT the same:

    point1;
    Point('point1');

The second statement does not select any blocks. The truly equivalent statement would be:

    Select(point1).Point('point1');


All test statements end with a semicolon.

And whitespace and comments are liberal:

    function(); function(
        'string'               # Comment
    ) == point;


Statements that don't select any blocks raise a runtime exception.

Consider:

    foo();

This is a valid statement to the parser, but it defines no data points. Therefore, we know that it selects no blocks and will not be run. TestML requires that it throw an exception. This is so that later versions might define a valid behavior without breaking old tests.

This implies that a document with zero data blocks, will fail at runtime too.


TestML is intended to run with all existing test frameworks.

TestML does not define any test harness, nor is it aligned with any particular test framework. Some languages, like Perl, have one strongly accepted test methodology. Others, like Python, have many. TestML is intended to be made to work with as many of them as possible

In practice, there is a TestML class called a Runner. There should be a subclass of Runner for every framework that TestML works with. For instance, most Perl testing is done with the TAP framework, so there is a class called TestML::Runner::TAP.


One common test pattern is for data transforms.

Consider a test that parses YAML and compares to JSON:

    %TestML: 1.0
    yaml.load().to_json() == json;
    === Hash
    --- yaml
    foo: bar
    --- json
    {"foo": "bar"}
    
    === Array
    --- yaml
    - foo
    - bar
    --- json
    ["foo", "bar"]

In this test scenario each block is tested once to make sure the YAML matches the JSON after being loaded and serialized.


Another common pattern is to load a fixture first.

    %TestML: 1.0
    fixture.Hash.new().Stash('myobject');
    Fetch('myobject').set(property, value).to_yaml() == yaml;
    ===
    --- fixture
    --- name: Mary
    --- sex: female
    ===
    --- property: age
    --- value: 35
    --- yaml
    age: 35
    name: Mary
    sex: female
    ===
    --- property: weight
    --- value: 135
    --- yaml
    name: Mary
    sex: female
    weight: 135

The first statement sets up a state that can be made use of by the rest of the tests.


The Data section is an ordered list of labeled mappings.

A data section is just a list of data blocks. The blocks are mappings of keywords to string values. The blocks have an optional descriptive label.

   === A label for the first block
   --- point1: a phrase value
   --- point2
   a multi
   line value
   === Label for 2nd block
   --- point1: foo
   --- point2: bar

If more than one file is used the lists are combined in order into one list.


A Data section can be in TestML markup, YAML, JSON or XML.

Any of these formats can be used to serialize the same TestML data blocks.

In fact, if you use more than one data section source, they can be in different formats.

The format within a single file must be the same throughout. ie You can't mix XML and YAML in the same file.


Errors can be caught and tested.

When a transform function dies or raises an exception, the runtime will catch it and skip all following transforms until a 'Catch()' transform is specified. If there is no catch, the error will be rethrown at the end of the expression evaluation.

You can do something like this:

    foo.failing_transform().not_called().Catch() == 'Actual error message';