Adding Custom Tests

Custom tests allow you to use any installed framework. If you find yourself using a custom test frequently, let us know! We can add a template for it, or maybe even an entire new type of test.

A custom test's options are described in detail below.

Select Template

Select a predefined test template to save yourself some time.


A summary of what the custom case is checking. Shown to users if test is shared.


The command to run the custom test with (e.g. python3 {{test}}). Supports templates.

Test Contents

The actual code of the test (code editor coming soon!).

Failed Pattern

A regular expression that will match the test's output when it fails.

Filename Override

Set a static filename to name the test as. If not provided, one will be generated using the format nt-test-XXXXXXXX.

For best results, start yours with nt-test- to prevent it from showing in the list of files. Alternatively, you can configure the requirement to run in a different directory or add the test name to the hidden files glob.

Build First

If enabled, the build script (if provided) will be executed before the test is executed.

Require Success

If enabled, the test will fail unless it returns a 0 exit status.