Custom Tests
Custom tests allow you to use any installed framework. If you find yourself using a custom test frequently, let us know! We can add a template for it, or maybe even an entire new type of test.

Adding a Custom Test

To add a custom test, navigate to Checks in the left sidebar and use the New Check + button. Next, select the Custom Test option in the drop-down and see the settings below:

Settings

Select Template

Select a predefined check template to save yourself some time.

Description

A summary of the custom test. Shown to users if the task is live and checks are shared (which is the default).

Command

The command to run the custom test with (e.g. python3 {{test}}). Supports templates.

Test Contents

The actual code of the test.

Failed Pattern

A regular expression that will match the test's output when it fails.

Filename Override

Set a static filename to name the test as. If not provided, one will be generated using the format nt-test-XXXXXXXX.
For best results, start yours with nt-test- to prevent it from showing in the list of files. Alternatively, you can configure the task to run in a different directory or add the test name to the hidden files glob.

Build First

If enabled, the build script (if provided) will be executed before the test is executed.

Require Success

If enabled, the test will fail unless it returns a 0 exit status.