Testing Jupyter Notebooks

Jupyter Notebook tests allow you to test code and output from Jupyter Notebook cells. This comprises of two parts:

  1. Setting up the Jupyter Notebook.

  2. Creating a custom test for Jupyter Notebooks. This test must be used in conjunction with other tests.

The steps are described in detail below.

This process is a little cumbersome at the moment, but we'll be improving it a lot soon! To get notified when we do, just upvote this feedback post.

Set Up the Jupyter Notebook

First, in the Jupyter Notebook, edit the metadata for the cell you'd like to test. You can get to this by clicking View > Cell Toolbar > Edit Metadata.

Editing the Jupyter Notebook cell's metadata

You will need to add a unique identifier to the cell's metadata, for example, by generating a GUID using this site. You should add this value to the eid keyword, as shown below:

Adding an `eid` value to the Jupyter Notebook cell's metadata

Next, add the autosave magic command to the top cell of the notebook, as follows:

The `autosave` command

This step is necessary because notebooks must be saved before the test can be run. Once this is completed, hide the cells' metadata, save the Jupyter Notebook, and save the lesson files.

Create a Custom Test

Create a custom task with the following specifications:

Field

Content

Command

bash {{test}}

Test Contents

wget "https://storage.googleapis.com/codevolve-staging-misc/ipynb_export.py?bust=$RANDOM" -O .ipynb_export.py

python3 .ipynb_export.py 'name-of-notebook.ipynb'

Failed Pattern

NONE

Filename Override

.converter.sh

Require Success

Enable

For example:

Example of a Jupyter Notebook custom test

When this custom test is run, it will save the cell contents and output as .enb_<guid>.<extension> (see next section for more details).

For example, if the unique identifier you added to the cell's metadata is d5137d, and you would like to use a code pattern test on the code within the cell, you would direct the code pattern test to search within .*d5137d.py:

This Code Pattern Test looks at the file created by the Jupyter Notebook Custom Test

You will need one custom test per Jupyter Notebook you'd like to test and it should be enabled before any other tests, like so:

Enable Jupyter Notebook Custom Test before other tests

Files Created by Jupyter Notebook Custom Tests

The following explains what files are created by the Jupyter Notebook Custom Tests and use cases for each:

Cell Input

Cell input will be saved as .py files. These can then be checked using unit tests and code patterns.

Cell Output

Cell output will be saved as .txt or .stdout files, depending on whether a print statement was used. These can be used with code patterns and unit tests.

Errors

Errors are saved as .er files. This is useful for testing whether an error was raised correctly.

Images

Images created will be saved to .png files, which can be used with image tests. Any text associated with the images will be saved as .src files. You can these with code patterns.

Tip

If you are unsure of whether or not a user will use a print statement, and you would like to test the cell's output, you can create a test like the following:

Using a Unit Test on either a `txt` or a `stdout` file

If you get stuck, don't hesitate to reach out for help!