Quick Start#

ploomber-engine is a toolbox for executing notebooks.

It provides enhanced capabilities such as debugging, profiling and experiment tracking.

Installation#

pip install -U ploomber-engine

or

conda install ploomber-engine -c conda-forge

Run notebook#

Tip

There is a command-line interface available. Run: ploomber-engine --help to learn more.

Let’s download a sample notebook.

%%bash
curl https://raw.githubusercontent.com/ploomber/ploomber-engine/main/examples/display.ipynb --output nb.ipynb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2672  100  2672    0     0  11234      0 --:--:-- --:--:-- --:--:-- 11226
from ploomber_engine import execute_notebook

_ = execute_notebook("nb.ipynb", "output.ipynb", parameters=dict(x=1, y=2))
Hide code cell output
There's a new ploomber-engine version available (0.0.25), you're running 0.0.26.dev0. To upgrade: pip install ploomber-engine --upgrade
  0%|                                                    | 0/11 [00:00<?, ?it/s]
Executing cell: 1:   0%|                                 | 0/11 [00:00<?, ?it/s]
Executing cell: 2:   0%|                                 | 0/11 [00:00<?, ?it/s]
Executing cell: 2:  27%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š                  | 3/11 [00:00<00:00,  9.20it/s]
Executing cell: 3:  27%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š                  | 3/11 [00:00<00:00,  9.20it/s]
Executing cell: 4:  27%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š                  | 3/11 [00:00<00:00,  9.20it/s]
sending something to standard error

Executing cell: 5:  27%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š                  | 3/11 [00:00<00:00,  9.20it/s]
Executing cell: 5:  55%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹           | 6/11 [00:00<00:00,  9.12it/s]
Executing cell: 6:  55%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹           | 6/11 [00:00<00:00,  9.12it/s]
Executing cell: 7:  55%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹           | 6/11 [00:00<00:00,  9.12it/s]
Executing cell: 7:  73%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–      | 8/11 [00:01<00:00,  7.24it/s]
Executing cell: 8:  73%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–      | 8/11 [00:01<00:00,  7.24it/s]
Executing cell: 9:  73%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–      | 8/11 [00:01<00:00,  7.24it/s]
Executing cell: 10:  73%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–      | 8/11 [00:01<00:00,  7.24it/s]
Executing cell: 10: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 11/11 [00:01<00:00, 10.46it/s]

Log print statements#

_ = execute_notebook("nb.ipynb", "output.ipynb", log_output=True, progress_bar=False)
some message
sending something to standard error
showing our logo :)

Plot cell’s runtime#

Note

Runtime profiling requires extra dependencies: pip install matplotlib

_ = execute_notebook(
    "nb.ipynb", "output.ipynb", profile_runtime=True, progress_bar=False
)
sending something to standard error
_images/88fdae4b11f149c53a7d2cca25eddac4482e884defd882f4e27abaf42497cec1.png
ls output-*
output-runtime.png

Plot cell’s memory usage#

Note

Memory profiling requires extra dependencies: pip install matplotlib psutil

_ = execute_notebook(
    "nb.ipynb", "output.ipynb", profile_memory=True, progress_bar=False
)
sending something to standard error
_images/2d426b7c24dfbf3471e4943156d84069d9d5eb265f7e9561978cd0c8c513ae68.png
ls output-*
output-memory-usage.png  output-runtime.png