Cell runtime

New in version 0.0.18: execute_notebook

Install requirements:

%pip install ploomber-engine matplotlib --quiet

Note: you may need to restart the kernel to use updated packages.

Example

Let’s create a sample notebook with a few calls to time.sleep:

import nbformat

nb = nbformat.v4.new_notebook()
cells = [
    "import time",
    "time.sleep(1)",
    "time.sleep(2)",
    "time.sleep(1)",
    "time.sleep(0.5)",
]
nb.cells = [nbformat.v4.new_code_cell(cell) for cell in cells]
nbformat.write(nb, "notebook.ipynb")

Let’s execute the notebook with profile_runtime=True

Command-line equivalent

ploomber-engine notebook.ipynb output.ipynb --profile-runtime

from ploomber_engine import execute_notebook

_ = execute_notebook("notebook.ipynb", "output", profile_runtime=True)

  0%|                                                     | 0/5 [00:00<?, ?it/s]

Executing cell: 1:   0%|                                  | 0/5 [00:00<?, ?it/s]

Executing cell: 2:   0%|                                  | 0/5 [00:00<?, ?it/s]

Executing cell: 2:  40%|██████████▍               | 2/5 [00:01<00:01,  1.99it/s]

Executing cell: 3:  40%|██████████▍               | 2/5 [00:01<00:01,  1.99it/s]

Executing cell: 3:  60%|███████████████▌          | 3/5 [00:03<00:02,  1.13s/it]

Executing cell: 4:  60%|███████████████▌          | 3/5 [00:03<00:02,  1.13s/it]

Executing cell: 4:  80%|████████████████████▊     | 4/5 [00:04<00:01,  1.08s/it]

Executing cell: 5:  80%|████████████████████▊     | 4/5 [00:04<00:01,  1.08s/it]

Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00,  1.14it/s]

Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00,  1.11it/s]

Customize plot

You might customize the plot by calling the plot_cell_runtime function and passing the output notebook, the returned object is a matplotlib.Axes.

from ploomber_engine.profiling import plot_cell_runtime

nb = execute_notebook("notebook.ipynb", "output.ipynb")

  0%|                                                     | 0/5 [00:00<?, ?it/s]

Executing cell: 1:   0%|                                  | 0/5 [00:00<?, ?it/s]

Executing cell: 2:   0%|                                  | 0/5 [00:00<?, ?it/s]

Executing cell: 2:  40%|██████████▍               | 2/5 [00:01<00:01,  1.99it/s]

Executing cell: 3:  40%|██████████▍               | 2/5 [00:01<00:01,  1.99it/s]

Executing cell: 3:  60%|███████████████▌          | 3/5 [00:03<00:02,  1.13s/it]

Executing cell: 4:  60%|███████████████▌          | 3/5 [00:03<00:02,  1.13s/it]

Executing cell: 4:  80%|████████████████████▊     | 4/5 [00:04<00:01,  1.08s/it]

Executing cell: 5:  80%|████████████████████▊     | 4/5 [00:04<00:01,  1.08s/it]

Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00,  1.14it/s]

Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00,  1.11it/s]

ax = plot_cell_runtime(nb)
_ = ax.set_title("My custom title")

Saving profiling data

You can save the profiling data by setting save_profiling_data=True.

%%capture
_ = execute_notebook(
    "notebook.ipynb", "output.ipynb",
    profile_runtime=True, save_profiling_data=True
)

import pandas as pd
pd.read_csv("output-profiling-data.csv")

	cell	runtime	memory
0	1	0.000391	NaN
1	2	1.001368	NaN
2	3	2.002565	NaN
3	4	1.001448	NaN
4	5	0.501020	NaN

Note: you must set profile_memory=True to get non-NA data saved for the memory usage.