Cell runtime#

New in version 0.0.18: execute_notebook (API Reference)

Install requirements:

%pip install ploomber-engine matplotlib --quiet
Note: you may need to restart the kernel to use updated packages.

Example#

Let’s create a sample notebook with a few calls to time.sleep:

import nbformat

nb = nbformat.v4.new_notebook()
cells = [
    "import time",
    "time.sleep(1)",
    "time.sleep(2)",
    "time.sleep(1)",
    "time.sleep(0.5)",
]
nb.cells = [nbformat.v4.new_code_cell(cell) for cell in cells]
nbformat.write(nb, "notebook.ipynb")

Let’s execute the notebook with profile_runtime=True

from ploomber_engine import execute_notebook

_ = execute_notebook("notebook.ipynb", "output", profile_runtime=True)
  0%|                                                     | 0/5 [00:00<?, ?it/s]
Executing cell: 1:   0%|                                  | 0/5 [00:00<?, ?it/s]
Executing cell: 2:   0%|                                  | 0/5 [00:00<?, ?it/s]
Executing cell: 2:  40%|██████████▍               | 2/5 [00:01<00:01,  1.99it/s]
Executing cell: 3:  40%|██████████▍               | 2/5 [00:01<00:01,  1.99it/s]
Executing cell: 3:  60%|███████████████▌          | 3/5 [00:03<00:02,  1.13s/it]
Executing cell: 4:  60%|███████████████▌          | 3/5 [00:03<00:02,  1.13s/it]
Executing cell: 4:  80%|████████████████████▊     | 4/5 [00:04<00:01,  1.08s/it]
Executing cell: 5:  80%|████████████████████▊     | 4/5 [00:04<00:01,  1.08s/it]
Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00,  1.14it/s]
Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00,  1.11it/s]

../../_images/24520d87974d9e90f9454a9e04331c7db3f1187c7b05a5d0ade6f9dc5d8e4a66.png

We can also set the path for the plot with profile_runtime=<path_to_png>

Customize plot#

You might customize the plot by calling the plot_cell_runtime function and passing the output notebook, the returned object is a matplotlib.Axes.

from ploomber_engine.profiling import plot_cell_runtime

nb = execute_notebook("notebook.ipynb", "output.ipynb")
  0%|                                                     | 0/5 [00:00<?, ?it/s]
Executing cell: 1:   0%|                                  | 0/5 [00:00<?, ?it/s]
Executing cell: 2:   0%|                                  | 0/5 [00:00<?, ?it/s]
Executing cell: 2:  40%|██████████▍               | 2/5 [00:01<00:01,  1.99it/s]
Executing cell: 3:  40%|██████████▍               | 2/5 [00:01<00:01,  1.99it/s]
Executing cell: 3:  60%|███████████████▌          | 3/5 [00:03<00:02,  1.13s/it]
Executing cell: 4:  60%|███████████████▌          | 3/5 [00:03<00:02,  1.13s/it]
Executing cell: 4:  80%|████████████████████▊     | 4/5 [00:04<00:01,  1.08s/it]
Executing cell: 5:  80%|████████████████████▊     | 4/5 [00:04<00:01,  1.08s/it]
Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00,  1.14it/s]
Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00,  1.11it/s]

ax = plot_cell_runtime(nb)
_ = ax.set_title("My custom title")
../../_images/42a192aa9110f5c3a812b452cd1aa1b23faa5fa92906514617649a38ac216d54.png

Saving profiling data#

You can save the profiling data by setting save_profiling_data=True, or providing custom path to save

Enable save_profiling_data by setting as True#

The file will be saved as output-profiling-data.csv by default

%%capture
_ = execute_notebook(
    "notebook.ipynb", "output.ipynb", profile_runtime=True, save_profiling_data=True
)
import pandas as pd

pd.read_csv("output-profiling-data.csv")
cell runtime memory
0 1 0.000398 NaN
1 2 1.001378 NaN
2 3 2.002581 NaN
3 4 1.001573 NaN
4 5 0.501044 NaN

Enable save_profiling_data with custom file path#

Please be aware that the file path must end with the .csv format.

%%capture
_ = execute_notebook(
    "notebook.ipynb",
    "output.ipynb",
    profile_runtime=True,
    save_profiling_data="./my_output.csv",
)
import pandas as pd

pd.read_csv("my_output.csv")
cell runtime memory
0 1 0.000451 NaN
1 2 1.001410 NaN
2 3 2.002093 NaN
3 4 1.001508 NaN
4 5 0.501071 NaN

Note: you must set profile_memory=True to get non-NA data saved for the memory usage.