Cell runtime#
New in version 0.0.18: execute_notebook
(API Reference)
Install requirements:
%pip install ploomber-engine matplotlib --quiet
Note: you may need to restart the kernel to use updated packages.
Example#
Let’s create a sample notebook with a few calls to time.sleep
:
import nbformat
nb = nbformat.v4.new_notebook()
cells = [
"import time",
"time.sleep(1)",
"time.sleep(2)",
"time.sleep(1)",
"time.sleep(0.5)",
]
nb.cells = [nbformat.v4.new_code_cell(cell) for cell in cells]
nbformat.write(nb, "notebook.ipynb")
Let’s execute the notebook with profile_runtime=True
Command-line equivalent
ploomber-engine notebook.ipynb output.ipynb --profile-runtime
from ploomber_engine import execute_notebook
_ = execute_notebook("notebook.ipynb", "output", profile_runtime=True)
0%| | 0/5 [00:00<?, ?it/s]
Executing cell: 1: 0%| | 0/5 [00:00<?, ?it/s]
Executing cell: 2: 0%| | 0/5 [00:00<?, ?it/s]
Executing cell: 2: 40%|██████████▍ | 2/5 [00:01<00:01, 1.99it/s]
Executing cell: 3: 40%|██████████▍ | 2/5 [00:01<00:01, 1.99it/s]
Executing cell: 3: 60%|███████████████▌ | 3/5 [00:03<00:02, 1.13s/it]
Executing cell: 4: 60%|███████████████▌ | 3/5 [00:03<00:02, 1.13s/it]
Executing cell: 4: 80%|████████████████████▊ | 4/5 [00:04<00:01, 1.08s/it]
Executing cell: 5: 80%|████████████████████▊ | 4/5 [00:04<00:01, 1.08s/it]
Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00, 1.14it/s]
Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00, 1.11it/s]
We can also set the path for the plot with profile_runtime=<path_to_png>
Customize plot#
You might customize the plot by calling the plot_cell_runtime
function and passing the output notebook, the returned object is a matplotlib.Axes
.
from ploomber_engine.profiling import plot_cell_runtime
nb = execute_notebook("notebook.ipynb", "output.ipynb")
0%| | 0/5 [00:00<?, ?it/s]
Executing cell: 1: 0%| | 0/5 [00:00<?, ?it/s]
Executing cell: 2: 0%| | 0/5 [00:00<?, ?it/s]
Executing cell: 2: 40%|██████████▍ | 2/5 [00:01<00:01, 1.99it/s]
Executing cell: 3: 40%|██████████▍ | 2/5 [00:01<00:01, 1.99it/s]
Executing cell: 3: 60%|███████████████▌ | 3/5 [00:03<00:02, 1.13s/it]
Executing cell: 4: 60%|███████████████▌ | 3/5 [00:03<00:02, 1.13s/it]
Executing cell: 4: 80%|████████████████████▊ | 4/5 [00:04<00:01, 1.08s/it]
Executing cell: 5: 80%|████████████████████▊ | 4/5 [00:04<00:01, 1.08s/it]
Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00, 1.14it/s]
Executing cell: 5: 100%|██████████████████████████| 5/5 [00:04<00:00, 1.11it/s]
ax = plot_cell_runtime(nb)
_ = ax.set_title("My custom title")
Saving profiling data#
You can save the profiling data by setting save_profiling_data=True
, or providing custom path to save
Enable save_profiling_data by setting as True
#
The file will be saved as output-profiling-data.csv
by default
%%capture
_ = execute_notebook(
"notebook.ipynb", "output.ipynb", profile_runtime=True, save_profiling_data=True
)
import pandas as pd
pd.read_csv("output-profiling-data.csv")
cell | runtime | memory | |
---|---|---|---|
0 | 1 | 0.000406 | NaN |
1 | 2 | 1.044609 | NaN |
2 | 3 | 2.002575 | NaN |
3 | 4 | 1.001583 | NaN |
4 | 5 | 0.501062 | NaN |
Enable save_profiling_data with custom file path#
Please be aware that the file path must end with the .csv
format.
%%capture
_ = execute_notebook(
"notebook.ipynb",
"output.ipynb",
profile_runtime=True,
save_profiling_data="./my_output.csv",
)
import pandas as pd
pd.read_csv("my_output.csv")
cell | runtime | memory | |
---|---|---|---|
0 | 1 | 0.000418 | NaN |
1 | 2 | 1.001418 | NaN |
2 | 3 | 2.002526 | NaN |
3 | 4 | 1.001532 | NaN |
4 | 5 | 0.501027 | NaN |
Note: you must set profile_memory=True
to get non-NA data
saved for the memory usage.