Debug later#
New in version 0.0.19.
ploomber-engine uses our debuglater package to serialize the error traceback so you can start a debugging session after your notebook crashes.
So, for example, if you’re running notebooks in production or remote servers, you can debug after they crash. Likewise, you can use the generated file to debug on a different machine (assuming the environment is the same) without having access to the source code.
Example#
Install requirements:
%pip install ploomber-engine --quiet
Note: you may need to restart the kernel to use updated packages.
Download sample notebook:
%%sh
curl https://raw.githubusercontent.com/ploomber/ploomber-engine/main/tests/assets/debuglater.ipynb --output debuglater-demo.ipynb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2056 100 2056 0 0 14500 0 --:--:-- --:--:-- --:--:-- 14581
Run the notebook with debug_later=True option (note that this notebook crashes on purpose):
from ploomber_engine import execute_notebook
execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)
Show code cell output
0%| | 0/2 [00:00<?, ?it/s]
Executing cell: 1: 0%| | 0/2 [00:00<?, ?it/s]
Executing cell: 2: 0%| | 0/2 [00:00<?, ?it/s]
Executing cell: 2: 50%|█████████████ | 1/2 [00:00<00:00, 1.96it/s]
---------------------------------------------------------------------------
ZeroDivisionError Traceback (most recent call last)
File /tmp/ipykernel_2878/3245917335.py:3
1 from ploomber_engine import execute_notebook
----> 3 execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/ploomber_core/telemetry/telemetry.py:700, in Telemetry.log_call.<locals>._log_call.<locals>.wrapper(*args, **kwargs)
698 result = func(_payload, *args, **kwargs)
699 else:
--> 700 result = func(*args, **kwargs)
701 except Exception as e:
702 metadata_error = {
703 # can we log None to posthog?
704 "type": getattr(e, "type_", None),
(...)
707 **_payload,
708 }
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/execute.py:180, in execute_notebook(input_path, output_path, parameters, log_output, profile_runtime, profile_memory, progress_bar, debug_later, verbose, remove_tagged_cells, cwd, save_profiling_data)
170 client = INIT_FUNCTION(
171 input_path,
172 display_stdout=log_output,
(...)
176 cwd=cwd,
177 )
179 try:
--> 180 out = client.execute(parameters=parameters)
181 except Exception:
182 if output_path:
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:500, in PloomberClient.execute(self, parameters)
490 add_debuglater_cells(
491 self._nb,
492 path_to_dump=(
(...)
496 ),
497 )
499 with self:
--> 500 self._execute()
502 if original is not None:
503 # restore original instance
504 InteractiveShell._instance = original
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:623, in PloomberClient._execute(self)
620 if self._progress_bar:
621 iterator.set_description(f"Executing cell: {execution_count}")
--> 623 self.execute_cell(
624 cell,
625 cell_index=index,
626 execution_count=execution_count,
627 store_history=False,
628 )
629 execution_count += 1
631 return self._nb
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:464, in PloomberClient.execute_cell(self, cell, cell_index, execution_count, store_history)
455 # Append to the start
456 self._nb.cells.insert(
457 0,
458 nbformat.v4.new_markdown_cell(
(...)
462 ),
463 )
--> 464 result.raise_error()
466 return output
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/IPython/core/interactiveshell.py:308, in ExecutionResult.raise_error(self)
306 raise self.error_before_exec
307 if self.error_in_exec is not None:
--> 308 raise self.error_in_exec
[... skipping hidden 1 frame]
File <ipython-input-1-514936153616>:3
1 x = 1
2 y = 0
----> 3 x / y
ZeroDivisionError: division by zero
Command-line equivalent
ploomber-engine nb.ipynb output.ipynb --debug-later
The above command generated an output.dump file which is the serialized traceback:
%%sh
ls *.dump
output.dump
We can use the dltr command (from our debuglater package) to start a debugging session:
dltr output.dump
Tip
By default, only built-in data structures are serialized, for other types, only their string representation is stored. If you want to serialize every data type: pip install 'debuglater[all]'