Debug later#
New in version 0.0.19.
ploomber-engine uses our debuglater package to serialize the error traceback so you can start a debugging session after your notebook crashes.
So, for example, if you’re running notebooks in production or remote servers, you can debug after they crash. Likewise, you can use the generated file to debug on a different machine (assuming the environment is the same) without having access to the source code.
Example#
Install requirements:
%pip install ploomber-engine --quiet
Note: you may need to restart the kernel to use updated packages.
Download sample notebook:
%%sh
curl https://raw.githubusercontent.com/ploomber/ploomber-engine/main/tests/assets/debuglater.ipynb --output debuglater-demo.ipynb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2056 100 2056 0 0 11237 0 --:--:-- --:--:-- --:--:-- 11296
Run the notebook with debug_later=True option (note that this notebook crashes on purpose):
from ploomber_engine import execute_notebook
execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)
Show code cell output
0%| | 0/2 [00:00<?, ?it/s]
Executing cell: 1: 0%| | 0/2 [00:00<?, ?it/s]
Executing cell: 2: 0%| | 0/2 [00:00<?, ?it/s]
Executing cell: 2: 50%|█████████████ | 1/2 [00:00<00:00, 2.01it/s]
---------------------------------------------------------------------------
ZeroDivisionError Traceback (most recent call last)
File /tmp/ipykernel_2608/3245917335.py:3
1 from ploomber_engine import execute_notebook
----> 3 execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/ploomber_core/telemetry/telemetry.py:745, in Telemetry.log_call.<locals>._log_call.<locals>.wrapper(*args, **kwargs)
743 result = func(_payload, *args, **kwargs)
744 else:
--> 745 result = func(*args, **kwargs)
746 except Exception as e:
747 metadata_error = {
748 # can we log None to posthog?
749 "type": getattr(e, "type_", None),
(...)
752 **_payload,
753 }
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/execute.py:168, in execute_notebook(input_path, output_path, parameters, log_output, profile_runtime, profile_memory, progress_bar, debug_later, verbose, remove_tagged_cells, cwd, save_profiling_data)
158 client = INIT_FUNCTION(
159 input_path,
160 display_stdout=log_output,
(...)
164 cwd=cwd,
165 )
167 try:
--> 168 out = client.execute(parameters=parameters)
169 except Exception:
170 if output_path:
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:467, in PloomberClient.execute(self, parameters)
459 add_debuglater_cells(
460 self._nb,
461 path_to_dump=self._debug_later
462 if isinstance(self._debug_later, (str, Path))
463 else None,
464 )
466 with self:
--> 467 self._execute()
469 if original is not None:
470 # restore original instance
471 InteractiveShell._instance = original
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:590, in PloomberClient._execute(self)
587 if self._progress_bar:
588 iterator.set_description(f"Executing cell: {execution_count}")
--> 590 self.execute_cell(
591 cell,
592 cell_index=index,
593 execution_count=execution_count,
594 store_history=False,
595 )
596 execution_count += 1
598 return self._nb
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:433, in PloomberClient.execute_cell(self, cell, cell_index, execution_count, store_history)
424 # Append to the start
425 self._nb.cells.insert(
426 0,
427 nbformat.v4.new_markdown_cell(
(...)
431 ),
432 )
--> 433 result.raise_error()
435 return output
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/IPython/core/interactiveshell.py:266, in ExecutionResult.raise_error(self)
264 raise self.error_before_exec
265 if self.error_in_exec is not None:
--> 266 raise self.error_in_exec
[... skipping hidden 1 frame]
File <ipython-input-1-514936153616>:3
1 x = 1
2 y = 0
----> 3 x / y
ZeroDivisionError: division by zero
Command-line equivalent
ploomber-engine nb.ipynb output.ipynb --debug-later
The above command generated an output.dump file which is the serialized traceback:
%%sh
ls *.dump
output.dump
We can use the dltr command (from our debuglater package) to start a debugging session:
dltr output.dump
Tip
By default, only built-in data structures are serialized, for other types, only their string representation is stored. If you want to serialize every data type: pip install 'debuglater[all]'