Debug later#
New in version 0.0.19.
ploomber-engine uses our debuglater package to serialize the error traceback so you can start a debugging session after your notebook crashes.
So, for example, if you’re running notebooks in production or remote servers, you can debug after they crash. Likewise, you can use the generated file to debug on a different machine (assuming the environment is the same) without having access to the source code.
Example#
Install requirements:
%pip install ploomber-engine --quiet
Note: you may need to restart the kernel to use updated packages.
Download sample notebook:
%%sh
curl https://raw.githubusercontent.com/ploomber/ploomber-engine/main/tests/assets/debuglater.ipynb --output debuglater-demo.ipynb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2056 100 2056 0 0 7766 0 --:--:-- --:--:-- --:--:-- 7787
Run the notebook with debug_later=True option (note that this notebook crashes on purpose):
from ploomber_engine import execute_notebook
execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)
Show code cell output
0%| | 0/2 [00:00<?, ?it/s]
Executing cell: 1: 0%| | 0/2 [00:00<?, ?it/s]
Executing cell: 2: 0%| | 0/2 [00:00<?, ?it/s]
Executing cell: 2: 50%|█████████████ | 1/2 [00:00<00:00, 2.01it/s]
---------------------------------------------------------------------------
ZeroDivisionError Traceback (most recent call last)
File /tmp/ipykernel_3304/3245917335.py:3
1 from ploomber_engine import execute_notebook
----> 3 execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/ploomber_core/telemetry/telemetry.py:644, in Telemetry.log_call.<locals>._log_call.<locals>.wrapper(*args, **kwargs)
642 result = func(_payload, *args, **kwargs)
643 else:
--> 644 result = func(*args, **kwargs)
645 except Exception as e:
646 metadata_error = {
647 # can we log None to posthog?
648 "type": getattr(e, "type_", None),
(...)
651 **_payload,
652 }
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/execute.py:179, in execute_notebook(input_path, output_path, parameters, log_output, profile_runtime, profile_memory, progress_bar, debug_later, verbose, remove_tagged_cells, cwd, save_profiling_data)
169 client = INIT_FUNCTION(
170 input_path,
171 display_stdout=log_output,
(...)
175 cwd=cwd,
176 )
178 try:
--> 179 out = client.execute(parameters=parameters)
180 except Exception:
181 if output_path:
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:498, in PloomberClient.execute(self, parameters)
490 add_debuglater_cells(
491 self._nb,
492 path_to_dump=self._debug_later
493 if isinstance(self._debug_later, (str, Path))
494 else None,
495 )
497 with self:
--> 498 self._execute()
500 if original is not None:
501 # restore original instance
502 InteractiveShell._instance = original
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:621, in PloomberClient._execute(self)
618 if self._progress_bar:
619 iterator.set_description(f"Executing cell: {execution_count}")
--> 621 self.execute_cell(
622 cell,
623 cell_index=index,
624 execution_count=execution_count,
625 store_history=False,
626 )
627 execution_count += 1
629 return self._nb
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:464, in PloomberClient.execute_cell(self, cell, cell_index, execution_count, store_history)
455 # Append to the start
456 self._nb.cells.insert(
457 0,
458 nbformat.v4.new_markdown_cell(
(...)
462 ),
463 )
--> 464 result.raise_error()
466 return output
File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/IPython/core/interactiveshell.py:270, in ExecutionResult.raise_error(self)
268 raise self.error_before_exec
269 if self.error_in_exec is not None:
--> 270 raise self.error_in_exec
[... skipping hidden 1 frame]
File <ipython-input-1-514936153616>:3
1 x = 1
2 y = 0
----> 3 x / y
ZeroDivisionError: division by zero
Command-line equivalent
ploomber-engine nb.ipynb output.ipynb --debug-later
The above command generated an output.dump file which is the serialized traceback:
%%sh
ls *.dump
output.dump
We can use the dltr command (from our debuglater package) to start a debugging session:
dltr output.dump
Tip
By default, only built-in data structures are serialized, for other types, only their string representation is stored. If you want to serialize every data type: pip install 'debuglater[all]'