Debug later#

New in version 0.0.19.

ploomber-engine uses our debuglater package to serialize the error traceback so you can start a debugging session after your notebook crashes.

So, for example, if you’re running notebooks in production or remote servers, you can debug after they crash. Likewise, you can use the generated file to debug on a different machine (assuming the environment is the same) without having access to the source code.

Example#

Install requirements:

%pip install ploomber-engine --quiet
Note: you may need to restart the kernel to use updated packages.

Download sample notebook:

%%sh
curl https://raw.githubusercontent.com/ploomber/ploomber-engine/main/tests/assets/debuglater.ipynb --output debuglater-demo.ipynb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2056  100  2056    0     0  11237      0 --:--:-- --:--:-- --:--:-- 11296

Run the notebook with debug_later=True option (note that this notebook crashes on purpose):

from ploomber_engine import execute_notebook

execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)
Hide code cell output
  0%|                                                     | 0/2 [00:00<?, ?it/s]
Executing cell: 1:   0%|                                  | 0/2 [00:00<?, ?it/s]
Executing cell: 2:   0%|                                  | 0/2 [00:00<?, ?it/s]
Executing cell: 2:  50%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ             | 1/2 [00:00<00:00,  2.01it/s]

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
File /tmp/ipykernel_2608/3245917335.py:3
      1 from ploomber_engine import execute_notebook
----> 3 execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/ploomber_core/telemetry/telemetry.py:745, in Telemetry.log_call.<locals>._log_call.<locals>.wrapper(*args, **kwargs)
    743             result = func(_payload, *args, **kwargs)
    744     else:
--> 745         result = func(*args, **kwargs)
    746 except Exception as e:
    747     metadata_error = {
    748         # can we log None to posthog?
    749         "type": getattr(e, "type_", None),
   (...)
    752         **_payload,
    753     }

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/execute.py:168, in execute_notebook(input_path, output_path, parameters, log_output, profile_runtime, profile_memory, progress_bar, debug_later, verbose, remove_tagged_cells, cwd, save_profiling_data)
    158 client = INIT_FUNCTION(
    159     input_path,
    160     display_stdout=log_output,
   (...)
    164     cwd=cwd,
    165 )
    167 try:
--> 168     out = client.execute(parameters=parameters)
    169 except Exception:
    170     if output_path:

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:467, in PloomberClient.execute(self, parameters)
    459     add_debuglater_cells(
    460         self._nb,
    461         path_to_dump=self._debug_later
    462         if isinstance(self._debug_later, (str, Path))
    463         else None,
    464     )
    466 with self:
--> 467     self._execute()
    469 if original is not None:
    470     # restore original instance
    471     InteractiveShell._instance = original

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:590, in PloomberClient._execute(self)
    587             if self._progress_bar:
    588                 iterator.set_description(f"Executing cell: {execution_count}")
--> 590             self.execute_cell(
    591                 cell,
    592                 cell_index=index,
    593                 execution_count=execution_count,
    594                 store_history=False,
    595             )
    596             execution_count += 1
    598 return self._nb

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:433, in PloomberClient.execute_cell(self, cell, cell_index, execution_count, store_history)
    424     # Append to the start
    425     self._nb.cells.insert(
    426         0,
    427         nbformat.v4.new_markdown_cell(
   (...)
    431         ),
    432     )
--> 433     result.raise_error()
    435 return output

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/IPython/core/interactiveshell.py:266, in ExecutionResult.raise_error(self)
    264     raise self.error_before_exec
    265 if self.error_in_exec is not None:
--> 266     raise self.error_in_exec

    [... skipping hidden 1 frame]

File <ipython-input-1-514936153616>:3
      1 x = 1
      2 y = 0
----> 3 x / y

ZeroDivisionError: division by zero

The above command generated an output.dump file which is the serialized traceback:

%%sh
ls *.dump
output.dump

We can use the dltr command (from our debuglater package) to start a debugging session:

dltr output.dump

Tip

By default, only built-in data structures are serialized, for other types, only their string representation is stored. If you want to serialize every data type: pip install 'debuglater[all]'