Debug later#

New in version 0.0.19.

ploomber-engine uses our debuglater package to serialize the error traceback so you can start a debugging session after your notebook crashes.

So, for example, if you’re running notebooks in production or remote servers, you can debug after they crash. Likewise, you can use the generated file to debug on a different machine (assuming the environment is the same) without having access to the source code.

Example#

Install requirements:

%pip install ploomber-engine --quiet
Note: you may need to restart the kernel to use updated packages.

Download sample notebook:

%%sh
curl https://raw.githubusercontent.com/ploomber/ploomber-engine/main/tests/assets/debuglater.ipynb --output debuglater-demo.ipynb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2056  100  2056    0     0   7766      0 --:--:-- --:--:-- --:--:--  7787

Run the notebook with debug_later=True option (note that this notebook crashes on purpose):

from ploomber_engine import execute_notebook

execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)
Hide code cell output
  0%|                                                     | 0/2 [00:00<?, ?it/s]
Executing cell: 1:   0%|                                  | 0/2 [00:00<?, ?it/s]
Executing cell: 2:   0%|                                  | 0/2 [00:00<?, ?it/s]
Executing cell: 2:  50%|█████████████             | 1/2 [00:00<00:00,  2.01it/s]

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
File /tmp/ipykernel_3304/3245917335.py:3
      1 from ploomber_engine import execute_notebook
----> 3 execute_notebook("debuglater-demo.ipynb", "output.ipynb", debug_later=True)

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/ploomber_core/telemetry/telemetry.py:644, in Telemetry.log_call.<locals>._log_call.<locals>.wrapper(*args, **kwargs)
    642             result = func(_payload, *args, **kwargs)
    643     else:
--> 644         result = func(*args, **kwargs)
    645 except Exception as e:
    646     metadata_error = {
    647         # can we log None to posthog?
    648         "type": getattr(e, "type_", None),
   (...)
    651         **_payload,
    652     }

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/execute.py:179, in execute_notebook(input_path, output_path, parameters, log_output, profile_runtime, profile_memory, progress_bar, debug_later, verbose, remove_tagged_cells, cwd, save_profiling_data)
    169 client = INIT_FUNCTION(
    170     input_path,
    171     display_stdout=log_output,
   (...)
    175     cwd=cwd,
    176 )
    178 try:
--> 179     out = client.execute(parameters=parameters)
    180 except Exception:
    181     if output_path:

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:498, in PloomberClient.execute(self, parameters)
    490     add_debuglater_cells(
    491         self._nb,
    492         path_to_dump=self._debug_later
    493         if isinstance(self._debug_later, (str, Path))
    494         else None,
    495     )
    497 with self:
--> 498     self._execute()
    500 if original is not None:
    501     # restore original instance
    502     InteractiveShell._instance = original

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:621, in PloomberClient._execute(self)
    618             if self._progress_bar:
    619                 iterator.set_description(f"Executing cell: {execution_count}")
--> 621             self.execute_cell(
    622                 cell,
    623                 cell_index=index,
    624                 execution_count=execution_count,
    625                 store_history=False,
    626             )
    627             execution_count += 1
    629 return self._nb

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/checkouts/latest/src/ploomber_engine/ipython.py:464, in PloomberClient.execute_cell(self, cell, cell_index, execution_count, store_history)
    455     # Append to the start
    456     self._nb.cells.insert(
    457         0,
    458         nbformat.v4.new_markdown_cell(
   (...)
    462         ),
    463     )
--> 464     result.raise_error()
    466 return output

File ~/checkouts/readthedocs.org/user_builds/ploomber-engine/conda/latest/lib/python3.10/site-packages/IPython/core/interactiveshell.py:270, in ExecutionResult.raise_error(self)
    268     raise self.error_before_exec
    269 if self.error_in_exec is not None:
--> 270     raise self.error_in_exec

    [... skipping hidden 1 frame]

File <ipython-input-1-514936153616>:3
      1 x = 1
      2 y = 0
----> 3 x / y

ZeroDivisionError: division by zero

The above command generated an output.dump file which is the serialized traceback:

%%sh
ls *.dump
output.dump

We can use the dltr command (from our debuglater package) to start a debugging session:

dltr output.dump

Tip

By default, only built-in data structures are serialized, for other types, only their string representation is stored. If you want to serialize every data type: pip install 'debuglater[all]'