FFI Memory Contract

NautilusTrader exposes several C-compatible types so that compiled Rust code can be consumed from C-extensions generated by Cython or by other native languages. The most important of these is CVec - a thin wrapper around a Rust Vec<T> that is passed across the FFI boundary by value.

The rules below are strict; violating them results in undefined behaviour (usually a double-free or a memory leak).

CVec represents unique ownership when it comes from Vec<T>. It is intentionally neither Copy nor Clone in Rust. Moving it into a consuming constructor or typed drop helper transfers that ownership, and the same value must not be reused. C can still copy the repr(C) fields, so foreign callers must enforce the same exactly-once rule themselves.

Fail-fast panics at the FFI boundary

Rust panics must never unwind across extern "C" functions. Unwinding into C or Python is undefined behaviour and can corrupt the foreign stack or leave partially-dropped resources behind. To enforce the fail-fast architecture we wrap every exported symbol in crate::ffi::abort_on_panic, which executes the body and calls process::abort() if a panic occurs. The panic message is still logged before the abort, so debugging output is preserved while avoiding undefined behaviour.

When adding new FFI functions, call abort_on_panic(|| { … }) around the implementation (or use a helper that does so) to maintain this guarantee.

CVec lifecycle

Empty CVec values have len == 0 and cap == 0. Their pointer is an opaque sentinel and must never be dereferenced or passed to Vec::from_raw_parts. Rust consumers must use CVec::into_vec, which returns Vec::new() before inspecting the pointer. Borrowing consumers must use CVec::as_slice, which returns &[] before inspecting the pointer when len == 0.

Both methods are unsafe because public FFI metadata cannot prove allocation provenance, alignment, initialization, or exclusive ownership. Rust functions that accept a CVec and call either primitive must also be unsafe extern "C" fn and include a # Safety contract. Rust callers must use an explicit unsafe block even though this does not change the generated C ABI.

Borrowed foreign buffers

Some constructors borrow buffers allocated outside Rust, including Python buffers created with PyMem_Malloc. These allocations must never be reconstructed as Vec<T> because Rust's global allocator did not create them. Borrow them with CVec::as_slice and clone with to_vec() when the Rust object needs owned storage. The caller retains ownership and remains responsible for freeing the original buffer with its matching allocator.

Typed CVec wrappers and Send

CVec is untyped ownership metadata. Do not implement Send for the raw CVec type: it can represent a Vec<T> for any T, including non-Send element types. When PyO3 requires Send for a capsule payload, introduce a narrow wrapper for the concrete payload type and put the unsafe impl Send on that wrapper only after documenting the payload invariant.

For example, DataFFI streaming capsules use DataFfiCVec, a transparent wrapper around CVec whose allocation always comes from Vec<DataFFI>.

Capsules created on the Python side

Several Cython routines allocate temporary C buffers with PyMem_Malloc, wrap them into a CVec, and return the address inside a PyCapsule. Every such capsule is created with a destructor (capsule_destructor or capsule_destructor_deltas) that frees both the buffer and the CVec. Callers must therefore not free the memory manually - doing so would double free.

Capsules created on the Rust side (PyO3 bindings)

When Rust code pushes a heap-allocated value into Python and Python becomes the final owner, it must use PyCapsule::new_with_destructor so that Python knows how to free the allocation once the capsule becomes unreachable. The closure/destructor is responsible for reconstructing the original Box<T> or Vec<T> and letting it drop.

use pyo3::types::PyCapsule;

Python::attach(|py| {
    // Allocate the value on the heap
    let my_data = Box::new(MyStruct::new());
    let ptr = Box::into_raw(my_data);

    // Move it into the capsule and register a destructor that frees the memory
    let capsule = PyCapsule::new_with_destructor(
        py,
        ptr,
        None,
        |ptr, _| {
            // Reconstruct the Box and let it drop, freeing the allocation
            let _ = unsafe { Box::from_raw(ptr) };
        },
    )
    .expect("capsule creation failed");

    // ... pass `capsule` back to Python ...
});

Do not use PyCapsule::new(…, None); that variant registers no destructor and will leak memory unless the recipient manually extracts and frees the pointer.

Rust-owned CVec capsules with explicit drop

Rust-owned CVec batch capsules are an explicit exception to the destructor-owned pattern above. Use this pattern only when the Python/Cython consumer must first extract the batch into Python objects and then release the Rust allocation explicitly.

Requirements for this pattern:

1. Wrap the raw CVec in a type-specific capsule payload, such as DataFfiCVec.
2. Mark that wrapper #[repr(transparent)] over CVec, or use #[repr(C)] with CVec as the first field, before casting capsule pointers back to *mut CVec.
3. Give the capsule a stable, explicit name, such as nautilus.DataFFI.CVec. Do not use the default unnamed capsule for this pattern.
4. Require all consumers to check the same capsule name before reading the pointer.
5. Expose one type-specific drop function, such as drop_cvec_pycapsule.
6. Call that drop function only for capsules created as CVec batches. Never pass a single-value capsule, such as one created by data_to_pycapsule, to a CVec drop function.
7. Validate len <= cap, reject null non-empty pointers, and handle empty CVec values.
8. Reset the stored CVec metadata to CVec::empty() before calling Vec::from_raw_parts, so cleanup paths can call the drop function more than once without double-freeing.
9. Add tests for wrong capsule names, invalid metadata, empty capsules, and repeated drops.

Why there is no generic cvec_drop anymore

Earlier versions of the codebase shipped a generic cvec_drop function that always treated the buffer as Vec<u8>. Using it with any other element type causes a size-mismatch during deallocation and corrupts the allocator's bookkeeping. Because the helper was not referenced anywhere inside the project it has been removed to avoid accidental misuse.

Instead, use the type-specific drop helper for your element type (e.g., vec_drop_book_levels, vec_drop_book_orders). If no helper exists for your type, add one following the pattern in crates/core/src/ffi/cvec.rs.

Box-backed *_API wrappers (owned Rust objects)

When the Rust core needs to hand a complex value (for example an OrderBook, SyntheticInstrument, or TimeEventAccumulator) to foreign code it allocates the value on the heap with Box::new and returns a small repr(C) wrapper whose only field is that Box.

#[repr(C)]
pub struct OrderBook_API(Box<OrderBook>);

#[unsafe(no_mangle)]
pub extern "C" fn orderbook_new(id: InstrumentId, book_type: BookType) -> OrderBook_API {
    OrderBook_API(Box::new(OrderBook::new(id, book_type)))
}

#[unsafe(no_mangle)]
pub extern "C" fn orderbook_drop(book: OrderBook_API) {
    drop(book); // frees the heap allocation
}

Memory-safety requirements are therefore:

1. Every constructor (*_new) must have a matching *_drop exported next to it.

2. Validate parameters before heap allocation to fail fast and avoid allocating invalid objects.

3. The Python/Cython binding must guarantee that *_drop is invoked exactly once. Two approaches exist:

   • Preferred for new code: Wrap the pointer in a PyCapsule created with PyCapsule::new_with_destructor, passing a destructor that calls the drop helper.

   • Legacy pattern (v1 Cython modules only): Call the helper explicitly in __del__/__dealloc__ on the Python side:

   cdef class OrderBook:
       cdef OrderBook_API _mem
   
       def __cinit__(self, ...):
           self._mem = orderbook_new(...)
   
       def __del__(self):
           if self._mem._0 != NULL:
               orderbook_drop(self._mem)

Whichever style is used, remember: forgetting the drop call leaks the entire structure, while calling it twice will double-free and crash.

New FFI code must use PyCapsule with destructors and follow this template before it can be merged.