# NautilusTrader Documentation Source: https://nautilustrader.io/docs/latest/ # NautilusTrader Documentation NautilusTrader is an open-source, production-grade, Rust-native trading engine infrastructure for multi-asset, multi-venue trading systems. Its core spans research, deterministic simulation, portfolio and risk modeling, and live execution under one event-driven architecture. Python serves as the control plane for strategy logic, configuration, and orchestration, while entire systems can also be built in Rust. NautilusTrader is engineered to minimize the gap between research and production. The same strategy and execution-algorithm code runs in both backtests and live systems, reducing deployment risk and unexpected behavior. Modular adapters keep venue integration neutral. Current integrations span crypto venues, traditional markets, betting exchanges, and data providers, with custom adapters available for REST, WebSocket, and provider-specific APIs. ## Where to start - [Architecture](concepts/architecture): event-driven design, core components, and crash-only design. - [Backtesting](concepts/backtesting): simulation APIs, data-loading contracts, and post-run analysis. - [Live trading](concepts/live): command outcomes, execution reconciliation, and startup recovery. }> Install NautilusTrader and run your first backtests. }> Architecture, data, execution, backtesting, live trading, and core domain model guides. }> Goal-oriented recipes for data workflows, backtesting, live trading, and common operational tasks. }> Runnable walkthroughs for backtesting, data workflows, strategy patterns, options, and Rust. }> Supported venue and data-provider adapters, status, and setup guides. }> Rust and Python internals, testing standards, adapter specs, and release process. # Accounting Source: https://nautilustrader.io/docs/latest/concepts/accounting/ The accounting subsystem tracks balances, margins, and PnL for every account the platform interacts with. This guide covers the data model, the query API that strategies use, and the conventions adapter authors must follow to stay consistent across venues. It applies equally to backtest and live trading. For backtest-specific configuration (starting balances, margin-model selection per venue), see [Backtesting](backtesting/). ## Account types When you attach a venue to the engine for either live trading or a backtest, you pick one of three accounting modes via `account_type`: Cash, Margin, or Betting. A fourth account type, Wallet, models on-chain wallet state. The Blockchain adapter selects it; its execution client supports locally signed Uniswap V3 market swaps but is not production-ready. | Account type | Typical use case | What the engine locks | | ------------ | ----------------------------------------------- | ------------------------------------------------------------------------- | | Cash | Spot trading (e.g., BTC/USDT, stocks) | Notional for pending buy orders; quantity for pending sell orders. | | Margin | Derivatives or any product that allows leverage | Initial margin for each order plus maintenance margin for open positions. | | Betting | Sports betting, bookmaking | Stake required by the venue; no leverage. | | Wallet | Blockchain wallets (DeFi) | Amounts reserved locally for pending orders; no leverage or borrowing. | ### Cash accounts Cash accounts **settle trades in full**; there is no leverage and therefore no concept of margin. Locked balances reflect the value reserved for pending orders: the notional value of each pending buy and the quantity each pending sell would deliver. ### Margin accounts Margin accounts support instruments that require collateral, such as futures or leveraged crypto perps. They track account balances, reserve margin for open orders and positions, and apply a configurable leverage per instrument. Margin is tracked in two scopes; see [Margin scopes](#margin-scopes) below. **Terms**: - **Leverage**: amplifies exposure relative to account equity. Higher leverage raises both potential returns and risk. - **Initial margin**: collateral reserved when an order is submitted. - **Maintenance margin**: minimum collateral required to keep an open position. - **Locked balance**: funds reserved as collateral, not available for new orders. :::note Reduce-only orders do not contribute to `balance_locked` on cash accounts and do not add to initial margin on margin accounts, since they can only decrease exposure. Wallet orders still reserve the input asset because the on-chain transaction spends that asset even when the order reduces a position. ::: ### Betting accounts Betting accounts are specialized for venues where you stake an amount to win or lose a fixed payout (prediction markets, sports books). The engine locks only the stake required by the venue; leverage and margin do not apply. ### Wallet accounts Wallet accounts represent blockchain wallets: unleveraged, multi-currency holdings of native and ERC-20 token balances with no margin and no borrowing. For reported states, `total` is the observed on-chain balance; `locked` tracks local pending-order reservations, and `free = total - locked`. Account state events **contribute totals only**: the account ignores incoming `locked` and `free` values, retains its local reservations, and rederives `free`. It rebuilds transient reservations from submitted and open orders during live startup. While an amendment is pending, the account reserves the full observed balance of the debit currency because the pending event does not carry the requested terms. If the reserved amount exceeds the latest observed total, `locked` is capped at `total` and `free` remains zero until the balance or reservation changes. A balance with a negative total is rejected rather than applied. ERC-20 allowances are spender authorizations and are never represented as balances or locked funds. ## Balance model An `AccountBalance` holds three values in the same currency: - `total`: the venue-reported total balance figure (wallet, net liquidation, or margin balance, depending on the venue). - `locked`: amount reserved against open orders and positions. - `free`: amount available for new orders (`total - locked`). The **balance invariant** `total == locked + free` must always hold at currency precision. The Python `AccountBalance(total, locked, free)` constructor requires all three fields up front. Adapter code written in Rust has two additional derived constructors that enforce the invariant centrally; prefer them over `AccountBalance::new` whenever the venue reports only two of the three values: | Rust constructor | When to use | | --------------------------------------- | ------------------------------------------------------------------------ | | `AccountBalance::from_total_and_locked` | Venue reports total and locked; `free` is derived from the two. | | `AccountBalance::from_total_and_free` | Venue reports total and free; `locked` is derived from the two. | | `AccountBalance::new` | All three values are already known and consistent (tests, pass-through). | Each derived constructor clamps the venue-reported field into `[0, total]` when `total >= 0`, so transient overshoots from venue rounding never leave the account in a broken state. ## Currency and valuation contracts Accounting values retain their source currency until an explicit conversion succeeds. This prevents a valid number from being labeled with the wrong currency or an unavailable value from being treated as zero. | Value | Currency contract | | ---------------------------- | -------------------------------------------------------------- | | Instrument cost currency | Base for inverse, settlement for quanto, and quote otherwise. | | Position PnL | Instrument cost currency captured when the position opens. | | Calculated locks and margins | Each calculated amount's currency, converted independently. | | Portfolio aggregates | Native buckets, or the account base after conversion succeeds. | Aggregations combine only compatible `Money` values. An account without a base currency keeps separate native currency buckets. A single-instrument realized PnL query returns unavailable instead of combining mixed currencies. The accounting and valuation paths also follow these rules: - Invalid or unrepresentable notional, PnL, fee, locked-balance, and margin results produce an error, unavailable value, or unpriced state. They do not substitute zero. A failed realized PnL recalculation also clears any earlier cached result. - `equity()` counts a credited non-inverse base asset once for a multi-currency cash account without a base currency. `mark_values()` remains a gross position-value query and includes that asset. - MTM snapshots distinguish carried stale inputs from positions that have never had complete valuation data. Stale-price metadata covers only open instrument and position-side pairs. See [Portfolio](portfolio.md#equity-and-mark-to-market) for equity formulas, price and xrate selection, snapshot metadata, and missing-price query scope. ## Margin scopes A `MarginBalance` has four fields: `initial`, `maintenance`, `currency`, and an `Optional[InstrumentId]` that selects one of two scopes. ### Per-instrument scope `MarginBalance.instrument_id` is set to a concrete instrument. Use this for: - Isolated margin (per-position collateral). - Backtest or calculated margin, where the `AccountsManager` derives margin locally from open orders and positions per instrument. ### Account-wide scope `MarginBalance.instrument_id` is `None`. The entry is keyed by its `currency` (the collateral currency). Use this for cross-margin venues that report a single aggregate per collateral currency. A venue may emit one account-wide entry (single-collateral cross margin) or several (one per collateral coin). Both scopes coexist on the same `MarginAccount` in separate internal stores. An `AccountState` event may carry entries in either or both scopes, and `MarginAccount.apply()` routes each entry to the correct store based on whether `instrument_id` is set. :::warning `MarginAccount.apply()` **replaces** both stores from the incoming event. It does not merge with prior state, and an event carrying neither balances nor margins leaves the prior stores in place. Adapters that emit partial snapshots must include every live margin entry on each update or those entries will be dropped until the next full snapshot. Balances the event carries replace the stored entry for their currency; currencies the event omits are retained. ::: ## Strategy query API Use the query that matches the venue's reporting shape. If a venue reports per-instrument margins, ask by `InstrumentId`. If it reports account-wide margins, ask by `Currency`. | Scope | Queries | | -------------- | ---------------------------------------------------------------------------- | | Per-instrument | `margin`, `initial_margin`, and `maintenance_margin` | | Account-wide | `account_margin`, `account_initial_margin`, and `account_maintenance_margin` | | Both scopes | `total_initial_margin` and `total_maintenance_margin` | The signatures below describe the Python bindings. Point queries return `None` when the entry is absent; total queries always return a `Money` (zero for the currency if nothing matches). ### Per-instrument queries (`MarginAccount`) - `margin(instrument_id) -> MarginBalance | None` - `initial_margin(instrument_id) -> Money | None` - `maintenance_margin(instrument_id) -> Money | None` - `margins() -> dict[InstrumentId, MarginBalance]` (all per-instrument entries) - `initial_margins() -> dict[InstrumentId, Money]` - `maintenance_margins() -> dict[InstrumentId, Money]` These methods only see the per-instrument store. On a cross-margin venue they return empty dicts or `None`. Use the account-wide queries below. ### Account-wide queries (`MarginAccount`) - `account_margin(currency) -> MarginBalance | None` - `account_initial_margin(currency) -> Money | None` - `account_maintenance_margin(currency) -> Money | None` - `account_margins() -> dict[Currency, MarginBalance]` (all account-wide entries) - `account_initial_margins() -> dict[Currency, Money]` - `account_maintenance_margins() -> dict[Currency, Money]` ### Totals (`MarginAccount`) These sum across per-instrument and account-wide entries for a given currency: - `total_initial_margin(currency) -> Money` - `total_maintenance_margin(currency) -> Money` Useful when a strategy trades on a venue where both scopes may appear (for example, isolated positions alongside cross-margin collateral). ### Python binding boundary This query surface does not expose the internal Rust mutation methods `update_margin`, `clear_margin`, `clear_account_margin`, `clear_initial_margin`, `clear_maintenance_margin`, or `set_margin_model`. Python does expose other mutation methods, including `update_initial_margin`, `update_maintenance_margin`, `set_default_leverage`, and `set_leverage`. ### Portfolio-level queries Margin queries: - `portfolio.instrument_initial_margins(venue=..., account_id=...) -> dict[InstrumentId, Money] | None` - `portfolio.instrument_maintenance_margins(venue=..., account_id=...) -> dict[InstrumentId, Money] | None` When a margin account resolves, these return the same per-instrument money views as `MarginAccount.initial_margins` and `MarginAccount.maintenance_margins`; otherwise, they return `None`. For account-wide data on cross-margin venues, query the account directly via `portfolio.account(venue=venue).account_initial_margin(ccy)`. The returned account is a **detached snapshot** and cannot mutate Portfolio state. PnL, exposure, mark-to-market, and equity queries all accept `venue` and an optional `account_id` to scope multi-account venues: - `portfolio.unrealized_pnls(venue=..., account_id=..., target_currency=...) -> dict[Currency, Money]` - `portfolio.realized_pnls(venue=..., account_id=..., target_currency=...) -> dict[Currency, Money]` - `portfolio.total_pnls(venue=..., account_id=..., target_currency=...) -> dict[Currency, Money]` - `portfolio.net_exposures(venue=..., account_id=..., target_currency=...) -> dict[Currency, Money] | None` - `portfolio.mark_values(venue=..., account_id=...) -> dict[Currency, Money]` - `portfolio.equity(venue=..., account_id=...) -> dict[Currency, Money]` - `portfolio.missing_price_instruments(venue, account_id=...) -> list[InstrumentId]` If both scope arguments are present, they must identify the same account. A missing price, failed target-currency conversion, or arithmetic overflow invalidates the whole affected collection: a query never returns partial or mixed-currency totals. See the [Portfolio guide](portfolio.md#equity-and-mark-to-market) for the equity formula, price fallback chain, base-currency conversion behavior, and the warn-once missing-price tracker. ### Worked examples Single-collateral cross margin (one account-wide entry): ```python usdc_margin = margin_account.account_initial_margin(USDC) usdc_total = margin_account.total_initial_margin(USDC) ``` Per-coin cross margin (one entry per collateral currency): ```python for ccy, margin_balance in margin_account.account_margins().items(): print(ccy, margin_balance.initial, margin_balance.maintenance) ``` ## Margin models NautilusTrader provides flexible margin calculation models for the calculated path (backtests, and live strategies running with `calculate_account_state=True` for reconciliation). Reported margins from a venue flow straight into the account's `margins` or `account_margins` stores without going through a model. ### Overview Different venues treat leverage differently: - **Traditional brokers** (e.g., Interactive Brokers): fixed margin percentages regardless of leverage. - **Crypto exchanges** (Binance, others): leverage may reduce margin requirements. Both built-in models compute margin as a percentage of notional using the instrument's `margin_init` and `margin_maint` fields. They differ only in whether leverage reduces the reservation. For venues with true per-contract fixed margin (CME / ICE), set `instrument.margin_init` and `margin_maint` so the percentage recovers the desired dollar amount. ### HEDGING-mode netting Under `OmsType.HEDGING`, the first fill for a new position ID opens a `Position`; later fills can update that position. An account can therefore hold many open sub-positions for the same instrument. The accounts manager nets those sub-positions onto a hypothetical NETTING position in `ts_opened` order, then runs the margin model once on the resulting net signed quantity and average open price. The replay follows the same rules as `Position.apply`: same-side fills produce a quantity-weighted average open price, opposite-side fills partial-close at the existing average, and a fill that crosses zero makes the residual take the flipping fill's price. Sub-positions sharing a `ts_opened` fold in `(ts_opened, position_id)` order so the result does not depend on cache iteration order. HEDGING and NETTING accounts compute the same maintenance margin when the folded net quantity and average open price match under the same margin model and leverage; the requirement scales with **net economic exposure**. ### Available models #### `StandardMarginModel` Uses **fixed percentages without leverage division**, matching traditional broker behavior. ```python # Fixed percentages - leverage ignored margin = notional * instrument.margin_init ``` - Initial margin: `notional_value * instrument.margin_init` - Maintenance margin: `notional_value * instrument.margin_maint` **Use cases**: traditional brokers (Interactive Brokers), forex brokers with fixed margin requirements. #### `LeveragedMarginModel` Divides margin requirements **by leverage**. ```python # Leverage reduces margin requirements adjusted_notional = notional / leverage margin = adjusted_notional * instrument.margin_init ``` - Initial margin: `(notional_value / leverage) * instrument.margin_init` - Maintenance margin: `(notional_value / leverage) * instrument.margin_maint` **Use cases**: crypto exchanges that reduce margin with leverage, venues where leverage affects margin requirements. ### Default behavior `MarginAccount` uses `LeveragedMarginModel` by default. Backtests select `StandardMarginModel` by passing it directly to `BacktestVenueConfig.margin_model`. ### Worked example: EUR/USD - **Instrument**: EUR/USD - **Quantity**: 100,000 EUR - **Price**: 1.10000 - **Notional**: $110,000 - **Leverage**: 50x - **`instrument.margin_init`**: 3% | Model | Calculation | Result | Percentage | | --------- | ---------------------- | ------ | ---------- | | Standard | $110,000 × 0.03 | $3,300 | 3.00% | | Leveraged | ($110,000 ÷ 50) × 0.03 | $66 | 0.06% | On a $1,000 account: the standard model blocks the trade; the leveraged model allows it. ### Python model selection Pass `StandardMarginModel()` or `LeveragedMarginModel()` directly to the backtest venue. The Python binding does not accept custom margin model subclasses or a `MarginModelConfig` wrapper. See [Backtesting](backtesting/accounts-and-margin.md#margin-models). ## Adapter convention Live adapters translate venue responses into `AccountBalance` and `MarginBalance` instances. The convention that adapter authors must follow: ### Building `AccountBalance` Prefer the derived constructors so that clamping and the `total == locked + free` invariant are enforced centrally. Hand-computing three fields and passing them to `AccountBalance::new` is only appropriate for pass-through paths where all three values are already authoritative (e.g., tests). ### Building `MarginBalance` Pick the scope that matches what the venue reports: | Venue reports | Scope | Emit with | | ---------------------------------------------- | -------------- | ---------------------------------------------------------- | | Per-instrument (isolated positions) | Per-instrument | `MarginBalance::new(initial, maint, Some(id))` | | Single aggregate per collateral (cross margin) | Account-wide | `MarginBalance::new(initial, maint, None)` | | Multiple aggregates, one per collateral | Account-wide | One `MarginBalance` per currency with `instrument_id=None` | :::info Synthetic `ACCOUNT.{VENUE}` or `ACCOUNT-{COIN}.{VENUE}` `InstrumentId` placeholders are not used. Account-wide entries carry `instrument_id=None` and are keyed by `currency`. ::: ## Related guides - [Backtesting](backtesting/): starting balances, margin models, and backtest-specific account setup. - [Portfolio](portfolio.md): portfolio-level PnL, exposures, and currency conversion. - [Positions](positions.md): position lifecycle, aggregation, and PnL. - [Adapters](adapters.md): requirements and best practices for adapter authors. - [Blockchain](../integrations/blockchain.md): the adapter that selects wallet accounts, and its execution status. # Actors Source: https://nautilustrader.io/docs/latest/concepts/actors/ A **data actor** receives requested and subscribed data, handles system events, and manages component state. In Python, extend the `DataActor` class; in Rust, implement the `DataActor` trait. A strategy adds order-management capabilities. **Capabilities**: - Market and custom data subscriptions and requests. - Custom data and signal publishing. - Event, timer, and alert handling. - Cache access. - Structured logging. ## Basic Python example Actors support configuration through a pattern similar to strategies. Declare custom fields as keyword-only arguments and accept `**_kwargs` so the base fields (`actor_id` and the log settings) pass through to `DataActorConfig.__new__`, which reads them from the same call. A positional argument would be matched against `actor_id` instead, raising a `TypeError`. ```python from nautilus_trader.common import DataActor from nautilus_trader.config import DataActorConfig from nautilus_trader.model import Bar from nautilus_trader.model import BarType class MyActorConfig(DataActorConfig): def __init__(self, *, bar_type: BarType, **_kwargs) -> None: super().__init__() self.bar_type = bar_type class MyActor(DataActor): def __init__(self, config: MyActorConfig) -> None: super().__init__(config) # Keep runtime state on the actor self.count_of_processed_bars: int = 0 def on_start(self) -> None: # Subscribe to bars matching the configured bar type self.subscribe_bars(self.config.bar_type) def on_bar(self, bar: Bar) -> None: self.count_of_processed_bars += 1 ``` ## Actor configuration and IDs Data actors can receive a `DataActorConfig` subclass. The base config accepts an optional `actor_id`. If supplied, the actor registers with that ID; otherwise a Python actor registers under its class name. Give each instance an explicit `actor_id` when running more than one instance of the same actor, because a duplicate ID is rejected at registration (a `RuntimeError` in Python). Treat configuration as construction data for the actor. Read user-supplied settings through `self.config`, and keep runtime state on the actor itself. :::info Rust implementation Rust actors store runtime identity and state in `DataActorCore`. Read the runtime ID through `actor_id()` rather than expecting a generated ID to be written back into `DataActorConfig`. A Rust actor without a configured `actor_id` registers as `DataActor` whatever its type, so give each Rust actor an explicit `actor_id`. Rust authors implement `DataActor` and use the facade methods on `self`. `DataActorNative` is native-only access for runtime wiring and borrowed core state. Import it only for same-binary performance paths or internal runtime wiring. ::: ## Lifecycle Actors move through the main stable states shown below: ```mermaid stateDiagram-v2 [*] --> READY : register() READY --> RUNNING : start() RUNNING --> STOPPED : stop() STOPPED --> RUNNING : resume() RUNNING --> DEGRADED : degrade() DEGRADED --> RUNNING : resume() STOPPED --> READY : reset() RUNNING --> FAULTED : fault() STOPPED --> DISPOSED : dispose() ``` Main flow only: transitional states and less common valid edges are omitted. For actions with a lifecycle handler, the actor reaches the destination state only after that handler succeeds. Override these methods to hook into lifecycle events: | Method | When called | | -------------- | ---------------------------------------------------------------------------------------------------------------------- | | `on_start()` | Actor is starting; subscribe to data here. | | `on_stop()` | Actor is stopping; clean up actor-owned resources. | | `on_resume()` | Actor is resuming after it stopped or degraded. | | `on_reset()` | Actor is resetting, including between backtest runs; retained data subscriptions are released after the hook succeeds. | | `on_degrade()` | Actor is entering a degraded state and may provide only partial functionality. | | `on_fault()` | Actor is entering the faulted state after it encounters a fault. | | `on_dispose()` | Actor is being disposed and must release its remaining resources. | ## Timers and alerts Actors have access to a clock for scheduling: ```python from datetime import timedelta from nautilus_trader.common import TimeEvent def on_start(self) -> None: self._schedule_clock_events() def on_resume(self) -> None: self._schedule_clock_events() def on_stop(self) -> None: self._cancel_clock_events() def on_degrade(self) -> None: self._cancel_clock_events() def _cancel_clock_events(self) -> None: self.clock.cancel_timer("my_actor.timer") self.clock.cancel_timer("my_actor.alert") def _schedule_clock_events(self) -> None: # Set a recurring timer with a callback that fires every 5 seconds self.clock.set_timer( "my_actor.timer", timedelta(seconds=5), callback=self._on_timer, ) # Set a one-time alert with a callback self.clock.set_time_alert( "my_actor.alert", self.clock.utc_now() + timedelta(minutes=1), callback=self._on_alert, ) def _on_timer(self, event: TimeEvent) -> None: self.log.info("Timer fired!") def _on_alert(self, event: TimeEvent) -> None: self.log.info("Alert triggered!") ``` Pass a `callback` to direct `TimeEvent` objects to your own method. Without one, the actor runtime connects the clock's registered default handler to `on_time_event()`. When components share a clock: - Use **explicit callbacks** to route events to the intended component. - Use **unique timer names** within the clock's namespace. Registering the same name replaces the existing timer. ## System access Actors have access to core system components: | API | Description | | ------------------------------------------- | ---------------------------------------------------- | | `self.cache` | Shared state for instruments, orders, and positions. | | `self.clock` | Current time and timer or alert scheduling. | | `self.log` | Structured logging. | | `publish_data()` / `subscribe_data()` | Structured custom data messaging. | | `publish_signal()` / `subscribe_signal()` | Lightweight alerts and notifications. | | `publish_message()` | Publish a Python object on an application topic. | | `subscribe_topic()` / `unsubscribe_topic()` | Manage Python object callbacks. | | `subscribe_queue_state()` | Live runner queue pressure state changes. | | `subscribe_socket_state()` | Live socket transport state changes. | | `reconnect_socket()` | Request recovery of one live socket endpoint. | | `unsubscribe_queue_state()` | Stop receiving runner queue pressure state changes. | | `unsubscribe_socket_state()` | Stop receiving socket transport state changes. | | `on_queue_state()` | Handle a runner queue pressure state change. | | `on_socket_state()` | Handle a socket transport state change. | The Python `DataActor` and `Strategy` APIs do not expose `self.msgbus`. Use custom data for structured payloads, signals for lightweight values, or [topic messaging](message_bus.md#python-topic-messaging) for arbitrary in-process Python objects. ### Queue pressure state Actors can subscribe to runner queue pressure state changes: ```rust tab="Rust" use nautilus_common::{ actor::DataActor, messages::system::QueueStateChanged, }; impl DataActor for MyActor { fn on_start(&mut self) -> anyhow::Result<()> { self.subscribe_queue_state(None, Some(50)); Ok(()) } fn on_stop(&mut self) -> anyhow::Result<()> { self.unsubscribe_queue_state(None); Ok(()) } fn on_queue_state(&mut self, event: &QueueStateChanged) -> anyhow::Result<()> { log::warn!( "Queue {:?} changed {:?} to {:?} at depth {}", event.channel, event.condition, event.state, event.queue_depth, ); Ok(()) } } ``` ```python tab="Python" from nautilus_trader.common import QueueStateChanged def on_start(self) -> None: self.subscribe_queue_state(priority=50) def on_stop(self) -> None: self.unsubscribe_queue_state() def on_queue_state(self, event: QueueStateChanged) -> None: self.log.warning( f"Queue {event.channel} changed {event.condition} to {event.state} " f"at depth {event.queue_depth}", ) ``` #### Filters and delivery The optional priority controls delivery order among matching subscribers. **Higher values run first.** Subscribing again does not change an existing priority; unsubscribe before subscribing with a new priority. Pass `channel` to select one runner channel, for example `self.subscribe_queue_state(channel=SystemChannel.DATA_EVENTS)` in Python. Import `SystemChannel` from `nautilus_trader.common`. - Omitting `channel` subscribes to all monitored channels. - To remove a subscription, pass the **same channel** to `unsubscribe_queue_state`. Omitting it removes only the unfiltered subscription. - Overlapping subscriptions each invoke the callback for a matching event. Runner queues are shared across clients, so they have no client filter. #### Queue event contents `QueueStateChanged` includes the trader ID, runner channel, queue condition, condition state, queue depth, mean dispatch time, event ID, and timestamps. Delivery uses the typed in-process message bus and has no external wire representation. See [Queue pressure monitoring](live.md#queue-pressure-monitoring) for the trigger and clear semantics. ### Socket transport state Actors can subscribe to socket state changes from live adapters that report them: ```rust tab="Rust" use nautilus_common::{ actor::DataActor, messages::system::SocketStateChanged, }; impl DataActor for MyActor { fn on_start(&mut self) -> anyhow::Result<()> { self.subscribe_socket_state(None, None, Some(50)); Ok(()) } fn on_stop(&mut self) -> anyhow::Result<()> { self.unsubscribe_socket_state(None, None); Ok(()) } fn on_socket_state(&mut self, event: &SocketStateChanged) -> anyhow::Result<()> { log::info!( "Socket {} for {} changed to {:?}", event.endpoint, event.client_id, event.state, ); Ok(()) } } ``` ```python tab="Python" from nautilus_trader.common import SocketStateChanged def on_start(self) -> None: self.subscribe_socket_state(priority=50) def on_stop(self) -> None: self.unsubscribe_socket_state() def on_socket_state(self, event: SocketStateChanged) -> None: self.log.info( f"Socket {event.endpoint} for {event.client_id} changed to {event.state}", ) ``` #### Filters and delivery The optional priority controls delivery order among matching subscribers. **Higher values run first.** Subscribing again does not change an existing priority; unsubscribe before subscribing with a new priority. Pass `client_id`, `endpoint`, or both to filter socket events. In Python, `self.subscribe_socket_state(client_id=ClientId("BINANCE"), endpoint="binance-futures-market-streams")` selects one transport. Import `ClientId` from `nautilus_trader.model`. - Each omitted filter matches all values of that field. - Supplied values match literally, including dots and wildcard characters. - Pass the **same filters** to `unsubscribe_socket_state` to remove that subscription. Omitting both removes only the unfiltered subscription. - Overlapping subscriptions each invoke the callback for a matching event. #### Socket event contents `SocketStateChanged` includes the trader ID, client ID, optional venue, stable endpoint label, transport state, event ID, and timestamps. The endpoint is a non-secret logical label, not a raw connection URL. `SocketState.DISCONNECTED` reports the loss of an active transport. :::note `SocketState.CONNECTED` reports **transport availability**, not authentication, subscription replay, or adapter readiness. ::: Delivery uses the typed in-process message bus and has no external wire representation. See [Socket transport state](live.md#socket-transport-state) for supported adapters and the precise connection edge semantics. ### Reconnect a socket endpoint Live actors and strategies can request recovery of one endpoint without restarting its data or execution client. Pass the `client_id` and the endpoint label reported by `SocketStateChanged`: ```rust tab="Rust" use nautilus_common::actor::DataActor; use nautilus_model::identifiers::ClientId; impl MyActor { fn recover_market_socket(&self) -> anyhow::Result<()> { self.reconnect_socket( ClientId::from("POLYMARKET"), "polymarket-market-streams", )?; Ok(()) } } ``` ```python tab="Python" from nautilus_trader.model import ClientId def recover_market_socket(self) -> None: self.reconnect_socket( client_id=ClientId("POLYMARKET"), endpoint="polymarket-market-streams", ) ``` #### Observe recovery :::note This API is **fire-and-observe**. A successful return means the command passed local validation and was queued. It does not acknowledge that the kernel accepted the request or that recovery completed. ::: Subscribe with `subscribe_socket_state` using the same `client_id` and `endpoint` filters, then inspect the `SocketStateChanged` events. An accepted request reports: 1. `SocketState.DISCONNECTED` as the transport enters reconnect mode. 1. `SocketState.CONNECTED` after transport recovery. #### Request failures - **Synchronous failures**: Invalid endpoint labels and unavailable or closed runner channels fail synchronously. Endpoint labels accept only ASCII letters, digits, `.`, `-`, and `_`; pass a logical label rather than a raw URL. - **Kernel rejections**: The kernel logs unknown clients, unsupported clients, unknown or ambiguous endpoints, duplicate requests, disconnecting transports, and closed transports. These rejections do not emit a socket state change or affect another endpoint. ## Data handling and callbacks The system dispatches request responses separately from subscribed updates. The operation determines which callback handles the data. ### Request responses and subscriptions The system distinguishes between two data flows: 1. **Request responses**: - Obtained through methods like `request_bars()`, `request_quotes()`, etc. - Processed through type-specific batch handlers such as `on_historical_bars()` and `on_historical_quotes()`. - Custom data uses `on_historical_data()` once per response. A scalar `CustomData` arrives as that object, while a batch arrives as one list, including an empty list. - Used for initial data loading and historical analysis. 2. **Subscribed data**: - Obtained through methods like `subscribe_bars()`, `subscribe_quotes()`, etc. - Processed through specific handlers like `on_bar()`, `on_quote()`, etc. - Used for ongoing event processing. ### Callback handlers Common data operations map to these handlers: | Operation | Category | Handler | Purpose | | ------------------------------- | ------------ | ------------------------------- | ------------------------------------------ | | `subscribe_data()` | Subscription | `on_data()` | Custom data updates. | | `subscribe_signal()` | Subscription | `on_signal()` | Signal updates. | | `subscribe_instrument()` | Subscription | `on_instrument()` | Instrument definition updates. | | `subscribe_instruments()` | Subscription | `on_instrument()` | Instrument definition updates for a venue. | | `subscribe_book_deltas()` | Subscription | `on_book_deltas()` | Order book deltas. | | `subscribe_book_depth10()` | Subscription | `on_book_depth()` | Order book depth snapshots. | | `subscribe_book_at_interval()` | Subscription | `on_book()` | Order book snapshots at intervals. | | `subscribe_quotes()` | Subscription | `on_quote()` | Quote updates. | | `subscribe_trades()` | Subscription | `on_trade()` | Trade updates. | | `subscribe_mark_prices()` | Subscription | `on_mark_price()` | Mark price updates. | | `subscribe_index_prices()` | Subscription | `on_index_price()` | Index price updates. | | `subscribe_bars()` | Subscription | `on_bar()` | Bar updates. | | `subscribe_funding_rates()` | Subscription | `on_funding_rate()` | Funding rate updates. | | `subscribe_instrument_status()` | Subscription | `on_instrument_status()` | Instrument status updates. | | `subscribe_instrument_close()` | Subscription | `on_instrument_close()` | Instrument close updates. | | `subscribe_option_greeks()` | Subscription | `on_option_greeks()` | Option Greek updates. | | `subscribe_option_chain()` | Subscription | `on_option_chain()` | Option chain slice snapshots. | | `request_data()` | Request | `on_historical_data()` | Historical custom data. | | `request_book_deltas()` | Request | `on_historical_book_deltas()` | Historical order book deltas. | | `request_book_depth()` | Request | `on_historical_book_depth()` | Historical order book depth. | | `request_book_snapshot()` | Request | `on_book()` | Order book snapshot. | | `request_instrument()` | Request | `on_instrument()` | Instrument definition. | | `request_instruments()` | Request | `on_instrument()` | Instrument definitions. | | `request_quotes()` | Request | `on_historical_quotes()` | Historical quotes. | | `request_trades()` | Request | `on_historical_trades()` | Historical trades. | | `request_bars()` | Request | `on_historical_bars()` | Historical bars. | | `request_funding_rates()` | Request | `on_historical_funding_rates()` | Historical funding rates. | ### Request and subscription example This example shows both request and subscription handling: ```python from collections.abc import Sequence from nautilus_trader.common import DataActor from nautilus_trader.config import DataActorConfig from nautilus_trader.model import Bar from nautilus_trader.model import BarType class MyActorConfig(DataActorConfig): def __init__(self, *, bar_type: BarType, **_kwargs) -> None: super().__init__() self.bar_type = bar_type class MyActor(DataActor): def __init__(self, config: MyActorConfig) -> None: super().__init__(config) def on_start(self) -> None: # Limit the historical response, which is handled by on_historical_bars() self.request_bars( bar_type=self.config.bar_type, limit=100, ) # Deliver subscribed updates to on_bar() self.subscribe_bars(self.config.bar_type) def on_historical_bars(self, bars: Sequence[Bar]) -> None: for bar in bars: self.log.info(f"Received historical bar: {bar}") def on_bar(self, bar: Bar) -> None: self.log.info(f"Received subscribed bar: {bar}") ``` Separate request and subscription handlers let an actor distinguish bootstrap data from ongoing updates. Use historical bars to initialize indicators or baseline state, and apply different validation or logging to response batches and individual subscribed updates. :::tip When debugging data flow issues, check that you're looking at the correct handler for your data source. If you're not seeing data in `on_bar()` but see log messages about receiving bars, check `on_historical_bars()` because the data might be coming from a request rather than a subscription. ::: ## Order event handling Data actors do not manage orders or define order event callbacks. Handle order events in a `Strategy` through its specific order callbacks or `on_order_event()`. Use custom data or signals to pass derived values to a data actor when another component needs them. See [Strategies: order management](strategies.md#order-management) for the callback list. ## Related guides - [Strategies](strategies.md): Strategies extend actors with order-management capabilities. - [Data](data/): Data types and subscriptions available to actors. - [Message Bus](message_bus.md): The messaging system actors use for communication. # Adapters Source: https://nautilustrader.io/docs/latest/concepts/adapters/ Adapters connect data providers and trading venues to NautilusTrader. They translate venue-specific protocols into the domain objects and events used by the data and execution engines. Official Python adapters are available from `nautilus_trader.adapters`; the [integration guides](../integrations/index.md) document their supported capabilities. An adapter typically comprises these components: ```mermaid flowchart LR subgraph Venue ["Trading Venue"] API[REST API] WS[WebSocket] end subgraph Adapter ["Adapter"] HTTP[HttpClient] WSC[WebSocketClient] IP[InstrumentProvider] DC[DataClient] EC[ExecutionClient] end subgraph Core ["Nautilus Core"] DE[DataEngine] EE[ExecutionEngine] end API <--> HTTP WS <--> WSC HTTP --> IP HTTP --> DC HTTP --> EC WSC --> DC WSC --> EC DC <--> DE EC <--> EE ``` | Component | Purpose | | -------------------- | --------------------------------------------------------- | | `HttpClient` | REST API communication. | | `WebSocketClient` | Real-time streaming connection. | | `InstrumentProvider` | Loads and parses instrument definitions from the venue. | | `DataClient` | Handles market data subscriptions and requests. | | `ExecutionClient` | Handles order submission, modification, and cancellation. | ## Configuration and routing Each adapter exposes configuration types and factories for the clients it supports. Configs select venue-specific settings such as the product, environment, credentials, and instrument loading policy. Factories construct the clients when a `LiveNode` is built. Actors and strategies then use the common Nautilus APIs rather than calling adapter transports directly. A node can register multiple data and execution clients. Pass `client_id` from an actor or strategy when a specific client must handle a request, subscription, or order. Without an explicit client, the data and execution engines use the venue and default routes configured by the node. ## Custom adapters You can develop an adapter as a separate Python package without rebuilding NautilusTrader. Custom adapters implement the same data and execution responsibilities as built-in adapters, and can run alongside them in one `LiveNode`. Strategies continue to use the standard request, subscription, and order APIs; the node routes those operations to the registered clients. A custom package can implement its clients in Python or delegate venue operations to its own Rust/PyO3 extension. Both use the [Python client interface](../developer_guide/python_adapters.md). An independent extension exchanges Python objects from the installed NautilusTrader wheel through PyO3 and the GIL. It does not require a shared Rust ABI or a second copy of the Nautilus model types. Choose a package version tested against your installed NautilusTrader release. Register the package's factories and configs with the node under distinct client names. Custom configs can retain venue-specific fields, and importable configs support loading factories from package paths. Set venue and default routes as you would for built-in clients, or select a client explicitly from a strategy. The [deterministic Python template](../../examples/live/_template/README.md) demonstrates data delivery, execution reconciliation, and an order fill without a venue connection. ### Cache and state ownership Custom clients receive a read-only cache view. They can inspect instruments, quotes, orders, accounts, and positions, but cannot mutate the core cache. Returned objects are snapshots: changing a snapshot does not change the state seen by strategies or engines. Adapters operate outside the synchronous core boundary. They submit typed data, responses, and execution events for the core to process, so an adapter's output call does not mean the corresponding cache update has already occurred. This keeps cache mutation and engine state transitions under core ownership. Cache access and output remain bound to the owning node's thread and lifetime; holding the GIL alone does not permit access from another thread or after disposal. ### Running and stopping Use `node.run()` when the node owns its event loop, or await `node.run_async()` inside an existing asyncio application. Both launch modes support custom Python and independent PyO3 clients. The node binds clients to the running loop before connection, then uses its normal startup, reconciliation, and shutdown sequence. Constructors do not start networking or background tasks. Commands and subscription changes execute in admission order for each custom client. Historical requests and reconciliation can progress separately. A slow command delays later commands for that client, and a full command queue rejects new work rather than silently dropping it. Await shutdown before closing a host event loop. Disconnection stops admission and asks adapter work to terminate; requesting cancellation does not prove that work has finished. Incomplete cleanup is reported. Client instances belong to one node run and must not be reused for another. Redis/PostgreSQL cache backing is unsupported with custom clients in either launch mode. The interface also requires migration of v1 Cython adapters rather than accepting their source unchanged. See the [migration details and remaining limitations](../developer_guide/python_adapters.md#migration-from-v1) for historical request completion, revised bars, custom publication/subscription, and networking API differences. For independent extensions, see [package construction and installed-wheel validation](../developer_guide/python_adapters.md#independent-rustpyo3-packages). ## Instrument providers Instrument providers load venue definitions and parse them into Nautilus `Instrument` objects. Each adapter owns this behavior. Its Python API may expose a standalone loader, a dedicated provider config, loading behavior through its client config, or a combination of these. An `InstrumentProvider` serves two use cases: - Standalone discovery of available instruments for research or backtesting - Runtime loading in a `sandbox` or `live` [environment context](architecture.md#environment-contexts) for actors and strategies ### Research and backtesting This example loads one Binance USD-M instrument through the public Python API: ```python import asyncio from nautilus_trader.adapters.binance import BinanceDataClientConfig from nautilus_trader.adapters.binance import BinanceEnvironment from nautilus_trader.adapters.binance import BinanceInstrumentProviderConfig from nautilus_trader.adapters.binance import BinanceProductType from nautilus_trader.adapters.binance import load_binance_instruments async def main() -> None: config = BinanceDataClientConfig( product_type=BinanceProductType.USD_M, environment=BinanceEnvironment.LIVE, instrument_provider=BinanceInstrumentProviderConfig( load_all=False, load_ids=["BTCUSDT-PERP.BINANCE"], ), ) instruments = await load_binance_instruments(config) for instrument in instruments: print(instrument.id) if __name__ == "__main__": asyncio.run(main()) ``` ### Live trading Each integration handles startup loading differently. For example, the Binance provider config can load the full catalog: ```python from nautilus_trader.adapters.binance import BinanceInstrumentProviderConfig BinanceInstrumentProviderConfig(load_all=True) ``` It can instead load only specified instruments: ```python BinanceInstrumentProviderConfig( load_all=False, load_ids=["BTCUSDT-PERP.BINANCE", "ETHUSDT-PERP.BINANCE"], ) ``` `load_ids` contains Nautilus instrument IDs, including the venue suffix, rather than raw venue symbols. Instrument-loading settings, defaults, and filters vary by integration. Check the relevant integration guide before copying a config between adapters. Subscriptions, order submission, and execution reconciliation do not load instruments by themselves. Configure the adapter to load each required instrument at startup, or request it explicitly and wait until it reaches the cache before using it. For how reconciliation treats a report whose instrument is not loaded, see [instrument availability](execution/reconciliation.md#instrument-availability). ## Data clients Data clients handle market data subscriptions and requests for a venue. They connect to venue APIs and normalize incoming data into Nautilus types. ### Requesting data Actors and strategies can request data using built-in methods. Data returns via callbacks: ```python from collections.abc import Sequence from typing import Any from nautilus_trader.model import Bar from nautilus_trader.model import BarType from nautilus_trader.model import InstrumentId from nautilus_trader.trading import Strategy class MyStrategy(Strategy): def on_start(self) -> None: # Request an instrument definition self.request_instrument(InstrumentId.from_str("BTCUSDT-PERP.BINANCE")) # Request historical bars self.request_bars(BarType.from_str("BTCUSDT-PERP.BINANCE-1-HOUR-LAST-EXTERNAL")) def on_instrument(self, instrument: Any) -> None: self.log.info(f"Received instrument: {instrument.id}") def on_historical_bars(self, bars: Sequence[Bar]) -> None: self.log.info(f"Received {len(bars)} historical bars") ``` ### Subscribing to data For real-time data, use subscription methods: ```python from nautilus_trader.model import Bar from nautilus_trader.model import BarType from nautilus_trader.model import InstrumentId from nautilus_trader.model import TradeTick from nautilus_trader.trading import Strategy class MyStrategy(Strategy): def on_start(self) -> None: # Assumes the instrument has already been loaded into the cache self.subscribe_trades(InstrumentId.from_str("BTCUSDT-PERP.BINANCE")) self.subscribe_bars(BarType.from_str("BTCUSDT-PERP.BINANCE-1-MINUTE-LAST-EXTERNAL")) def on_trade(self, trade: TradeTick) -> None: self.log.info(f"Trade: {trade}") def on_bar(self, bar: Bar) -> None: self.log.info(f"Bar: {bar}") ``` :::tip See the [Actors](actors.md) documentation for a complete reference of available request and subscription methods with their corresponding callbacks. ::: ## Execution clients Execution clients handle order management for a venue. They translate Nautilus order commands into venue-specific API calls and process execution reports back into Nautilus events. Responsibilities: - Submit, modify, and cancel orders. - Process fills and execution reports. - Reconcile order state with the venue. - Handle account and position updates. Execution clients can declare the lower time limit applied to historical reconciliation and whether the required order, fill, and position sources completed. When an adapter supplies this contract, the engine can recover authoritative order state without applying historical position or portfolio economics that the available evidence cannot support. See [Bounded history safety](execution/reconciliation.md#bounded-history-safety). ### Reduce-only execution contract An execution client **must never silently discard** `reduce_only=true`. It must either send a documented venue instruction that enforces the same intent or reject the order before transport. The venue can still reject an encoded instruction when the product does not support it, the order would not reduce an open position, or another venue rule makes the combination invalid. An equivalent enforcing instruction can replace the literal venue flag. For example, a Binance Futures close-all conditional order retains `reduce_only=true` in the Nautilus order, but the adapter sends `closePosition=true` without Binance's incompatible `reduceOnly` field. See [Binance Futures close-position orders](../integrations/binance.md#close-position). Backtest and sandbox matching engines enforce reduce-only when `use_reduce_only` is enabled and reject reduce-only orders when it is disabled. Custom execution clients and external routes follow the same send-or-reject contract. Order commands and venue results are asynchronous. `OrderSubmitted` means that the adapter has started the submission path, not that the venue has accepted the order. A transport failure can leave the outcome unknown, so adapters use stream updates, queries, or reconciliation rather than assuming a rejection. For a new order, the `ExecutionEngine` uses an explicitly selected client, venue routing, or the configured default. Later commands for an existing order return to its originating client when known. See the [Execution](execution/) guide for order management from a strategy perspective. :::tip For building a custom adapter, see the [Adapter Developer Guide](../developer_guide/adapters.md). ::: ## Related guides - [Live trading](live.md): Configure and run live trading with adapters. - [Execution](execution/): Order execution through adapters. - [Data](data/): Market data provided by adapters. # Architecture Source: https://nautilustrader.io/docs/latest/concepts/architecture/ This page describes NautilusTrader's components, runtime boundaries, and data flows. The [design principles and policies](../developer_guide/design_principles.md) guide this structure. :::note For this guide, the **Nautilus system boundary** means the runtime of one Nautilus node instance. ::: ## Architectural style NautilusTrader uses these architectural techniques and design patterns: - [Domain-driven design (DDD)](https://en.wikipedia.org/wiki/Domain-driven_design) - [Event-driven architecture](https://en.wikipedia.org/wiki/Event-driven_programming) - [Messaging patterns](https://en.wikipedia.org/wiki/Messaging_pattern) (publish/subscribe, request/response, and point-to-point) - [Ports and adapters](https://en.wikipedia.org/wiki/Hexagonal_architecture_(software)) - [Crash-only design](#crash-only-design) The [design principles and policies](../developer_guide/design_principles.md) state the priorities and contracts these techniques support. ## System architecture NautilusTrader provides both a framework for composing trading systems and default implementations for several [environment contexts](#environment-contexts). ```mermaid flowchart LR data_clients[Data clients] exec_clients[Execution clients] cache_backing[(Optional cache backing)] bus_backing[(Optional message bus backing)] subgraph kernel[NautilusKernel] data[DataEngine] risk[RiskEngine] execution[ExecutionEngine] portfolio[Portfolio] trader[Trader: actors, strategies, algorithms] bus[MessageBus] cache[(Cache)] end data_clients -->|market data| data data -->|store| cache data -->|publish| bus bus -->|callbacks| trader trader -->|strategy portfolio access| portfolio trader -->|trading commands| risk risk -->|validated commands| execution execution <--> exec_clients execution -->|execution state| cache execution -->|events| bus bus -->|order and position events| risk risk -->|read state| cache risk -->|read portfolio state| portfolio bus -->|account, order, position, and price events| portfolio portfolio <-->|state| cache cache <--> cache_backing bus <--> bus_backing ``` The kernel owns the shared trading core; adapters exchange market data and execution messages through the engine boundaries. ### Core components #### `NautilusKernel` The central orchestration component: - Initializes and manages the shared core components. - Configures the messaging infrastructure. - Selects environment-specific clocks and behavior. - Coordinates shared resources and lifecycle management. - Provides one lifecycle boundary for system operations. #### `MessageBus` `MessageBus` centrally routes inter-component communication: - **Publish/subscribe**: Broadcasts events and data to multiple consumers. - **Request/response**: Correlates requests with their responses. - **Command/event messaging**: Routes actions and state changes through typed endpoints. - **Optional external backing**: Sends selected publications and, for live nodes, receives configured external streams through a backing such as Redis. These streams provide live transport; durable state recovery belongs to the cache or event store. #### `Cache` `Cache` keeps trading state in memory: - Stores instruments, accounts, orders, positions, and more. - Provides indexed reads for trading components. - Optionally persists configured state through a cache database backing. #### `DataEngine` Processes and routes market data throughout the system: - Handles quotes, trades, bars, order books, custom data, and other supported types. - Manages subscriptions and correlated request/response flows through data clients. - Keeps each client subscription active until its final owner releases it, retaining the original client route and parameters for the final unsubscribe. - Routes resulting data to consumers according to their subscriptions and requests. - Manages data flow from external sources to internal components. #### `ExecutionEngine` Manages order lifecycle and execution: - Routes trading commands to the appropriate execution clients. - Tracks order and position states. - Coordinates with risk management systems. - Handles execution reports and fills from venues. - Applies reconciliation events and reconciles individual venue reports. For live execution, `ExecutionManager` tracks reconciliation state and coordinates individual reconciliation operations. `LiveNode` owns recurring checks, deadlines, and cancellation. See the [reconciliation component diagram](execution/reconciliation.md#component-responsibilities) for their ownership and dependencies. #### `RiskEngine` Provides risk management: - Validates order fields, balances, quantities, notionals, reduce-only behavior, and trading state. - Applies configurable submission and modification rate limits. - Monitors order and position events used by its controls. #### `Portfolio` Maintains derived account and position state: - Tracks balances, net positions, margin, realized and unrealized profit and loss (PnL), and exposure. - Updates state and valuations from account, order, position, quote, bar, and mark-price events. #### `Trader` Coordinates user trading components: - Registers actors, strategies, and execution algorithms. - Manages their lifecycle, clocks, and event subscriptions. ### Environment contexts An environment context defines the data source and execution setting for a node: - `Backtest`: Historical data with simulated execution. - `Sandbox`: Real-time data with simulated execution. - `Live`: Real-time data with live venue connections, including paper or real accounts. ### Common core Backtest, sandbox, and live systems share the `NautilusKernel` struct from the `nautilus-system` crate. The kernel owns the common cache, portfolio, engines, trader, clock, and messaging infrastructure. The *ports and adapters* architectural style enables modular components to be integrated into the core through explicit client, backing-store, and component interfaces, including custom implementations. ### Data and execution flow patterns #### Data flow: life of a quote tick The following trace shows the path a `QuoteTick` takes from the network to a strategy. Trades and bars follow the same cache-then-publish path with different handler names. Order book deltas and depth snapshots take a different route (see the note below the steps). ```mermaid sequenceDiagram participant Adapter as DataClient adapter participant Channel as MPSC channel participant DE as DataEngine participant Cache as Cache participant MB as MessageBus participant Strategy as Strategy Adapter->>Channel: DataEvent::Data(Data::Quote(quote)) Channel->>DE: process_data(Data::Quote) DE->>DE: handle_quote(quote) DE->>Cache: add_quote(quote) DE->>MB: publish_quote(topic, quote) MB->>Strategy: on_quote(quote) ``` 1. **Adapter receives raw data.** A venue-specific `DataClient`, such as Binance or Bybit, receives a WebSocket message, parses it, and constructs a `QuoteTick`. 1. **Adapter sends a data event.** The adapter sends `DataEvent::Data(Data::Quote(quote))` through an MPSC channel. In live mode this is an async unbounded channel; in backtests the engine feeds data directly. 1. **`DataEngine` processes the event.** The channel receiver routes the event to `DataEngine::process_data`, which dispatches to `handle_quote`. 1. **`Cache` stores the quote.** `handle_quote` calls `cache.add_quote(quote)`. When the insertion succeeds, components can read it through `self.cache.quote(instrument_id)`. 1. **`MessageBus` publishes.** The engine publishes the quote on a topic derived from the instrument ID, such as `data.quotes.BINANCE.BTCUSDT-PERP`. The `MessageBus` finds all handlers subscribed to that topic. 1. **Strategy handler runs.** Each subscribed strategy's `on_quote(quote)` runs on the single-threaded core. `self.cache.quote(instrument_id)` returns the same quote only if insertion succeeded and no later quote has replaced it before the read. :::note For quotes, trades, and bars, the engine attempts cache insertion before publication. A synchronous persistence or enqueue error prevents the in-memory insertion, but the engine logs the error and still publishes the value. Built-in database backings perform the actual write asynchronously, so a later database error does not roll back the cache insertion. A published quote can therefore be absent from the cache after a synchronous insertion failure. A callback can also observe newer cache state than the event it handles, including during synchronous reentry. Delivery order does not provide an event-time snapshot, and the [queued dispatch requirements](../developer_guide/callback_dispatch.md#maintenance-and-observable-state) do not add one. Order book deltas and depth snapshots are published directly, while `BookUpdater` subscriptions maintain book state separately. ::: #### Execution flow: life of an order This simplified trace shows direct order submission with successful venue acceptance followed by a fill. It omits execution algorithms, emulation, and ambiguous transport outcomes; see [Execution](execution/index.md) and [command outcomes](execution/policies.md#command-outcomes) for those paths. Adapters translate venue messages into the execution events shown below: ```mermaid sequenceDiagram participant Strategy as Strategy participant RE as RiskEngine participant EE as ExecutionEngine participant EC as ExecutionClient participant Venue as Venue participant MB as MessageBus Strategy->>RE: submit_order(command) RE->>RE: pre-trade risk checks RE->>EE: route command EE->>EC: submit_order EC->>Venue: place order (REST/WS) Venue-->>EC: OrderAccepted EC->>EE: OrderAccepted event EE->>MB: publish OrderAccepted MB->>Strategy: on_order_accepted(event) Venue-->>EC: OrderFilled EC->>EE: OrderFilled event EE->>MB: publish OrderFilled MB->>Strategy: on_order_filled(event) ``` 1. **Strategy creates a command.** The strategy calls `self.submit_order(order)`. 1. **`RiskEngine` validates.** Configured order, balance, quantity, notional, trading-state, and rate checks run. If a check fails, the strategy receives `OrderDenied`, and the order never reaches the venue. 1. **`ExecutionEngine` routes.** The command is routed to the `ExecutionClient` for the target venue. 1. **`ExecutionClient` submits.** The adapter sends the order to the venue over REST or WebSocket. 1. **Events flow back.** The venue responds with acknowledgments and fills. Each event (`Accepted`, `Filled`, `Canceled`, `Rejected`, or `Expired`) flows through the `ExecutionEngine`, which updates order state in the `Cache` and delivers the event to the strategy's handler. Fill events also trigger position and portfolio updates. #### Component state management Types that implement the `Component` trait use a finite state machine. `ComponentState` defines stable and transitional states, while `ComponentTrigger` constrains valid transitions: ```mermaid stateDiagram-v2 [*] --> PRE_INITIALIZED PRE_INITIALIZED --> READY : initialize() READY --> STARTING : start() STARTING --> RUNNING STARTING --> STOPPING : stop() STARTING --> FAULTING : fault() RUNNING --> STOPPING : stop() STOPPING --> STOPPED STOPPING --> DISPOSING : dispose() STOPPING --> FAULTING : fault() STOPPED --> RESETTING : reset() RESETTING --> READY STOPPED --> RESUMING : resume() DEGRADED --> RESUMING : resume() RESUMING --> RUNNING RESUMING --> STOPPING : stop() RESUMING --> FAULTING : fault() RUNNING --> DEGRADING : degrade() DEGRADING --> DEGRADED DEGRADED --> STOPPING : stop() DEGRADED --> FAULTING : fault() RUNNING --> FAULTING : fault() STOPPED --> FAULTING : fault() FAULTING --> DISPOSING : dispose() FAULTING --> FAULTED READY --> RESETTING : reset() READY --> DISPOSING : dispose() STOPPED --> DISPOSING : dispose() DISPOSING --> DISPOSED DISPOSING --> FAULTING : on_dispose() error DISPOSED --> [*] ``` **Stable states:** - **PRE_INITIALIZED**: The component exists but is not ready to fulfill its specification. - **READY**: The component is configured and can start. - **RUNNING**: The component operates normally and can fulfill its specification. - **STOPPED**: The component has stopped successfully. - **DEGRADED**: The component may not meet its full specification. - **FAULTED**: The component has shut down because of a detected fault. - **DISPOSED**: The component has shut down and released its resources. **Transitional states:** - **STARTING**: The component is executing its `start` actions. - **STOPPING**: The component is executing its `stop` actions. - **RESUMING**: The component is executing its `resume` actions after a stop or degradation. - **RESETTING**: The component is executing its `reset` actions. - **DISPOSING**: The component is executing its `dispose` actions. - **DEGRADING**: The component is executing its `degrade` actions. - **FAULTING**: The component is executing its `fault` actions. Transitional states cover the corresponding lifecycle callback and should remain brief. If a callback returns an error, the transition halts in its transitional state. `dispose()` is the exception: a failing `on_dispose` moves the component to FAULTED so it can still be retired. A failed `on_stop` or `on_fault` leaves the component in its transitional state, from which trader retirement can still invoke `dispose()` and finish cleanup. After a successful reset, the component releases its retained data subscriptions before returning to READY. Its next start acquires fresh subscriptions against the reset data engine. If `on_reset` fails, the component remains in RESETTING with its subscriptions intact. During normal retirement, the trader runs `on_dispose`, releases the component's retained data subscriptions, and then removes its registry and bookkeeping entries. If `on_dispose` fails, the component remains registered with its subscriptions intact. Retiring the resulting FAULTED component releases those subscriptions without invoking the failed disposal hook again. Each component release removes its message bus handlers and decrements ownership on the routed data client. The data client sends an upstream unsubscribe only after the final owner releases the same physical subscription. Engine-managed resources, including book snapshots, synthetic feeds, spread quotes, internal bars, and option chains, follow the same final-owner rule. #### Actor vs Component traits The Rust implementation separates targeted message dispatch from lifecycle management: ```mermaid classDiagram class Actor { <> +id() Ustr +handle(message) } class Component { <> +component_id() ComponentId +state() ComponentState +register() +initialize() +start() +stop() +resume() +reset() +dispose() +degrade() +fault() } class ActorRegistry { +insert(actor) +get(id) shared actor handle } class ComponentRegistry { +insert(component) +get(id) shared component handle } Actor <|.. Throttler : implements Actor <|.. Strategy : implements Component <|.. Strategy : implements Component <|.. Trader : implements ActorRegistry --> Actor : manages ComponentRegistry --> Component : manages class Throttler { Actor only } class Strategy { Actor + Component } class Trader { Component only } ``` **`Actor` trait: message dispatch** - Provides the `handle` method for receiving messages dispatched through the actor registry. - Supports lookup by actor ID; typed unchecked accessors check the concrete actor type at runtime and return an `ActorRef` guard. - Used by types that receive targeted messages, such as strategies and throttlers. **`Component` trait: lifecycle management** - Manages state transitions such as `start`, `stop`, `resume`, `reset`, and `dispose`. - Registers a component with the trader ID, clock, and cache. - Tracks component state via the finite state machine described above. - Used by actors, strategies, execution algorithms, and the `Trader` when they need managed lifecycle behavior. The data, risk, and execution engines expose their own lifecycle methods but do not implement this trait. :::note Message bus access does not depend on the `Actor` trait. Code running on the node thread can use the thread-local `MessageBus` APIs, while `Actor` specifically enables registry-based dispatch to an actor ID. ::: This separation allows: - **Actor only**: Lightweight message handlers without lifecycle, such as `Throttler`. - **Component only**: Lifecycle-managed types without targeted actor dispatch, such as `Trader`. - **Both traits**: Data actors, including strategies and execution algorithms, that need lifecycle management and targeted dispatch. :::warning Separate thread-local registries support these access patterns. Both registry `get` methods return shared `Rc>` handles. Component lifecycle wrapper functions use a private borrow guard to reject overlapping lifecycle access; that protection does not apply to arbitrary access through a raw registry handle. Typed actor accessors return `ActorRef` guards, which do not prevent two simultaneous guards for the same actor. Creating overlapping mutable references is undefined behavior. Obtain, use, and drop an `ActorRef` within one synchronous scope. Never store one or hold it across an `.await` point. Same-actor re-entrant lookup is a constraint of the current dispatch model, not a safe aliasing guarantee. ::: For queued dispatch, releasing an actor guard alone does not establish a safe delivery boundary: enclosing mutable runtime borrows must also end. Subscriber admission order alone does not preserve publication order during nested fan-out. Pending deliveries need registration and lifecycle identity to enforce the [dispatch requirements](../developer_guide/callback_dispatch.md). ### Messaging The `MessageBus` passes data, commands, and events between components without requiring direct component references. #### Threading model Within a node, the core consumes and dispatches messages on a **single thread**. This includes: - The `MessageBus` and actor callback dispatch. - Strategy logic and order management. - Risk engine checks and execution coordination. - Cache reads and writes. Serial processing coordinates state changes within the node. Synchronous reentry still has the constraints described above; the [queued dispatch requirements](../developer_guide/callback_dispatch.md) define publication ordering across nested callbacks. Live inputs and latency can cause behavioral differences from backtests. Components consume messages synchronously in a pattern *similar* to the [actor model](https://en.wikipedia.org/wiki/Actor_model). :::note The [LMAX architecture](https://martinfowler.com/articles/lmax.html) is a related example of single-threaded transaction processing. ::: Background services use separate threads or the process-wide, multi-threaded Tokio runtime. The runtime's worker count is configurable: - **Network and adapters**: WebSocket connections, REST clients, and data feeds run as async tasks. - **Logging**: A worker receives log events outside the synchronous core. - **Persistence**: Redis and PostgreSQL cache backings queue writes to async tasks. DataFusion runs catalog query futures on a Tokio runtime. Async producers send data and execution events through channels. The node runner receives them and uses the thread-local `MessageBus` to dispatch them to engine endpoints on the core thread. Each thread has its own bus instance; channels bridge work from other threads or tasks. ## Crash-only design NautilusTrader draws on [crash-only design](https://en.wikipedia.org/wiki/Crash-only_software) when handling unrecoverable faults. Repository release builds abort on panic, allowing an external supervisor to restart the process instead of letting it continue with potentially invalid state. Recovery behavior: - **Startup recovery**: Configured cache and event-store recovery run through normal startup rather than through a separate crash-only entry point. Ordinary startup and focused recovery tests exercise the same initialization flow. - **External state**: Configured backing stores preserve selected state across process restarts, reducing recovery work and the risk of losing state. Durability depends on the backing store and its settings. - **Supervisor-managed restart**: An external process supervisor owns restart policy after an unrecoverable failure. Aborting skips graceful cleanup inside the failed process; actual downtime depends on the supervisor, configured state, and backing store. - **Prompt recovery**: The design aims to minimize downtime by using normal startup recovery after a supervisor restarts the process. Recovery time depends on the state to restore and its backing store. - **Execution recovery**: Venue commands are not generally safe to retry blindly; execution reconciliation handles that boundary. :::note Normal operation still uses graceful shutdown flows such as `stop` and `dispose`. They tear down clients and, when configured, save state and flush writers. Crash-only behavior applies to unrecoverable faults, where continuing normal cleanup may be unsafe. ::: This design complements the [data-integrity policy](../developer_guide/design_principles.md#data-integrity-and-failure): a panic caused by an unrecoverable invariant violation immediately terminates a process built with the repository release profile. **References:** - [Crash-Only Software](https://www.usenix.org/conference/hotos-ix/crash-only-software): Candea and Fox, HotOS 2003. - [Microreboot: A technique for cheap recovery](https://www.usenix.org/events/osdi04/tech/candea.html): Candea et al., OSDI 2004. - [The properties of crash-only software](https://brooker.co.za/blog/2012/01/22/crash-only.html): Marc Brooker. - [Crash-only software: More than meets the eye](https://lwn.net/Articles/191059/): LWN.net. - [Recovery-Oriented Computing (ROC) Project](http://roc.cs.berkeley.edu/): UC Berkeley and Stanford. ## Framework organization The Rust workspace groups related behavior into crates under `crates/`. The public package under `python/nautilus_trader/` provides Python facades and supporting utilities over the Rust implementation. ### Core and domain - `core`: Low-level time, string, serialization, and runtime primitives. - `model`: Trading domain types, including instruments, accounts, orders, positions, and market data. - `common`: Shared runtime services, including the cache, message bus, clocks, actors, components, and logging. - `serialization`: Schema and encoding support for model and event types. ### Trading and analysis - `analysis`: Trading performance statistics and analysis. - `indicators`: Technical indicators. - `data`: Market-data engines, aggregation, and data tooling. - `execution`: Order execution, emulation, and reconciliation primitives. - `portfolio`: Portfolio accounting and state. - `risk`: Pre-trade controls, position sizing, and trading state. - `trading`: Strategies and execution algorithms. ### Infrastructure and runtimes - `network` and `cryptography`: Networking clients, transport support, signing, and cryptographic providers. - `infrastructure`, `persistence`, and `event_store`: Database backings, data catalogs, object storage, and event-store integration. - `system`: The kernel shared by backtest, sandbox, and live [environment contexts](#environment-contexts). - `backtest` and `live`: Environment-specific engines and nodes. - `adapters/*`: Venue, broker, data, blockchain, and sandbox integrations. - `pyo3`: The Python extension aggregator. - `plugin`, `cli`, and `testkit`: Plugin interfaces, command-line tools, and test support. ## Code structure The `crates/` directory contains the Rust implementation. PyO3 collects its Python bindings into the `nautilus_trader._libnautilus` extension module, and `python/nautilus_trader/` exposes the public Python facades. The `nautilus-core` and `nautilus-model` crates retain an optional C FFI for native consumers. Other workspace crates use Rust APIs or PyO3 bindings. ### Dependency flow ```mermaid flowchart TB subgraph trader["python/nautilus_trader
Python"] end subgraph bindings["crates/pyo3
PyO3"] end subgraph core["crates
Rust"] end trader --> bindings bindings --> core ``` ### Rust crates Rust crate manifests declare workspace dependencies and optional feature flags. Features enable optional functionality without adding it to minimal builds. Selected direct workspace dependencies are shown below; arrows point to dependencies. The diagram omits edges that do not clarify the overall direction. ```mermaid flowchart BT subgraph Core["Core and domain"] core model common serialization end subgraph Trading trading data execution portfolio risk end subgraph Infrastructure network cryptography persistence end subgraph Runtime system live backtest end adapters pyo3 model --> core common --> core common --> model system --> common trading --> common serialization --> model network --> cryptography persistence --> serialization data --> common execution --> common portfolio --> common risk --> portfolio live --> system live --> trading backtest --> system backtest --> persistence adapters --> live adapters --> network pyo3 --> adapters ``` **Feature flags:** | Feature | Main crates | Effect | | ----------- | ----------------------------------------- | ---------------------------------------------------------------- | | `streaming` | `data`, `system`, `live`, `backtest` | Adds persistence support for catalog streaming. | | `cloud` | `persistence` | Adds AWS, Azure, GCP, and HTTP object-store backends. | | `python` | Python-facing crates | Adds PyO3 bindings and the transitive features each crate needs. | | `defi` | Domain, data, runtime, and binding crates | Adds DeFi and blockchain types and runtime paths. | :::note Source builds require Rust. Prebuilt Python wheels do not require a Rust toolchain at runtime. ::: ### Type safety The Rust codebase relies on the compiler's guarantees for safe code. Each `unsafe` block explicitly opts out of those guarantees, so memory and type safety depend on its documented invariants. See the Rust section of the [Developer Guide](../developer_guide/rust.md). PyO3 validates bound arguments and converts Rust errors into Python exceptions: :::info Passing an incompatible Python value to a typed PyO3 parameter raises a Python exception before the Rust method body runs. ::: ### Errors and exceptions API documentation describes expected errors from NautilusTrader and the conditions that produce them. :::warning Python's standard library and third-party dependencies can also raise exceptions outside those documented contracts. ::: ### Processes and threads :::warning[One node per process] Running multiple `LiveNode` or `BacktestNode` instances **concurrently** in the same process is not supported because their runtime state is not isolated: - **Logger mode and timestamps**: The logging subsystem uses global state; backtests switch the logging clock between static and real-time modes. - **Thread-local runtime state**: A node installs its message bus, actor and component registries, and channel senders for the thread that drives it. - **Process-wide runtime state**: The Tokio runtime and logging worker are shared by the process. Sequential execution of multiple nodes is supported when each node is disposed before the next one starts. Focused tests exercise sequential node construction and cache-backed state recovery across disposed nodes. For production deployments, add multiple strategies to one `LiveNode` within a process. For parallel execution or workload isolation, run each node in its own separate process. ::: ### Memory allocation The event-driven core allocates and frees objects during message dispatch, order event handling, and order book maintenance. Allocator choice can affect throughput and resident memory; the effect depends on the workload, platform, and build configuration. The `nautilus` CLI and Python wheels use [mimalloc](https://crates.io/crates/mimalloc) for Rust allocations. Compare allocators on representative workloads before choosing one for a custom binary. Record the source revision, build settings, environment, measurement method, throughput, and resident memory using the [benchmarking guide](../developer_guide/benchmarking.md). A Rust binary links exactly one global allocator, and libraries do not impose one, so the NautilusTrader crates remain allocator-neutral. When building directly against the crates, opt in from your own binary (see the [Rust guide](rust.md#memory-allocator)). ## Related guides - [Design principles](../developer_guide/design_principles.md): Principles, policies, and trade-offs. - [Identifier storage](../developer_guide/rust.md#identifier-storage): Identifier lifetimes and memory costs. - [Behavioral models](behavioral_models.md): Model representation and dispatch. - [Overview](overview.md): High-level introduction to NautilusTrader. - [Python](python.md): Python ownership, runtime, and public API boundaries. - [Rust](rust.md): Native Rust APIs and runtime use. - [Message Bus](message_bus.md): Core messaging infrastructure. # Behavioral Models Source: https://nautilustrader.io/docs/latest/concepts/behavioral_models/ This page describes the pluggable models that change or extend NautilusTrader behavior. A **behavioral model** supplies the rules for a specific calculation or decision made by the system. For example, a fill model determines simulated fill eligibility and liquidity, while a fee model calculates the commission on a fill. The engine calls the configured model through a defined interface, so users can change these rules without modifying the engine or their strategy. Models are supplied as objects in configuration or through runtime APIs. Users can select a built-in implementation, configure its parameters, or supply a custom implementation where the model family and API support it. Here, pluggable means replacing behavior through these interfaces; [runtime loading of native libraries](#native-extension-boundary) is a separate capability. ## Model families | Family | Controls | Built-in examples | | ------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------- | | Fill | Simulated limit-fill eligibility, slippage, and optional synthetic liquidity. | `DefaultFillModel`, `BestPriceFillModel`, `TwoTierFillModel`. | | Fee | Commission calculated from an order, fill, instrument, and optional pricing context. | `MakerTakerFeeModel`, `FixedFeeModel`, `PerContractFeeModel`. | | Latency | Simulated delays for order submission, modification, and cancellation. | `StaticLatencyModel`. | | Margin | Initial order margin and maintenance position margin. | `StandardMarginModel`, `LeveragedMarginModel`. | Fill models range from using the recorded book to supplying synthetic liquidity with tiered, partial-fill, size-aware, competition-aware, volume-sensitive, or market-hours behavior. See [fill models](backtesting/fill-models.md) for the complete built-in list and the effect of book type on fill simulation. Fee models also include `ProbabilityPriceFeeModel`, `CappedOptionFeeModel`, and `TieredNotionalOptionFeeModel` for probability-priced instruments and option fee schedules. `StaticLatencyModel` adds a base delay to separately configured insert, update, and cancel delays. The [margin models](backtesting/accounts-and-margin.md#margin-models) select whether instrument margin requirements are reduced by account leverage. Fill and latency models control simulated execution; they do not determine whether or when a live venue fills an order. Model behavior applies at the engine or account boundary that consumes it. Changing a model does not replace order-state validation, execution routing, or the rest of the runtime. ## Simulation modules [Simulation modules](backtesting/simulation-modules.md) provide a related extension point for behavior across the simulated exchange lifecycle. Instead of answering a fill, fee, latency, or margin calculation, a module processes exchange state and returns account adjustments. Built-in FX rollover and CFD swap modules use this interface. Custom modules can extend simulation behavior through the same lifecycle and acknowledgement contract. ## Supplying implementations Rust callers can implement the model family's trait and pass the implementation through its runtime handle. Python support depends on **both the family and the configuration API**: | Family | `BacktestVenueConfig` from Python | `BacktestEngine.add_venue()` from Python | | ------- | ---------------------------------------------------- | ---------------------------------------------------- | | Fill | Built-in model objects. | Built-in models or custom Python fill-model objects. | | Fee | Built-in models or custom Python commission objects. | Built-in models or custom Python commission objects. | | Latency | `StaticLatencyModel`. | `StaticLatencyModel`. | | Margin | `StandardMarginModel` or `LeveragedMarginModel`. | `StandardMarginModel` or `LeveragedMarginModel`. | A custom Python fill model implements the [fill-model protocol](backtesting/fill-models.md#configuration). A custom Python fee model supplies `get_commission`; it can also provide `get_commission_with_context` when its calculation needs the underlying price. The engine invokes these methods when it needs the corresponding decision or calculation. ## Model family structure Behavioral model families use a common representation across simulation and execution: - A Rust `Model` trait defines the behavioral contract. - Concrete Rust types implement the built-in models. - A `ModelAny` enum lists the core built-ins and any language bridges that require enum storage, then implements the trait through explicit enum dispatch. - A `ModelHandle` stores a shared trait object where runtime components accept linked Rust implementations beyond the enum variants. Supported concrete built-ins are exposed as PyO3 classes. Their [type stub annotations](../developer_guide/rust.md#type-stub-annotations) feed the [generated Python artifacts](../developer_guide/rust.md#generated-python-artifacts). Backtest configuration accepts these concrete model objects directly rather than using separate model configuration and factory wrappers. Adapter-specific models live in their adapter crate when a core enum variant would create a reverse dependency. Low-level Rust code passes these models through the corresponding handle. Python exposure uses an explicit bridge for the model family, either as an enum variant or as a trait implementation passed through the handle, depending on the storage boundary. Latency and margin configuration accept built-in models only from Python. ## Dispatch boundary | Form | Accepted implementations | Dispatch | Role | | ------------------------ | -------------------------------------- | ------------ | --------------------------------------------- | | Concrete type or generic | One concrete implementation | Static | Model internals and specialized callers. | | `ModelAny` | Declared built-ins and bridge variants | Enum match | Built-in and bridge configuration or storage. | | `ModelHandle` | Any accepted Rust trait implementation | Trait object | Shared runtime storage and custom types. | `ModelAny` uses enum dispatch. `ModelHandle` uses dynamic dispatch through a trait object, so a built-in converted from the enum into a handle crosses a vtable before its enum match. Built-in-only storage remains typed as `ModelAny` where avoiding trait-object dispatch matters. The handle has no separate built-in fast path; the simpler single representation remains because no measured performance case justifies the additional variant and dispatch complexity. ## Native extension boundary The open-source [plug-in crate](../developer_guide/plugins.md) defines an artifact ABI, but model registration and the loading host are not part of this repository. The open-source distribution does not provide runtime native model plugins. Native models are composed at compile time and passed through the corresponding enum or handle. [Simulation modules](backtesting/simulation-modules.md) use these representations at backtest configuration and runtime boundaries. # Cache Source: https://nautilustrader.io/docs/latest/concepts/cache/ The `Cache` is the central in-memory store for trading state and recent market data. Actors and strategies use it to read data maintained by the data and execution engines or to share raw bytes under application-defined keys. The cache: - Stores current order books and bounded histories of quotes, trades, bars, and other market data. - Tracks orders, positions, accounts, instruments, and currencies until they are purged or reset. - Shares caller-serialized data between components and persists it when a backing database is configured. ## How caching works The engines add built-in data to the `Cache` as events flow through the system. Live adapters feed events to the engine asynchronously, so the cache changes **when the engine processes an event**, not when the adapter first receives it. For quotes, trades, and bars, the `DataEngine` attempts to write to the `Cache` before publishing to subscribers. After a successful write, the latest value is available by the time the strategy handler runs. Order book deltas and depth snapshots are published directly; `BookUpdater` subscriptions maintain current book state separately: ```mermaid flowchart LR data[Data] engine[DataEngine] cache[Cache] callback["Strategy callback:
on_quote(...)"] data --> engine --> cache --> callback ``` For the full step-by-step trace, see [Data flow: life of a quote tick](architecture.md#data-flow-life-of-a-quote-tick). ### Basic example Within a strategy, access the shared `Cache` through `self.cache`: ```python def on_bar(self, bar: Bar) -> None: # Read recent bars from the cache. last_bar = self.cache.bar(self.bar_type, index=0) # Same bar after a successful cache write. previous_bar = self.cache.bar(self.bar_type, index=1) third_last_bar = self.cache.bar(self.bar_type, index=2) # Read current position state. if self.last_position_opened_id is not None: position = self.cache.position(self.last_position_opened_id) if position is not None and position.is_open: open_quantity = position.quantity # Read open orders for the instrument. open_orders = self.cache.orders_open(instrument_id=self.instrument_id) ``` ## Configuration Use the `CacheConfig` class to configure the `Cache` behavior and capacity. Pass it to a `BacktestEngine` or `LiveNode`, depending on the [environment context](architecture.md#environment-contexts). The same capacity settings apply in both environments: ```python from nautilus_trader.config import BacktestEngineConfig from nautilus_trader.config import CacheConfig from nautilus_trader.config import LiveNodeConfig # For backtesting engine_config = BacktestEngineConfig( cache=CacheConfig( tick_capacity=10_000, # Store last 10,000 ticks per instrument bar_capacity=5_000, # Store last 5,000 bars per bar type ), ) # For live trading node_config = LiveNodeConfig( cache=CacheConfig( tick_capacity=10_000, bar_capacity=5_000, ), ) ``` :::tip By default, the `Cache` keeps up to 10,000 values in each per-instrument tick sequence and 10,000 bars for each bar type. These are separate limits, not combined totals. Set each capacity to a value in `[1, 1_000_000]`. Increase them when a strategy needs a longer in-memory lookback and the additional memory use is acceptable. ::: ### Configuration options The `CacheConfig` type supports these parameters: ```rust use nautilus_common::{cache::CacheConfig, enums::SerializationEncoding}; let config = CacheConfig { encoding: SerializationEncoding::MsgPack, timestamps_as_iso8601: false, buffer_interval_ms: None, bulk_read_batch_size: None, use_trader_prefix: true, use_instance_id: false, flush_on_start: false, drop_instruments_on_reset: true, tick_capacity: 10_000, bar_capacity: 10_000, persist_account_events: true, save_market_data: false, }; ``` :::note Each bar type maintains its own capacity. For example, if you use both 1-minute and 5-minute bars, each stores up to `bar_capacity` bars. When `bar_capacity` is reached, the `Cache` automatically removes the oldest data. ::: ### Database configuration Configure a database backing to recover successfully persisted, supported cache records after a restart. Restorable records include general data, currencies, instruments, instrument closes, accounts, orders, and positions. Startup does not restore bounded market-data histories or the running process. Instrument closes persist whenever a backing database is configured. `save_market_data` does not gate them because they are recovery snapshots rather than bounded market-data history. `CacheConfig` controls cache behavior. Connection settings belong to the concrete backing config, such as `RedisCacheConfig` or `PostgresCacheConfig`. A backing is a recovery mechanism, not a complete event archive or a synchronized distributed cache. Each node owns its in-memory cache; pointing multiple nodes at the same database namespace does not keep those caches coherent. Rust-native callers build a concrete database config and use the `CacheDatabaseFactory` trait to construct the adapter passed into the system builder: ```rust use nautilus_common::{ cache::{CacheConfig, database::CacheDatabaseFactory}, enums::SerializationEncoding, }; use nautilus_infrastructure::redis::cache::RedisCacheConfig; let config = CacheConfig { encoding: SerializationEncoding::MsgPack, timestamps_as_iso8601: true, buffer_interval_ms: Some(100), ..Default::default() }; let database = RedisCacheConfig { host: Some("localhost".to_string()), port: Some(6379), connection_timeout: 2, response_timeout: 2, ..Default::default() }; let cache_database = database .create(trader_id, instance_id, config.clone()) .await?; ``` For a Rust-native live node, attach the adapter before startup: ```rust let node_config = LiveNodeConfig { trader_id, ..Default::default() }; let mut node = LiveNode::build("LiveNode".to_string(), Some(node_config))?; node.set_cache_database(cache_database)?; node.run().await?; ``` :::warning With the default `LiveExecutionEngineConfig.load_cache = true`, the node restores persisted cache state and rebuilds derived indexes before connecting clients or reconciling execution state. Setting `CacheConfig.flush_on_start = true` clears the backing instead. ::: Python passes the same database config to `LiveNodeBuilder.with_cache_database_factory`. The node constructs and owns the adapter when it starts, so the connection opens only when the node runs: ```python from nautilus_trader.common import Environment from nautilus_trader.infrastructure import RedisCacheConfig from nautilus_trader.live import LiveNode from nautilus_trader.model import TraderId node = ( LiveNode.builder("LiveNode", TraderId("TRADER-001"), Environment.LIVE) .with_cache_database_factory(RedisCacheConfig(host="localhost", port=6379)) .build() ) try: node.run() finally: node.dispose() ``` Pass `PostgresCacheConfig` instead to back cache data with Postgres. Postgres does not support actor or strategy state persistence, so do not combine it with `load_state` or `save_state`. Both configs come from `nautilus_trader.infrastructure`. :::warning Always dispose the node. `dispose()` closes the backing, which flushes writes still held in the buffer when `CacheConfig.buffer_interval_ms` is set. Returning straight from `run()` can drop them. ::: ## Using the cache ### Accessing market data The `Cache` provides access to order books, quotes, trades, bars, and other market data. Bounded market-data sequences use reverse indexing, so the **most recent entry sits at index 0**. #### Bar access ```python # Get all cached bars for a bar type. bars = self.cache.bars(bar_type) # Returns list[Bar] or None. # Get the most recent bar. latest_bar = self.cache.bar(bar_type) # Returns Bar or None. # Get a historical bar by index (0 = most recent). second_last_bar = self.cache.bar(bar_type, index=1) # Returns Bar or None. # Check whether bars exist and get the count. bar_count = self.cache.bar_count(bar_type) has_bars = self.cache.has_bars(bar_type) ``` #### Quote ticks ```python # Get quotes. quotes = self.cache.quotes(instrument_id) # Returns list[QuoteTick] or None. latest_quote = self.cache.quote(instrument_id) # Returns QuoteTick or None. second_last_quote = self.cache.quote(instrument_id, index=1) # Returns QuoteTick or None. # Check quote availability. quote_count = self.cache.quote_count(instrument_id) has_quotes = self.cache.has_quote_ticks(instrument_id) ``` #### Trade ticks ```python # Get trades. trades = self.cache.trades(instrument_id) # Returns list[TradeTick] or None. latest_trade = self.cache.trade(instrument_id) # Returns TradeTick or None. second_last_trade = self.cache.trade(instrument_id, index=1) # Returns TradeTick or None. # Check trade availability. trade_count = self.cache.trade_count(instrument_id) has_trades = self.cache.has_trade_ticks(instrument_id) ``` #### Order book ```python # Get the current order book. book = self.cache.order_book(instrument_id) # Returns OrderBook or None. # Check whether an order book exists. has_book = self.cache.has_order_book(instrument_id) # Get the number of applied book updates. update_count = self.cache.book_update_count(instrument_id) ``` #### Price access ```python from nautilus_trader.model import PriceType # Get the current price by type. Returns Price or None. price = self.cache.price( instrument_id=instrument_id, price_type=PriceType.MID, # Options: BID, ASK, MID, LAST ) ``` #### Bar types ```python from nautilus_trader.model import AggregationSource, PriceType # Get all available bar types for an instrument. Returns list[BarType]. bar_types = self.cache.bar_types( instrument_id=instrument_id, price_type=PriceType.LAST, # Options: BID, ASK, MID, LAST aggregation_source=AggregationSource.EXTERNAL, ) ``` #### Simple example ```python from nautilus_trader.model import Bar, BarType from nautilus_trader.trading import Strategy class MarketDataStrategy(Strategy): def on_start(self) -> None: # Subscribe to 1-minute bars. self.bar_type = BarType.from_str(f"{self.instrument_id}-1-MINUTE-LAST-EXTERNAL") self.subscribe_bars(self.bar_type) def on_bar(self, bar: Bar) -> None: bars = (self.cache.bars(self.bar_type) or [])[:3] if len(bars) < 3: return # Access the latest three bars for analysis. current_bar = bars[0] prev_bar = bars[1] prev_prev_bar = bars[2] # Read the latest quote and trade. latest_quote = self.cache.quote(self.instrument_id) latest_trade = self.cache.trade(self.instrument_id) if latest_quote is not None: current_spread = latest_quote.ask_price - latest_quote.bid_price self.log.info(f"Current spread: {current_spread}") ``` ### Trading objects The `Cache` provides access to trading objects such as: - Orders - Positions - Accounts - Instruments #### Orders Query orders by venue, strategy, instrument, account, or order side. ##### Basic order access ```python # Get a specific order by its client order ID order = self.cache.order(ClientOrderId("O-123")) # Get all orders in the system orders = self.cache.orders() # Get orders filtered by specific criteria orders_for_venue = self.cache.orders(venue=venue) # All orders for a specific venue orders_for_strategy = self.cache.orders( strategy_id=strategy_id ) # All orders for a specific strategy orders_for_instrument = self.cache.orders( instrument_id=instrument_id ) # All orders for an instrument ``` ##### Order state queries ```python # Get orders by their current state open_orders = self.cache.orders_open() # Orders currently active at the venue closed_orders = self.cache.orders_closed() # Orders that have completed their lifecycle emulated_orders = self.cache.orders_emulated() # Orders being simulated locally by the system inflight_orders = ( self.cache.orders_inflight() ) # Orders submitted (or modified) to venue, but not yet confirmed local_active_orders = ( self.cache.orders_active_local() ) # Orders still managed locally (initialized, emulated, or released) # Check specific order states exists = self.cache.order_exists( client_order_id ) # Checks if an order with the given ID exists in the cache is_open = self.cache.is_order_open(client_order_id) # Checks if an order is currently open is_closed = self.cache.is_order_closed(client_order_id) # Checks if an order is closed is_emulated = self.cache.is_order_emulated( client_order_id ) # Checks if an order is being simulated locally is_inflight = self.cache.is_order_inflight( client_order_id ) # Checks if an order is submitted or modified, but not yet confirmed is_active_local = self.cache.is_order_active_local( client_order_id ) # Checks if an order is still managed locally ``` ##### Order statistics ```python # Get counts of orders in different states open_count = self.cache.orders_open_count() # Number of open orders closed_count = self.cache.orders_closed_count() # Number of closed orders emulated_count = self.cache.orders_emulated_count() # Number of emulated orders inflight_count = self.cache.orders_inflight_count() # Number of inflight orders local_active_count = ( self.cache.orders_active_local_count() ) # Number of locally active orders (initialized, emulated, or released) total_count = self.cache.orders_total_count() # Total number of orders in the system # Get filtered order counts buy_orders_count = self.cache.orders_open_count( side=OrderSide.BUY ) # Number of currently open BUY orders venue_orders_count = self.cache.orders_total_count( venue=venue ) # Total number of orders for a given venue ``` #### Positions The `Cache` retains positions until they are purged or reset and provides several ways to query them. ##### Position access ```python # Get a specific position by its ID position = self.cache.position(PositionId("P-123")) # Get positions by their state all_positions = self.cache.positions() # All positions in the system open_positions = self.cache.positions_open() # All currently open positions closed_positions = self.cache.positions_closed() # All closed positions # Get positions filtered by various criteria venue_positions = self.cache.positions(venue=venue) # Positions for a specific venue instrument_positions = self.cache.positions( instrument_id=instrument_id ) # Positions for a specific instrument strategy_positions = self.cache.positions( strategy_id=strategy_id ) # Positions for a specific strategy long_positions = self.cache.positions(side=PositionSide.LONG) # All long positions ``` ##### Position state queries ```python # Check position states exists = self.cache.position_exists(position_id) # Checks if a position with the given ID exists is_open = self.cache.is_position_open(position_id) # Checks if a position is open is_closed = self.cache.is_position_closed(position_id) # Checks if a position is closed # Get position and order relationships orders = self.cache.orders_for_position(position_id) # All orders related to a specific position position = self.cache.position_for_order( client_order_id ) # Find the position associated with a specific order ``` ##### Position statistics ```python # Get position counts in different states open_count = self.cache.positions_open_count() # Number of currently open positions closed_count = self.cache.positions_closed_count() # Number of closed positions total_count = self.cache.positions_total_count() # Total number of positions in the system # Get filtered position counts long_positions_count = self.cache.positions_open_count( side=PositionSide.LONG ) # Number of open long positions instrument_positions_count = self.cache.positions_total_count( instrument_id=instrument_id ) # Number of positions for a given instrument ``` #### Accounts ```python # Access account information account = self.cache.account(account_id) # Retrieve account by ID account = self.cache.account_for_venue(venue) # Retrieve account for a specific venue account_id = self.cache.account_id(venue) # Retrieve account ID for a venue ``` #### Instruments ```python # Get instrument information instrument = self.cache.instrument(instrument_id) # Retrieve a specific instrument by its ID all_instruments = self.cache.instruments() # Retrieve all instruments in the cache # Get instruments for a venue. venue_instruments = self.cache.instruments(venue=venue) # Instruments for a specific venue # Get instrument identifiers instrument_ids = self.cache.instrument_ids() # Get all instrument IDs venue_instrument_ids = self.cache.instrument_ids( venue=venue ) # Get instrument IDs for a specific venue ``` ### Purging cached data Long-running sessions accumulate closed orders, closed positions, account events, and unused instruments. The cache exposes targeted and bulk purge methods so strategies and the live trading engine can keep memory bounded without restarting the system. #### Targeted purges Use these to drop a single entity. Each refuses to purge while the entity is still active. - `cache.purge_order(client_order_id)`: removes the order and every order-keyed index entry. Skips open orders. - `cache.purge_position(position_id)`: removes the position, its snapshots, and position-keyed index entries. Skips open positions. - `cache.purge_instrument(instrument_id)`: removes the instrument and its transient per-instrument maps (order book, quotes, trades, mark/index/funding prices, instrument status and close, greeks, and bars referencing the instrument). Skips while any associated order is non-terminal (anything that has not reached a closed state, including initialized, submitted, accepted, emulated, released, and inflight orders) or any associated position is non-closed. :::warning `purge_instrument` is intended for actors and strategies with their own lifecycle logic for deciding when an instrument is no longer needed. Purging an instrument that another component still relies on causes missing instrument lookups and loses market-data history. Active subscriptions belong to the data engine, so unsubscribe before purging if you no longer want updates. ::: #### Bulk purges Use these to sweep older entries by age. They take the current timestamp and a buffer or lookback window in seconds. - `cache.purge_closed_orders(ts_now, buffer_secs)`: closed orders whose close timestamp is older than `buffer_secs`. - `cache.purge_closed_positions(ts_now, buffer_secs)`: closed positions whose close timestamp is older than `buffer_secs`. - `cache.purge_account_events(ts_now, lookback_secs)`: account state events older than `lookback_secs`. A value of `0` purges all events. #### Automatic purging in live trading `LiveExecutionEngineConfig` schedules the bulk purges on a timer. All purge intervals default to `None`, which disables the corresponding loop. Set an interval to enable a loop and set its buffer or lookback to control how recent entries remain protected. This example uses the recommended starting values from the live-trading configuration guide: ```python from nautilus_trader.config import LiveExecutionEngineConfig exec_engine = LiveExecutionEngineConfig( purge_closed_orders_interval_mins=15, purge_closed_orders_buffer_mins=60, purge_closed_positions_interval_mins=15, purge_closed_positions_buffer_mins=60, purge_account_events_interval_mins=15, purge_account_events_lookback_mins=60, ) ``` A shorter interval runs a purge more often, while a shorter buffer or lookback removes newer data. Choose each value separately based on memory limits and the recent execution context needed for reconciliation or analysis. See [Configure live trading: memory management](../how_to/configure_live_trading.md) for the full parameter reference. :::note The instrument purge has no automatic loop because the right time to drop an instrument depends on strategy state, not age. Call `cache.purge_instrument` from the actor or strategy that owns the instrument's lifecycle. ::: ### Custom data The `Cache` stores raw bytes under application-defined string keys. Serialize values before adding them and deserialize them after retrieval. Actors and strategies can use these entries to share small amounts of application data. #### Basic storage and retrieval ```python # Store serialized data. self.cache.add(key="my_key", value=b"some binary data") # Retrieve serialized data. stored_data = self.cache.get("my_key") # Returns bytes or None. ``` :::warning The `Cache` is not a general database. Use a dedicated store for large datasets or complex queries. ::: ## Best practices and common questions ### Cache vs. portfolio usage The `Cache` and `Portfolio` serve different purposes: **Cache**: - Retains execution objects, selected object histories, and bounded recent market data until purge or reset. - Applies local state changes immediately, such as initializing an order before submission. - Applies external events when the engine processes them, such as when an order fills. **Portfolio**: - Aggregates position, exposure, and account information. - Computes current portfolio values from cached state and market prices. ```python from nautilus_trader.model import PositionChanged from nautilus_trader.trading import Strategy class MyStrategy(Strategy): def on_position_changed(self, event: PositionChanged) -> None: # Read the fills retained by the cached position. position = self.cache.position(event.position_id) fills = position.events() if position is not None else [] # Read current aggregate exposure from the portfolio. current_exposure = self.portfolio.net_exposure(event.instrument_id) ``` ### Cache vs. strategy variables Use cache entries for shared, serialized data and strategy variables for local working state. **Cache storage**: - Available to actors and strategies that share the system cache. - Can persist general byte entries when a backing database is configured and writes complete. - Remains available when an individual strategy resets, but a cache or execution-engine reset clears the in-memory entries. **Strategy variables**: - Keep typed, strategy-specific calculations and intermediate values encapsulated. - Do not expose values to other components or persist them automatically. Actor and strategy state persistence across process restarts uses separate `on_save` and `on_load` hooks with a supported backing. See the [cache database configuration](../how_to/configure_live_trading.md#cache-database-configuration) section of the live-trading guide. Serialize shared data before adding it to the cache: ```python import json from nautilus_trader.trading import Strategy class MyStrategy(Strategy): def on_start(self) -> None: shared_data = { "last_reset": self.clock.timestamp_ns(), "trading_enabled": True, } self.cache.add("shared_strategy_info", json.dumps(shared_data).encode()) ``` Another strategy can retrieve the cached data as follows: ```python import json from nautilus_trader.trading import Strategy class AnotherStrategy(Strategy): def on_start(self) -> None: data_bytes = self.cache.get("shared_strategy_info") if data_bytes is not None: shared_data = json.loads(data_bytes) self.log.info(f"Shared data retrieved: {shared_data}") ``` ## Related guides - [Data](data/): Data types stored in the cache. - [Strategies](strategies.md): Strategies access cache for market data and state. - [Reports](reports.md): Generate reports from cached data. # Configuration Source: https://nautilustrader.io/docs/latest/concepts/configuration/ NautilusTrader uses typed configuration objects for data clients, execution clients, engines, and strategies. Higher-level configs compose these component configs. For example, `LiveNodeConfig` owns the node's core component settings; register adapter clients through `LiveNode.builder(...)`. Adapters keep separate data and execution client configs when their capabilities or credentials differ. ## Design principles ### Concrete fields carry resolved values Rust config fields normally carry concrete values when the component requires a resolved setting. For example, adapter timeouts, retry counts, backoff delays, and heartbeat intervals often use plain types such as `u64` or `u32`. Construction resolves these fields before the component starts, so downstream code can consume them without repeating defaulting logic. ### Option semantics are field-specific In a stored Rust config, `Option` contains either `Some(value)` or `None`. The stored value does not record whether a caller omitted an input. A component may interpret `None` as disabling a feature, leaving a lookback window unbounded, falling back to the runtime environment, or applying an internal default. The **field documentation defines its meaning**. This distinction makes config semantics visible in the type. A plain `u64` always has a value, while the code consuming an `Option` handles the absent case. ### Defaults are type-specific Rust config types define defaults through `#[builder(default = value)]` annotations, a custom `Default` implementation, or both. PyO3 constructors generally resolve omitted concrete parameters from the Rust `Default` implementation instead of maintaining separate Python defaults. Container-level `#[serde(default)]` on a config struct fills its missing serialized fields from that config's `Default` implementation. Field-level `#[serde(default)]` instead uses the field type's default, unless the attribute names another function. `Type::default()` and `Type::builder().build()` are separate construction paths. A custom `Default` implementation may delegate part of its construction to the builder, but this is type-specific. Do not assume that the two paths are interchangeable unless the implementation or documentation guarantees it. ### Unknown-field handling depends on the construction path Rust deserialization and Python constructor binding enforce unknown fields independently. `BybitDataClientConfig`, for example, uses `#[serde(deny_unknown_fields)]` and rejects extra serialized keys. A Rust type without that attribute may accept them. Fixed Python config constructors raise `TypeError` for unsupported keywords. `DataActorConfig`, `StrategyConfig`, and `ExecutionAlgorithmConfig` accept additional keywords for Python subclasses. Do not infer one construction path's strictness from another. ## Python configs Import core config types from `nautilus_trader.config`. Import adapter configs from the adapter's public module, such as `nautilus_trader.adapters.bybit`. Most runtime config classes are PyO3 wrappers around Rust config structs. In a Python constructor, omitting a parameter whose signature default is `None` is equivalent to passing `None` explicitly. The wrapper then either selects the Rust default or preserves an absent optional value, depending on the field. Check the field documentation rather than inferring its behavior from the Python annotation. Properties expose selected config values. Configs that hold secrets can omit their values or expose only presence checks; consult the config API before displaying or logging a config. Mutability is type-specific: many configs expose only read-only getters, while extensible component configs and some adapter configs expose documented setters. `DataActorConfig`, `StrategyConfig`, and `ExecutionAlgorithmConfig` also accept extra fields for Python subclasses. Python-owned analysis configs retain their documented dataclass behavior. ```python from nautilus_trader.adapters.bybit import BybitDataClientConfig omitted = BybitDataClientConfig() explicit_none = BybitDataClientConfig( http_timeout_secs=None, base_url_http=None, ) assert omitted.http_timeout_secs == explicit_none.http_timeout_secs == 60 assert omitted.base_url_http is explicit_none.base_url_http is None # Override the timeout config = BybitDataClientConfig(http_timeout_secs=30) # Read the resolved value assert config.http_timeout_secs == 30 ``` When a wrapper maps `None` to a non-`None` Rust default, Python cannot use that parameter to store Rust `None`. For example, passing `instrument_status_poll_secs=None` to `BybitDataClientConfig` retains its 60-second default. Rust callers can set `instrument_poll_interval_secs` to `None` to disable periodic instrument and status polling. ## Rust configs Many Rust config structs derive [`bon::Builder`](https://bon-rs.com), which generates a type-safe builder with compile-time checks for required fields. A builder can omit fields that declare a builder default. Use the construction style documented for the config type. For `DataEngineConfig`, the builder and struct update forms below both enable delta buffering and retain the declared defaults for other fields: ```rust use nautilus_data::engine::config::DataEngineConfig; let with_builder = DataEngineConfig::builder() .buffer_deltas(true) .build(); let with_struct_update = DataEngineConfig { buffer_deltas: true, ..Default::default() }; ``` Use `DataEngineConfig::default()` when no fields need an override. ## Adapter config fields Names recur across adapter configs, but their types and defaults depend on the adapter and client. For example, `BybitDataClientConfig` defines the fields below. The `Default` column shows the values from `BybitDataClientConfig::default()`: | Rust field | Rust type | Default | Purpose | | ------------------------------- | ------------- | ---------- | ------------------------------ | | `http_timeout_secs` | `u64` | `60` | REST request timeout. | | `max_retries` | `u32` | `3` | Maximum retry attempts. | | `retry_delay_initial_ms` | `u64` | `1_000` | Initial backoff delay. | | `retry_delay_max_ms` | `u64` | `10_000` | Maximum backoff delay. | | `heartbeat_interval_secs` | `u64` | `20` | WebSocket keepalive interval. | | `recv_window_ms` | `u64` | `5_000` | Signed request expiry window. | | `instrument_poll_interval_secs` | `Option` | `Some(60)` | Instrument and status polling. | Python exposes `instrument_poll_interval_secs` as `instrument_status_poll_secs`. `BybitDataClientConfig::builder().build()` instead leaves `instrument_poll_interval_secs` as `None`, which disables periodic instrument and status polling. This is one case where the type's **default and builder paths differ**. Adapter-specific fields such as rate limits, polling intervals, and margin modes are documented in the [integration guides](../integrations/index.md). ## Engine configs Engine configs use the same typed-field approach. In `LiveExecutionEngineConfig`, fields such as `reconciliation`, `inflight_check_interval_ms`, and `open_check_threshold_ms` have concrete defaults: | Field | Default | Purpose | | ---------------------------- | ------- | ------------------------------------------------------ | | `reconciliation` | `True` | Run reconciliation during startup. | | `inflight_check_interval_ms` | `2_000` | Check whether in-flight orders exceed their threshold. | | `open_check_threshold_ms` | `5_000` | Wait before acting on an open-order discrepancy. | Optional fields such as `open_check_interval_secs` and `position_check_interval_secs` enable or disable their periodic checks: ```python from nautilus_trader.config import LiveExecutionEngineConfig config = LiveExecutionEngineConfig( open_check_interval_secs=30.0, # Enable open order polling open_check_lookback_mins=60, # Look back 60 minutes ) assert config.open_check_interval_secs == 30.0 assert config.open_check_lookback_mins == 60 assert config.position_check_interval_secs is None # Disabled by default ``` After the live node completes startup, an available execution client lets this config schedule open-order report requests every 30 seconds and limit each request to the previous 60 minutes. It does not schedule periodic position report requests. Supplied periodic intervals must be positive, finite values of at least one nanosecond. The separate `reconciliation` field enables startup reconciliation and defaults to `True`; the interval fields control periodic checks independently. When startup reconciliation is enabled, `reconciliation_startup_delay_secs` also delays the first periodic check after startup. For the full set of live engine options, see [ExecutionEngine configuration](../how_to/configure_live_trading.md#executionengine-configuration). # Continuous Futures Source: https://nautilustrader.io/docs/latest/concepts/continuous_futures/ A **continuous future** is a derived series that splices consecutive futures contracts into one adjusted price stream. Each underlying contract expires, so the continuous series rolls to the next contract at a transition point. Each segment is adjusted into a common price frame so contract changes do not introduce artificial price jumps. Nautilus models a continuous future as a target `BarType` plus an explicit list of roll transitions supplied in request or subscription params. The data engine selects the real contract for each time segment, computes its cumulative price adjustment, and feeds the adjusted source data through the normal bar aggregation path. ## Adjustment modes `ContinuousFutureAdjustmentType` combines direction (backward or forward) with operation (spread or ratio): | Mode | Operation | Anchor segment | | ----------------- | -------------- | ----------------------------------- | | `BACKWARD_SPREAD` | Additive | Last contract in adjustment range. | | `FORWARD_SPREAD` | Additive | First contract in adjustment range. | | `BACKWARD_RATIO` | Multiplicative | Last contract in adjustment range. | | `FORWARD_RATIO` | Multiplicative | First contract in adjustment range. | The cumulative adjustment at segment `k` of `N` transitions is: ```text BACKWARD_SPREAD: sum over i in [k, N) of (post_i - pre_i) FORWARD_SPREAD: sum over i in [0, k) of (pre_i - post_i) BACKWARD_RATIO: product over i in [k, N) of (post_i / pre_i) FORWARD_RATIO: product over i in [0, k) of (pre_i / post_i) ``` Spread modes accumulate additive offsets. Ratio modes accumulate multiplicative factors and require strictly positive prices. ## Inputs A continuous-future request or subscription is any `RequestBars` or `SubscribeBars` that carries a `continuous_future_transitions` entry in `params`: ```python params = { "continuous_future_transitions": [ { "transition_time_ns": 1773671460000000000, # when ESH26 rolls to ESM26 "pre_instrument_id": "ESH26.XCME", "post_instrument_id": "ESM26.XCME", "pre_price": "6001.00", # last ESH26 price pre-roll "post_price": "5995.50", # first ESM26 price post-roll }, # ... more transitions ... ], "continuous_future_adjustment_mode": "BACKWARD_SPREAD", # Optional: cap the upper end of cumulative adjustment at the transition whose # post_instrument_id matches (the backward-mode anchor). # "last_post_instrument_id": "ESM26.XCME", # Optional: cap the lower end of cumulative adjustment at the transition whose # pre_instrument_id matches (the forward-mode anchor). # "first_pre_instrument_id": "ESM26.XCME", } ``` `continuous_future_adjustment_mode` defaults to `BACKWARD_SPREAD` when omitted. The `bar_type` on the request or command is the **target** continuous bar type, for example `"ES.XCME-1-MINUTE-LAST-INTERNAL@1-MINUTE-EXTERNAL"`. The root identifier (`ES.XCME`) is the continuous root, not a real contract. Each segment's raw source data comes from the real contract in the transitions list. The continuous target bar type must be **internally aggregated**. Externally aggregated bars are not supported as continuous targets, but they can serve as the per-segment source. ### Bounded chains The two optional bounds restrict which transitions contribute to the cumulative adjustment. They do not remove contract segments from the request or subscription: - `last_post_instrument_id` caps the upper end at the first transition whose `post_instrument_id` matches. Backward modes use the matching post contract as the zero-adjustment anchor; forward modes exclude later transitions from the cumulative adjustment. - `first_pre_instrument_id` caps the lower end at the first transition whose `pre_instrument_id` matches. Forward modes use the matching pre contract as the zero-adjustment anchor; backward modes exclude earlier transitions from the cumulative adjustment. These bounds let callers pass a wider transition table while choosing the adjustment range. ## Validation The request and subscription paths apply the same transition-parameter validation rules before allocating an aggregator or child segment state: - When supplied, `continuous_future_adjustment_mode` must parse as a valid `ContinuousFutureAdjustmentType`. - `continuous_future_transitions` must be a non-empty array of transition rows. - Each row must include a non-negative integer `transition_time_ns`, and transition times must be strictly increasing. - Each `pre_instrument_id` and `post_instrument_id` must parse as a valid `InstrumentId` whose venue equals the target venue. - The chain must be continuous: row `i`'s `post_instrument_id` must equal row `i + 1`'s `pre_instrument_id`. - Each row must include finite `pre_price` and `post_price`. Ratio modes additionally require both prices to be positive. - If the caller supplies `last_post_instrument_id`, it must parse as an `InstrumentId`, match the target venue, and appear as a `post_instrument_id` in the transition list. The same applies to `first_pre_instrument_id`. A validation error therefore returns before either path starts an aggregation workflow. After validation, the request path releases its request-scoped aggregators if setup or the initial segment dispatch fails. A failure while dispatching a later segment still ends the request with a completion response and normal aggregator cleanup. ## Target instrument auto-synthesis The continuous root (for example `ES.XCME`) is a synthetic id with no market data of its own, but downstream consumers (aggregators, cache lookups, serialization) still expect an `Instrument` in the cache. After validation, both the request and subscription paths ensure the target instrument exists: - If the target id is already cached, the target setup is a no-op. Callers can pre-register a custom continuous instrument and the engine respects it. - Otherwise the target setup fetches the first segment's instrument from the cache and clones it, overriding only `id`, `raw_symbol`, and clearing `activation_ns` and `expiration_ns` to `0`. Every other field (currency, precision, increment, multiplier, lot size, underlying, fees, margins, exchange, tick scheme, info) is reused from the segment. - If the first segment is not yet in the cache or is not a `FuturesContract`, the setup logs a warning and returns. The caller must then register the continuous instrument manually. ## Architecture overview ```mermaid flowchart TD User([User/Strategy]) -->|"params['continuous_future_transitions']"| Entry{"Entry point"} Entry -->|RequestBars| ReqPath[Request path] Entry -->|SubscribeBars| SubPath[Subscription path] ReqPath --> ReqSegments[Segment dispatcher] SubPath --> SubRoller[Active segment + time alert] ReqSegments -->|per segment| ChildReq[Child request for segment contract] SubRoller -->|active segment| ChildSub[Child subscription for segment contract] ChildReq --> Agg[(Primary aggregator
BarBuilder.set_adjustment)] ChildSub --> Agg2[(Live aggregator
BarBuilder.set_adjustment)] Agg -->|adjusted bars| ReqAgg[(Request-scoped aggregator chain)] ReqAgg -->|bars at every level| Cache[(Cache)] Agg2 -->|adjusted bars| MsgBus[(msgbus: data.bars.*)] ``` Request-path bars land in the cache; subscription-path bars publish to the message bus. Both paths use the same segmentation, source resolution, and adjustment calculation. The request path processes segments in sequence; the subscription path keeps one source active and switches it when the time alert for the next transition fires. ## Segments A **segment** is a contiguous time slice owned by one real contract. Transitions separate segments. Given `transitions[0..N)`: - Segment 0: `(-inf, transitions[0].time)` on `transitions[0].pre_instrument_id`. - Segment k, with k in `[1, N)`: `[transitions[k-1].time, transitions[k].time)` on `transitions[k].pre_instrument_id`. - Segment N: `[transitions[N-1].time, +inf)` on `transitions[N-1].post_instrument_id`. The request path clips each segment to the requested time range and dispatches the segments in order. The subscription path uses the engine clock to select the active segment and schedules the next remaining transition. ## Request flow The request path dispatches one child request at a time. When a child response arrives, the engine aggregates its data and advances to the next segment. ```mermaid sequenceDiagram participant User participant Engine as DataEngine participant Agg as Primary aggregator participant Client as DataClient User->>Engine: RequestBars with transitions Engine->>Agg: initialize aggregators and cursor loop one iteration per segment Engine->>Agg: BarBuilder.set_adjustment(offset, mode) Engine->>Client: child request for segment contract Client-->>Engine: DataResponse Engine->>Agg: aggregate child response Engine->>Engine: advance cursor end Engine->>User: completion response ``` :::info Adjusted bars are written to the cache as each child response is processed. The completion response signals that the request has finished and reports the source record count; it does not contain a combined vector of adjusted bars. ::: ### Chain aggregators If a request sets `bar_types = (bar_type_1, bar_type_2)` for multi-level internal aggregation, the engine creates an isolated request-scoped aggregator for each level. Segment source responses enter the primary continuous target, and its emitted bars feed matching downstream aggregators. Only the primary builder receives the adjustment; higher levels re-aggregate already adjusted data. ## Subscription flow A small state machine drives each active subscription via a single pending time alert: ```mermaid stateDiagram-v2 [*] --> Active: subscribe(segment_i active, timer for transition_i) Active --> Active: roll(deactivate segment_i, activate segment_{i+1}, schedule next timer) Active --> [*]: unsubscribe(cancel timer, deactivate segment) ``` When a transition fires, the engine deactivates the current segment (unsubscribes the source), applies the next segment's adjustment, subscribes to the new source, and arms the timer for the following transition. ## Source resolution For any continuous-future target `BarType`, the raw data feeding the primary aggregator lives on the **segment contract**, not the continuous id. The target's shape decides the source type: ```mermaid flowchart TD Target[target_bar_type] --> Check1{is_composite?} Check1 -->|yes| Ref[reference = target.composite] Check1 -->|no| RefNo[reference = target] Ref --> Check2{externally_aggregated?} RefNo --> Check2 Check2 -->|yes| Bars["source = bars (RequestBars / SubscribeBars)"] Check2 -->|no| Check3{price_type} Check3 -->|LAST| Trades["source = trades (TradeTicks)"] Check3 -->|MID/BID/ASK| Quotes["source = quotes (QuoteTicks)"] ``` For internally aggregated sources, `LAST` uses trades, while `BID`, `ASK`, and `MID` use quotes. Other price types are not supported by quote aggregation. ## BarBuilder adjustment The builder applies the adjustment **at ingress** on every `update(price, ...)` and `update_bar(bar, ...)` call. The running OHLC state therefore remains in the adjusted common frame. Changing the adjustment during a bar affects only subsequent input. ```mermaid flowchart LR Tick[raw price] --> AdjCheck{adjustment_mode} AdjCheck -->|inactive| Raw[pass through] AdjCheck -->|spread| SpreadApply[price + adjustment_raw] AdjCheck -->|ratio| RatioApply[price * adjustment_ratio] Raw --> Update[update OHLC state] SpreadApply --> Update RatioApply --> Update Update --> Build[build on trigger] ``` The `BarBuilder` uses the mode only to choose addition or multiplication. The engine resolves the adjustment direction into a cumulative value before calling `set_adjustment`. The `reset()` method clears per-bar OHLCV state for the next bar but preserves the segment-scoped adjustment. ## Mid-bar roll boundary If a roll lands inside an in-progress target bar, the builder keeps the current OHLC state and applies the new adjustment only to subsequent updates. The pre-boundary portion stays at the old offset; the post-boundary portion uses the new offset. Rewriting the existing OHLC under the new adjustment would require raw input that the builder does not retain. ## Limitations - The feature requires supplied transition metadata. The engine does not discover rolls, choose contracts, or infer roll prices: that is the caller's responsibility. - Ratio adjustment converts the factor and each price through `f64` before rebuilding the adjusted `Price`. For high-precision instruments, the result can differ from equivalent `Decimal` multiplication. Spread adjustment remains exact in the fixed-point representation because it adds directly to `PriceRaw`. # Custom Data Source: https://nautilustrader.io/docs/latest/concepts/custom_data/ NautilusTrader supports custom data authored in Python or Rust. Both forms use the same runtime routing, persistence, and query pipeline as built-in data. This document explains how custom data is: - Registered at runtime. - Wrapped across the Python/Rust boundary. - Serialized to and from Arrow/Parquet. - Routed through actors and strategies. ## Goals The custom-data architecture satisfies the following requirements: - Let users define custom data in pure Python without writing Rust code. - Let Rust-defined custom data use native Rust JSON and Arrow handlers. - Preserve a single user-facing `CustomData` wrapper at the PyO3 boundary. - Support persistence in `ParquetDataCatalog` using dynamic type registration instead of hardcoded schemas. - Make custom data routable through the normal data-engine, actor, and strategy subscription flow. ## High-level model There are two supported authoring modes: | Mode | Authoring form | Registration path | Encode/decode path | Wrapper backend | | ---------------- | ----------------------------------------------- | ----------------------------------------------------------------- | ----------------------------- | ------------------------- | | Pure Python | Class with JSON and Arrow methods | `register_custom_data_class(...)` | Python callback + Arrow C FFI | `PythonCustomDataWrapper` | | Same-binary Rust | `#[custom_data]` or `#[custom_data(pyo3)]` type | `ensure_custom_data_registered::()` and extractor registration | Native Rust | Native Rust payload | Both modes converge on the same outer PyO3 `CustomData` wrapper and the same `DataType` identity model. ## End-to-end flow ```mermaid sequenceDiagram participant U as User code participant P as Python layer participant R as Rust model/catalog participant G as Process-wide registries participant S as Storage U->>P: define class/type U->>P: register_custom_data_class(...) or module init P->>R: install type registration R->>G: store JSON/Arrow and optional extractor handlers U->>P: CustomData(data_type, data) P->>R: write_custom_data([...]) R->>G: lookup encoder by type_name G-->>R: encoder R->>S: write RecordBatch to Parquet U->>P: query(type_name, ...) P->>R: query catalog R->>S: read RecordBatch + metadata R->>G: lookup decoder by type_name G-->>R: decoder R-->>P: CustomData wrappers P-->>U: typed data via .data ``` ## Core components ### Registry module `crates/model/src/data/registry.rs` holds the process-wide JSON, Arrow, and Python extraction registries. Registration uses atomic `DashMap::entry()` operations so concurrent `register_*` and `ensure_*` calls do not race when claiming an entry. The module initializes its registry state through `OnceLock` and stores: - JSON deserializers keyed by `type_name`. - Arrow schemas, encoders, and decoders keyed by `type_name`. - Python extractors that convert a Python object into `Arc`. - Rust extractor factories that produce Python extractors for same-binary types. Instead of hardcoding every type into the main binary, NautilusTrader resolves handlers at runtime using the `type_name` stored in `DataType` and Parquet metadata. ### `CustomData` The outer PyO3 `CustomData` wrapper is the common container that crosses the FFI boundary. Constructor signature: `CustomData(data_type, data)` where `DataType` comes first, then the inner payload. It contains: - A `DataType`. - An inner custom payload implementing `CustomDataTrait` (wrapped in `Arc`). Timestamps (`ts_event`, `ts_init`) are delegated to the inner `CustomDataTrait` implementation and exposed as properties on the wrapper. On the Python side, `CustomData` implements `__eq__` and `__repr__`. The Rust `PartialEq` implementation compares the `DataType` and delegates payload equality to the inner value. Instances are intentionally unhashable so equality remains consistent with the payload comparison. This wrapper is shared across both custom-data modes. User code interacts with one API even though the underlying payload may be: - A Python-backed wrapper. - A same-binary Rust value. #### `CustomData` JSON envelope When serialized to JSON, such as for `to_json_bytes` / `from_json_bytes`, the SQL cache, or Redis, `CustomData` uses one canonical envelope. Deserialization therefore does not depend on user payload field names: - `type`: The custom type name (from `CustomDataTrait::type_name`). - `data_type`: An object with `type_name`, `metadata`, and optional `identifier`. - `payload`: The inner payload only (the result of `CustomDataTrait::to_json` parsed as a value). Registered deserializers pass only this value to `from_json`, so user structs can use any field names, including `value`, without conflicting with wrapper metadata. This envelope is produced by Rust `CustomData` serialization and consumed by the registry module when deserializing custom data from JSON. ### `DataType` `DataType` identifies custom data for routing and persistence. Constructor: `DataType(type_name, metadata=None, identifier=None)`. It includes: - `type_name`. - Optional `metadata`. - Optional `identifier` for persistence paths and cache database lookups. It does not affect routing, equality, or hashing. **Equality, hashing, and topic routing** are derived from `type_name` and `metadata` only. Two `DataType` values with the same type name and metadata but different identifiers compare equal and publish to the same message bus topic. The `identifier` selects the catalog path under `data/custom//` and participates in PostgreSQL and Redis filtering. Persistence stores the full `DataType` with each `CustomData` value and restores it on query, while handler lookup uses `type_name`. The same logical type can therefore carry different metadata or identifiers and still decode through the same registered handler. ## Registration architecture Registration bridges the gap between Python objects and Rust trait objects. ```mermaid flowchart TD A[User-defined custom type] --> B{Mode} B --> C[Pure Python] B --> D[Same-binary Rust] C --> F[register_custom_data_class] D --> G[ensure_custom_data_registered and native extractor] F --> I[Python callbacks registered] G --> J[Native JSON and Arrow handlers registered] I --> L[Process-wide registries] J --> L ``` ### Pure Python registration When Python code calls `register_custom_data_class(MyType)`: 1. Rust retains the class for JSON reconstruction. 1. Rust registers JSON and Arrow handlers that invoke the class callbacks. 1. When constructing `CustomData`, Rust uses a registered native extractor if it accepts the object. Otherwise, it wraps the object in `PythonCustomDataWrapper`. JSON and Arrow callbacks on this path run under the Python GIL. ### Same-binary Rust registration For Rust types compiled into the process: 1. `#[custom_data]` or `#[custom_data(pyo3)]` generates the trait and JSON implementations, plus Arrow implementations by default. 1. `ensure_custom_data_registered::()` inserts native schema/encoder/decoder handlers into the process-wide registries. 1. `ensure_rust_extractor_registered::()` registers an extractor factory for PyO3-exposed types. Once activated through Python class registration, the extractor can recover the concrete Rust type instead of using the Python wrapper. This path stays fully native in Rust for encode/decode. ### Registration precedence `register_custom_data_class(...)` resolves handlers in the following order: 1. Use a native extractor and existing native JSON/Arrow handlers when they are registered. 1. Otherwise, use the Python wrapper and callback handlers. The idempotent `ensure_*` registrations do not overwrite existing native handlers. ## Wrapper backends Internally, the outer `CustomData` wrapper can hold different payload implementations. ### `PythonCustomDataWrapper` Used for pure Python custom data. Responsibilities: - Stores a reference to the Python object. - Caches `ts_event`, `ts_init`, and `type_name`. - Implements `CustomDataTrait`. - Supports JSON and Arrow callback paths that invoke Python under the GIL. This is the construction fallback when no registered extractor accepts the object. Python JSON and Arrow decoders also produce this wrapper directly. ### Native same-binary Rust payload For Rust types compiled into the process, the inner payload is the concrete Rust type and can be downcast directly from `Arc`. No Python callback path is needed for serialization or decode. ## Persistence architecture ### Why dynamic Arrow registration is needed Built-in NautilusTrader data types have schemas and encoders known statically to the Rust binary. Custom data does not. The persistence layer therefore resolves custom data dynamically using the registered `type_name`. ### Catalog write flow `ParquetDataCatalog` expects custom writes to come in as `CustomData` values. The custom-data write path: 1. Takes `type_name` from the inner payload and `metadata` and `identifier` from the first value's `DataType`. 1. Looks up the Arrow encoder in the process-wide registry. 1. Encodes the values to a `RecordBatch`. 1. Appends a `data_type` column containing the persisted `DataType`. 1. Attaches `type_name` and metadata to the Arrow schema. 1. Writes the batch to Parquet under the custom-data path. The path layout is `data/custom//`. Identifiers are normalized before becoming path segments. ### Catalog read flow On query: 1. The catalog reads matching Parquet files. 1. Extracts `type_name` from schema metadata. 1. Asks the process-wide registry for the decoder. 1. Decodes the `RecordBatch` into `Vec`. 1. Reconstructs `CustomData` with the original `DataType`. This makes custom-data query resolution symmetric with write-time registration. When converting a Feather stream to Parquet, such as after a backtest, the custom-data branch is designed to transform the Arrow batches and write the result directly to the matching custom-data path. :::info The direct Python `StreamingFeatherWriter.write()` method rejects `CustomData` with an `OSError`. Write custom data directly to the catalog with `ParquetDataCatalog.write_custom_data` from Python. The Rust Feather writer supports custom data, but `convert_stream_to_data` does not currently convert custom-data Feather streams to Parquet. ::: ## The Arrow C FFI bridge Pure Python custom data does not provide native Rust Arrow encode logic. For these types, NautilusTrader uses the Arrow C FFI interface to pass `RecordBatch` data between Python and Rust without JSON or binary serialization. ```mermaid sequenceDiagram participant R as Rust encoder participant P as Python payload participant F as Arrow C FFI structs participant C as Parquet writer R->>P: encode_record_batch_py(items) P->>P: build pyarrow.RecordBatch P-->>F: _export_to_c (FFI_ArrowArray + FFI_ArrowSchema) F-->>R: reconstruct native RecordBatch R->>C: write Parquet ``` ### Pure Python encode path For pure Python classes: 1. Rust acquires the GIL. 1. Rust calls `encode_record_batch_py(...)` on the first Python payload. 1. Python converts objects to a `pyarrow.RecordBatch`. 1. Python exports the batch via `_export_to_c` into Arrow C FFI structs. 1. Rust reconstructs a native `RecordBatch` from the FFI structs and writes it. ### Pure Python decode path For the reverse direction: 1. Rust converts its `RecordBatch` into Arrow C FFI structs. 1. Python imports the batch via `RecordBatch._import_from_c`. 1. Python calls `decode_record_batch_py(metadata, batch)` on the class. 1. Rust wraps the returned Python objects in `PythonCustomDataWrapper`. ### Native paths The Arrow C FFI bridge is not used for same-binary Rust custom data. Those types use native Rust encode/decode handlers registered in the main process. ## Reconstruction on query When custom data is loaded back from the catalog, reconstruction depends on the backend: - Same-binary Rust types decode directly to native Rust values. - Pure Python types reconstruct through the registered class's `decode_record_batch_py(...)` callback. In all cases the caller receives the same outer `CustomData` wrapper at the PyO3 API boundary. ## Runtime integration Custom data also participates in NautilusTrader runtime routing. Relevant integrations include: - `crates/data/src/engine/mod.rs` publishes `CustomData` through the message bus. - `crates/common/src/msgbus/switchboard.rs` derives custom topics from `DataType`. - `crates/common/src/actor/*` routes custom data into actor subscriptions. - `crates/trading/src/python/strategy.rs` exposes custom data to Python strategy `on_data`. - `crates/backtest/src/engine.rs` treats `Data::Custom` as data-engine-delivered input rather than exchange-routed data. A registered custom type can be persisted, queried, subscribed to, and consumed through the same runtime interfaces as built-in data families. ## Cache database integration The PostgreSQL and Redis cache database implementations support `CustomData`. - PostgreSQL stores custom data in the `custom` table. - The stored record includes `data_type`, `metadata`, `identifier`, and full JSON payload. - Reads reconstruct `CustomData` using `CustomData::from_json_bytes(...)`. - Python SQL bindings expose `add_custom_data` and `load_custom_data`. - Redis cache stores custom data under keys `custom::` with full `CustomData` JSON as value. - Redis `add_custom_data` and `load_custom_data` filter by `DataType` (type_name, metadata, identifier) and return results sorted by `ts_init`; this is exposed via the PyO3 `RedisCacheDatabase` API. ## Practical implications Python-only authoring and native Rust encode/decode remain two backends of one conceptual custom-data system rather than separate Python-only and Rust-only feature sets. # DST Source: https://nautilustrader.io/docs/latest/concepts/dst/ Deterministic simulation testing (DST) runs NautilusTrader under a seed-controlled runtime. This guide defines: - The reproducibility guarantees for seed-controlled execution. - The conditions required for those guarantees. - The source seams and checks that enforce the contract. - The paths where deterministic execution stops. Source locations accompany each claim so users and auditors can check the contract against the code. :::note A downstream harness that depends on NautilusTrader's determinism consumes the version of this document at its pinned NautilusTrader commit. A change to this document is a contract change for those consumers and should be reviewed as one. ::: ## What DST is DST controls the sources of nondeterminism in a concurrent system. Within the contract defined below, one seed determines task scheduling, timer firings, and random values. Two runs with the same seed, binary, configuration, and platform produce identical observable behavior. Record a failing seed to replay the same execution. A conventional async runtime draws scheduling decisions from ambient process state, including: - Task wake order. - Timer resolution. - OS thread scheduling. - Randomized hash seeds. A conventional test harness does not control these sources, so a race that appears once in CI can be difficult to reproduce. DST replaces them with a seeded pseudorandom sequence. Varying the seed explores different interleavings; reusing it selects the same interleaving. [FoundationDB](https://apple.github.io/foundationdb/testing.html) uses the pattern to test a production distributed database. In the Rust ecosystem, [madsim](https://crates.io/crates/madsim) intercepts `tokio` primitives to provide a deterministic scheduler. DST targets concurrency defects such as: - Channel wakeup ordering. - Drain races during shutdown. - Startup sequencing. - Reconciliation ordering. - Recovery-path correctness. Other test layers cannot exhaustively cover these interleavings. A deterministic scheduler can explore them across seeds and replay a failing schedule. ## Goals | Goal | Requirement | | ----------------------------- | ----------------------------------------------------------------------------------------------------------- | | Seed-reproducible execution | The in-scope runtime produces the same observable behavior for the same seed and inputs. | | Explicit scope | Every fallback to real time, unseeded randomness, or other nondeterminism is documented. | | Enforcement in source | Static checks reject banned patterns on the DST path before they rely on reviewer attention. | | Minimum required interception | Time, task scheduling, and randomness route through deterministic sources only where the contract needs it. | ## Approach The implementation has two layers. The first replaces selected `tokio` primitives with `madsim`. The second controls sources of nondeterminism outside those primitives. ### Layer 1: runtime swap Under the `simulation` Cargo feature on `nautilus-common`, four facade submodules select simulation behavior when `RUSTFLAGS="--cfg madsim"` is set: | Submodule | Covers | Normal build | Simulation build | Adoption | | --------- | ----------------------------------------------- | ------------- | ---------------------------------------------------------------- | ------------------------------------------------- | | `time` | Timers, intervals, and monotonic `Instant`. | `tokio` | `madsim` | Complete. | | `task` | Spawning and joining async tasks. | `tokio` | `madsim` | Complete. | | `runtime` | Runtime builder and handle. | `tokio` | `madsim` | Complete. | | `signal` | Process signals such as `ctrl_c` and `SIGTERM`. | `tokio` or OS | `madsim` for `ctrl_c`; a never-completing future for `terminate` | Partial. See [Signal handling](#signal-handling). | These re-exports live in `nautilus_common::live::dst`. DST-path call sites for `time`, `task`, and `runtime` import from this module. Normal builds resolve the imports to `tokio`; `simulation` with `cfg(madsim)` resolves them to `madsim`. The `sync`, `io`, `fs`, and `net` submodules, plus the `select!` macro, continue to use real `tokio`. The network crate supplies a separate [transport boundary](#simulated-http-and-websocket-transport) for plaintext HTTP and Tungstenite WebSocket connections; it does not replace Tokio inside dependencies. ### Layer 2: nondeterminism substitution Nondeterminism outside the aliased runtime needs explicit seams. #### Wall-clock time Wall-clock reads route through `nautilus_core::time::duration_since_unix_epoch`. Under simulation, the seam calls `madsim::time::TimeHandle::try_current()` to preserve Unix-epoch semantics for order and fill timestamps. Plain `#[rstest]` bodies run outside a madsim runtime. In that context, the seam falls back to `SystemTime::now()`, which uses the same real syscall as a normal build. Production paths under simulation run inside a madsim runtime and receive virtual time. #### Monotonic time Monotonic reads route through `nautilus_common::live::dst::time::Instant`. Normal builds resolve the type to `tokio::time::Instant`, preserving compatibility with `#[tokio::test(start_paused = true)]`. Simulation builds resolve it to `madsim::time::Instant`. #### Network-local monotonic time Network code routes monotonic reads through `nautilus_network::dst::time`. `nautilus-network` sits below `nautilus-common` in the dependency graph, so it provides a local re-export module with the same behavior. #### Iteration order Observable iteration order uses `IndexMap`, `IndexSet`, or an explicit sort instead of relying on `AHashMap` or `AHashSet`. `AHash` randomizes its hasher per process. Stable order is required when iteration controls event publication or the sequence of draws from a seeded `FillModel`. #### Select polling order Every production `tokio::select!` site on the DST path starts with `biased;`. Without it, an unintercepted RNG chooses the branch polling order. ## Determinism contract Under the conditions below, a run identified by `(seed, binary hash, configuration hash)` on the same platform produces bitwise-identical: - Scheduling order of async tasks. - Timer firings (virtual monotonic and virtual wall-clock). - RNG output from `madsim::rand`. - Delivery order on `tokio::sync` channels. ### Required conditions The contract holds **only when every row below is satisfied**: | Source of nondeterminism | Required condition | Failure when bypassed | | ------------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | Build selection | Enable the `simulation` feature and set `RUSTFLAGS="--cfg madsim"`. | Either setting alone falls back to real `tokio` without an error. The cfg also activates madsim's libc intercepts for `clock_gettime` and `getrandom`. | | `tokio::select!` | Put `biased;` first in every production block on the DST path. | An unintercepted RNG chooses the polling order. | | Monotonic time | Use `nautilus_common::live::dst::time` or `nautilus_network::dst::time`. | Direct `std::time::Instant::now` reads the host clock. | | Wall-clock time | Use `nautilus_core::time::duration_since_unix_epoch`. | Direct clock reads bypass virtual time. | | Randomness | Use `madsim::rand`. | `rand::thread_rng`, `rand::rng()`, `fastrand`, `getrandom`, and `OsRng` are not intercepted. | | Iteration order | Use `IndexMap`, `IndexSet`, or sort at the point of use. | Randomized hash iteration changes observable ordering. | | Local tasks | Gate out `tokio::task::LocalSet` under simulation. | `madsim` does not provide `LocalSet`; use `spawn_local` without it. | | Blocking tasks | Gate out or remove `tokio::task::spawn_blocking`. | The blocking call escapes the deterministic scheduler. | ## Static enforcement Static enforcement has two layers: | Layer | Enforces | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------- | | Clippy policy | `clippy.toml` and `[workspace.lints.clippy]` reject direct `getrandom::{fill,u32,u64}` calls and `tokio::task::LocalSet`. | | `check-dst-conventions` | The pre-commit hook applies path-aware and cfg-aware structural checks that Clippy cannot express cleanly. | The hook lives at `.pre-commit-hooks/check_dst_conventions.sh` and runs in the standard pre-commit suite and CI. Rules 1 to 4 and 6 scan the 17 in-scope workspace crates and the selected OKX files listed below. Rule 5 covers its two audited files. Rule 7 scans the nine crates on the madsim build path, those OKX files, and `crates/network/src/websocket/client.rs`. | Rule | Rejects | Scope or exception | | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | 1 | Raw `std::time::Instant::now()`, `SystemTime::now()`, `jiff::Timestamp::now()`, and `jiff::Zoned::now()` reads, including imported bare forms. | Allows the wall-clock seam, audited log or progress timing, and marked lines. | | 2 | Raw `rand::thread_rng`, `rand::rng()`, `fastrand::`, `getrandom::`, `OsRng`, and `Uuid::new_v4()` usage. | Allows cfg-gated and marked lines. | | 3 | Production `tokio::select!` blocks without `biased;` in the first three lines. | Excludes tests and marked lines. | | 4 | `std::thread::spawn`, `std::thread::Builder::new`, and `tokio::task::spawn_blocking`. | Allows test-only, non-madsim, and marked sites. | | 5 | `AHashMap` or `AHashSet` in the reconciliation manager and matching engine. | Covers the two audited files; the remaining file set stays outside this static rule until audited. | | 6 | Direct `tokio::net::TcpStream::connect` and `tokio::net::TcpListener::bind` calls. | Callers must use `nautilus_network::net`, which swaps to `turmoil::net` under the `turmoil` feature. | | 7 | Raw `tokio::{time,task,runtime,signal}` paths in production code on the madsim build path. | Allows the facade, process-wide real Tokio runtime, test infrastructure, cfg-gated sites, and marked lines. | Supported Madsim HTTP and WebSocket paths use `nautilus_network::dst::net`; it selects the simulated byte stream and re-exports `nautilus_network::net` in normal builds. The hook supports two exception forms: - An inline `// dst-ok` marker on a specific line, typically accompanied by a short reason (for example, log-only wall-clock timing that does not affect state). - A small file-level allowlist in the hook script itself for sites classified as leave-alone in the codebase audit (log timing in the cache module, log-record timestamping in the logging bridge and writer, progress reporting in the DeFi module). The hook excludes: - Files under `tests/`, `python/`, and `ffi/` directories. - Files named `tests.rs`, `*_test.rs`, or `*_tests.rs`. - Lines inside an inline `#[cfg(test)]` module. These paths are not part of the production DST contract. ### In-scope crates The transitive closure of `nautilus-live` contains 16 in-scope crates: - `analysis` - `common` - `core` - `cryptography` - `data` - `execution` - `indicators` - `live` - `model` - `network` - `persistence` - `portfolio` - `risk` - `serialization` - `system` - `trading` The hook also covers `backtest`, bringing the total to 17 crates. Adapter crates and infrastructure crates (Redis, Postgres) are out of scope unless an audited slice is listed here. Audited OKX DST-path production files route state-affecting clock reads and timers through the DST seams and sort reconnect and bulk-unsubscribe subscription commands. The static hook covers `book_sync.rs`, `common/task.rs`, `data.rs`, `execution.rs`, `http/client.rs`, `websocket/client.rs`, `websocket/dispatch.rs`, and `websocket/handler.rs` in `crates/adapters/okx/src`. These files also serve paths outside a proven runtime slice: static coverage alone does not establish their runtime eligibility. Focused Madsim tests in `crates/adapters/okx/tests/integration/dst.rs` cover subscribe-wire bytes for public WebSocket quotes, trades, and books, business WebSocket bars, and multi-instrument quote reconnect in topic order. Reconnect also clears quote and funding caches in `data.rs` so a new generation cannot reuse prior values. Complete request-to-wire-to-domain fresh-process comparison stays in the downstream DST harness. Other public channels, private data, and execution share the DST facades and convention gate but remain unproven runtime slices. ## Simulated HTTP and WebSocket transport With `simulation` and `cfg(madsim)`, `nautilus-network` executes plaintext HTTP/1.1 requests and Tungstenite WebSocket connections over Madsim byte streams. Production and simulated HTTP share request defaults, URL and query encoding, and response validation, including buffered body limits and a deadline covering the body read. The simulation branch in `crates/network/src/http/simulation.rs` uses Hyper for the HTTP/1.1 exchange and owns its Madsim connection task. WebSocket traffic uses the Tungstenite codec; `crates/network/src/dst.rs` routes its owned tasks and transport streams to Madsim. Normal builds use pooled Hyper HTTP connections and Tokio tasks. Configure controlled `http://` and `ws://` endpoints and select `TransportBackend::Tungstenite`. Simulation rejects HTTPS/WSS, explicit proxies, and Sockudo before opening a connection. `HttpRedirectPolicy::Follow` is the default: a redirect response with `Location` returns an error instead of following it. `Reject` returns the redirect response to the caller. Sockudo's internal timers require a Tokio runtime. Ambient HTTP proxy settings do not affect the simulated exchange. The model covers HTTP request bytes, WebSocket message payloads, and domain events. It does not promise reproducible WebSocket handshake keys or frame masks, TLS records, OS TCP scheduling, partial writes, socket options, or TCP half-close. In `crates/network/src/dst/stream.rs`, stream shutdown flushes bytes; dropping the stream closes the connection. Combining `simulation`, `cfg(madsim)`, and `turmoil` is a compile error; run the two simulators in separate builds. Simulation sorts request headers by name for reproducible HTTP bytes. This canonical order does not promise byte-for-byte equality with live request header ordering. Each HTTP request opens a fresh connection; connection pooling and keep-alive reuse are not modeled. Network tests cover transport behavior; they do not establish end-to-end conformance for an adapter, product, or channel. ## Network seed soaks [Turmoil](https://crates.io/crates/turmoil) simulates the network under a seeded scheduler. The `nautilus-network` transport tests use it to reach link and reconnect orderings outside the madsim runtime swap. ### Fixed-seed nightly tests The nightly suite runs reproducible scenarios for: - Initial connection. - Reconnection. - Network partitions. - Closing during reconnection. - Closing during backoff. - Repeated server drops with exact message-order assertions. ### Reconnect seed soak An ignored reconnect test sweeps Turmoil seeds until stopped or until the configured limit is reached. For each seed, the soak runs the Tungstenite WebSocket backend first. It then runs Sockudo when the `transport-sockudo` feature is enabled, giving both backends the same schedule search path. | Variable | Default | Effect | | ----------------------------------------- | --------- | --------------------------------------------------- | | `NAUTILUS_TURMOIL_SOAK_START` | `0` | Selects the first seed, allowing a sweep to resume. | | `NAUTILUS_TURMOIL_SOAK_COUNT` | Unbounded | Stops after the specified number of seeds. | | `NAUTILUS_TURMOIL_SOAK_PROGRESS_INTERVAL` | `100` | Logs progress after this many seeds per backend. | ### Run the soak Run the continuous soak with: ```bash scripts/soak-network-turmoil.sh ``` Run a bounded soak with: ```bash env NAUTILUS_TURMOIL_SOAK_COUNT=100 scripts/soak-network-turmoil.sh ``` Each seed enables random node order and link latency from 1 ms to 25 ms. The scenario repeatedly drops the server, cycles the client through reconnect states, and asserts exact application-message order. The soak does not enable Turmoil `fail_rate`. For TCP, packet loss without a retransmit model would overstate the client delivery contract in an order-preservation test. ### Platform coverage The Turmoil tests use a simulated network and are not gated to Linux, so the seed sweep also runs on macOS. Several real localhost socket and WebSocket unit tests use `target_os = "linux"` for CI stability. A macOS run therefore leaves that host TCP coverage untouched. Treat the complete network test set as covered only after a run on Linux CI or a Linux workstation. ## Implementation notes These sections map each deterministic seam to its source, purpose, and exceptions. ### Iteration-order seams Production sites use ordered collections or explicit sorting when iteration is observable on the DST path. #### Matching engine `crates/execution/src/matching_engine/mod.rs` uses eleven ordered collections: - `execution_bar_types` - `execution_bar_deltas` - `account_ids` - `cached_filled_qty` - `post_match_order_ids` - `bid_consumption` - `ask_consumption` - `queue_pending` - `queue_ahead_orders` - `queue_ahead_total` - `queue_excess` Removals from `queue_pending`, `queue_ahead_orders`, and `queue_ahead_total` use `.shift_remove()` so deleting an entry does not reorder the remainder. #### Reconciliation manager Rule 5 covers `crates/live/src/execution/manager.rs`. The `orders` and `fills` maps in `ReconciliationResult` use `IndexMap` in `crates/execution/src/reconciliation/types.rs`. #### Account balances The account trait returns `IndexMap` from: - `balances` - `balances_total` - `balances_free` - `balances_locked` - `starting_balances` Balance and margin storage on `BaseAccount` and `MarginAccount` also uses `IndexMap`. The `commissions` and `leverages` fields remain `AHashMap`. #### Position commissions `Position::commissions` in `crates/model/src/position.rs` uses `IndexMap`. Position snapshots consume the map through `.values()` in `crates/model/src/events/position/snapshot.rs`, making its order observable. #### Portfolio aggregation `crates/portfolio/src/portfolio.rs` stores `unrealized_pnls`, `realized_pnls`, and `net_positions` in `IndexMap`. `accumulate_mark_values` builds an `IndexMap`. #### Data engine `crates/data/src/engine/` uses ordered storage for `book_snapshot_counts`, `bar_aggregators`, and `BookSnapshotInfos`. Iterated removals use `.shift_remove()`. #### Execution engine `ExecutionEngine.clients` uses `IndexMap`. The `client_ids` and `venues` accumulators in `get_clients_for_orders()` use `IndexSet`. #### Backtest engine and exchange `BacktestEngine.venues` and `SimulatedExchange.matching_engines` preserve venue and instrument iteration order for: - Settlement. - Expiration. - Liquidation. - Seeded `FillModel` draws. The source locations are `crates/backtest/src/engine.rs` and `crates/backtest/src/exchange.rs`. #### Trading algorithm `strategy_event_handlers` in `crates/trading/src/algorithm/core.rs` uses `IndexMap` to drive ordered `msgbus::unsubscribe_*` fan-out. #### Analyzer `account_balances` and `account_balances_starting` in `crates/analysis/src/analyzer.rs` use `IndexMap`. #### Cache API `get_orders_for_ids` and `get_positions_for_ids` in `crates/common/src/cache/mod.rs` sort returned vectors by `client_order_id` and `position_id`. The underlying storage keeps `AHashSet` because it has set semantics. #### Instrument store `InstrumentStore.instruments` in `crates/common/src/providers.rs` uses `IndexMap` with the `ahash` hasher. The order is observable because these adapters publish one `DataEvent::Instrument` per entry from `get_all()` or `list_all()`: - Betfair. - Derive. - Polymarket. #### Order emulator `on_reset` in `crates/execution/src/order_emulator/emulator.rs` sorts three drained sets before ordered `msgbus::unsubscribe_*` fan-out: - `subscribed_quotes` - `subscribed_trades` - `subscribed_strategies` The quote and trade paths also advance the seeded `UUID4::new` draw sequence. Storage remains on `AHashSet`. #### WebSocket subscriptions `topics_from_map` in `crates/network/src/websocket/subscription.rs` sorts its returned vector to preserve reconnect replay order behind `all_topics()` while storage remains on `DashMap` with `AHashSet` values. #### Unordered collection limits `AHashMap` / `AHashSet` sites in the `nautilus-live` closure are lookup-only, behind concurrent shared-ownership wrappers (`Arc`, `AtomicMap`), or feed into commutative aggregation. `backtest` contains additional hash collections outside rule 5's two-file enforcement scope, including pre-run validation and result maps. Treat their iteration order as outside the static guarantee until each path is audited. ### Time seams Remaining `Instant::now` and `SystemTime::now` sites are tests, file-allowlisted locations, or marked exceptions: | Location | Use | Treatment | | ------------------------------------ | -------------------------------------------------------- | ---------------------------------------------------------- | | `crates/common/src/testing.rs` | `wait_until` and `wait_until_async` timers. | Inline `// dst-ok`: test controls use real time by design. | | `crates/execution/src/engine/mod.rs` | Initialization log timing in `load_cache`. | Inline `// dst-ok`: timing does not affect DST state. | | `crates/common/src/cache/mod.rs` | Timing in `check_integrity` and `audit_own_order_books`. | File allowlist. | | `crates/model/src/defi/reporting.rs` | Progress logging. | File allowlist. | | `crates/core/src/time.rs` | Wall-clock seam definition. | Explicit seam exception. | `jiff::Timestamp::now` and `jiff::Zoned::now` are hook-banned in the in-scope crates. The remaining timestamp call sites are the logging bridge and writer, scoped out under [Logging runs on real OS threads](#logging-runs-on-real-os-threads). `crates/core/src/datetime.rs::is_within_last_24_hours` routes through `nautilus_core::time::nanos_since_unix_epoch()` and compares in `u64` nanos directly. ### Randomness seams Production randomness on the DST path uses the following seams. #### UUID generation `crates/core/src/uuid.rs::UUID4::new()` uses `madsim::rand::thread_rng()` inside a madsim runtime under simulation. Normal builds and plain `#[rstest]` bodies outside a madsim runtime use `rand::rng()`. Production simulation paths run inside the runtime and consume seeded bytes. Order and event factories in `nautilus-common` and `nautilus-risk` reach this seam. #### Fill model randomness `crates/execution/src/models/fill.rs::default_std_rng()` follows the same runtime split. `ProbabilisticFillState::new()` calls it when no seed is provided. When a caller supplies a seed, `StdRng::seed_from_u64` is deterministic without the seam. #### Random matching-engine IDs The `use_random_ids` path in `crates/execution/src/matching_engine/ids_generator.rs` calls `nautilus_core::UUID4::new()` for position and venue order IDs. The default ID scheme, `{venue}-{raw_id}-{count}`, is deterministic without random bytes. #### Reconnect jitter `crates/network/src/backoff.rs` samples `madsim::rand::thread_rng()` inside the simulation runtime. Normal builds and calls outside a Madsim runtime use `rand::rng()`. Restarting the runtime with the same seed restarts the simulated jitter sequence. ### Tokio submodule split The common facade routes `time`, `task`, `runtime`, and `signal` through `madsim`. Direct uses of Tokio's `sync`, `io`, `fs`, and `net` submodules, plus `select!`, stay on real `tokio` under simulation. The network-local `dst::net` boundary separately selects Madsim byte streams for the supported HTTP and WebSocket paths. A global Tokio network replacement would also affect dependencies such as: - `tokio-tungstenite` - `tokio-rustls` - `hyper-util` - `hyper-rustls` The network-local boundary avoids that global replacement by supplying a stream to Hyper and Tungstenite. TLS remains outside its simulation scope. The in-scope direct uses are: | Location | Real Tokio surface | Boundary | | ------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------ | | `crates/network/src/net.rs` | `tokio::net::{TcpListener, TcpStream}` | Re-exports the normal transport behind the `crate::net` seam. | | `crates/network/src/socket/client.rs` | `tokio::io::{AsyncReadExt, AsyncWriteExt}` | Performs I/O on the selected transport. | | `crates/network/src/tls.rs` | `tokio::io::{AsyncRead, AsyncWrite}` | Defines TLS I/O bounds. | | `crates/network/src/socket/types.rs` | `tokio::io::{ReadHalf, WriteHalf}` | Splits `MaybeTlsStream`; `TcpStream` comes from `crate::net`. | These paths use real sockets even under simulation. The `tokio::sync` channel implementation also remains real, but madsim schedules its sender and receiver tasks. Channel delivery order therefore remains part of the deterministic scheduling contract. ### Raw thread escape rules Rule 4 of the hook bans raw thread spawning outside three escape cases: - `#[cfg(test)]` test modules. - `#[cfg(not(madsim))]` or `#[cfg(not(all(feature = "simulation", madsim)))]` production sites (for example, the logging writer thread). - An inline `// dst-ok` marker. `tokio::task::LocalSet` and `tokio::task::spawn_blocking` are not supported under `madsim`. The codebase audit found no production sites for either inside the in-scope crates; new sites must carry a cfg gate or `// dst-ok` marker. ### Logging tests under simulation The logging writer thread is cfg-gated out under simulation; under `cfg(madsim)` log events are dropped. Tests that initialize the file-logging writer would either hang or assert against an empty log file, so the affected submodules are gated out at the module boundary: - `crates/common/src/logging/logger.rs::tests::serial_tests`. - `crates/common/src/logging/macros.rs::tests`. `logger.rs::tests::sim_tests::test_init_under_madsim_skips_writer_thread_and_forces_bypass` runs under simulation and pins the gated behavior. ## Scope boundaries The contract is limited by the boundaries below. Each subsection identifies behavior that can vary between runs or remains outside the audited DST path. ### Python and FFI are not in DST scope DST runs under a native Rust test harness and does not start a Python interpreter. The contract excludes: - PyO3 bindings under `crates/*/src/python/`. - Rust FFI modules under `crates/core/src/ffi/` and `crates/model/src/ffi/`. - The Python package under `python/nautilus_trader/`. Code reachable only through these bindings is out of scope. Any Rust path reachable from the native DST harness must satisfy the contract, even when the same type is also exported through a binding. The `check-dst-conventions` hook encodes this policy by skipping `/python/` and `/ffi/` paths in the in-scope crates. Clock, RNG, and threading calls behind those paths do not apply to the contract. DST primarily covers the order lifecycle, reconciliation, matching, risk, and execution state machines in the Rust engine. User strategies are replayable only when written in Rust or driven through a Rust-native test harness. A Python strategy can vary its command stream by: - Calling `time.time()`. - Issuing arbitrary network requests. - Relying on OS thread scheduling. The Rust core processes that command stream according to its deterministic contract, but DST does not guarantee end-to-end replay from a Python entry point. ### Platform-scoped `madsim`'s libc overrides for `clock_gettime` and `getrandom` are platform-specific. The contract does not claim cross-platform bitwise reproducibility. A seed that reproduces a failure on Linux x86_64 may not reproduce it on macOS aarch64. ### Non-aliased dependencies escape silently A dependency escapes the simulator without an error when it reaches the OS through: - Direct `libc` calls. - A `std::net` bypass. - Unrouted randomness such as `fastrand` or `OsRng`. The in-scope crates have been audited. Adapter and infrastructure crates require separate audits before entering the DST path. ### Transport scope limits The following dependencies use real `tokio` internally: - `tokio-tungstenite` - `tokio-rustls` - `hyper-util` - `hyper-rustls` - `redis` - `sqlx` The [simulated HTTP and WebSocket transport](#simulated-http-and-websocket-transport) supplies Madsim streams to the supported plaintext paths. Raw socket clients, TLS, Redis, and SQL remain outside that boundary. Turmoil provides the separate network simulation described in [Network seed soaks](#network-seed-soaks). The following test modules drive real localhost sockets and are cfg-gated out under `all(feature = "simulation", madsim)`: - `crates/network/src/http/client.rs::tests` - `crates/network/src/http/tests.rs` - `crates/network/src/socket/client.rs::tests` - `crates/network/src/socket/client.rs::rust_tests` - `crates/network/src/websocket/client.rs::tests` - `crates/network/src/websocket/client.rs::rust_tests` - `crates/network/tests/integration/websocket_proxy.rs` Their production paths reach madsim time primitives through `dst::time::*`, which panic when called from a `#[tokio::test]` runtime. The retry modules in `crates/network/src/retry.rs` run under both runtimes. Their test attributes switch between `#[tokio::test(start_paused = true)]` and `#[madsim::test]`; time reads and sleeps use `crate::dst::time`; explicit virtual-time advances use a cfg-gated `advance_clock` function. The same test bodies therefore cover normal and simulation builds. ### Signal handling `nautilus_common::live::dst::signal` exposes routed `ctrl_c` and `terminate` re-exports. The run loop in `crates/live/src/node/mod.rs` uses them. Under `cfg(madsim)`, tests can inject node shutdown through `madsim::runtime::Handle::send_ctrl_c`. Adapter binary entry points still call `tokio::signal::ctrl_c` directly and remain out of scope. ### Logging runs on real OS threads The logging subsystem spawns a writer thread via `std::thread::Builder` and uses `std::sync::mpsc`. Under simulation, the thread is not spawned and log events are dropped. Log output is outside the determinism contract: the writer only writes, never reads or mutates simulation state. ### Adapters End-to-end adapter behavior remains outside the upstream simulation test scope. Depending on the adapter, unaudited paths contain: - Direct `jiff::Timestamp::now` or `jiff::Zoned::now` calls. - Direct `SystemTime::now` calls. - Unrouted RNG calls. - Raw transport clients. An adapter must be audited for these sites before the DST contract covers its behavior. ## Relationship to other testing layers DST complements existing testing; it does not replace any of it. | Layer | Covers | DST relationship | | ----------------------- | ---------------------------------------------------- | ----------------------------------------------- | | Unit tests | Pure logic, calculations, parsers, transformers. | Unchanged. | | Integration tests | Component interaction, I/O boundaries. | Unchanged. DST runs alongside, not in place of. | | Property-based tests | Invariants over input domains (parsers, roundtrips). | Unchanged. | | Acceptance tests | End-to-end backtest and live scenarios. | Unchanged. | | Deterministic sim (DST) | Async timing, scheduling, recovery correctness. | Adds seed-replayable exploration. | DST covers async concurrency and state-machine correctness. Representative failures include: - A shutdown message dropped under one task wakeup order. - A reconciliation event lost when iteration order changes. The other testing layers retain responsibility for their existing scopes. ## Status ### Runtime swap Layer 1 is implemented. `nautilus_common::live::dst` exposes routed re-exports for `time`, `task`, `runtime`, and `signal`. Production call sites for `time`, `task`, and `runtime` use the seam. Signal adoption remains partial; see [Signal handling](#signal-handling). ### Nondeterminism substitution Layer 2 is implemented across the 17 in-scope crates. Seams cover wall-clock time, monotonic time, randomness, and observable iteration order. [Implementation notes](#implementation-notes) lists the audited paths and remaining exceptions. ### Static enforcement status `check-dst-conventions` runs in pre-commit and CI. It covers the load-bearing structural conditions and permits reviewed per-line exceptions through `// dst-ok`. ### Runtime verification limit Verification covers seam tests and static checks. This repository does not compare complete adapter runs across fresh processes. ### Simulation smoke gate The nightly workflow and local pre-flight use the same DST targets: | Entry point | Relevant order | Purpose | | ------------------------------------- | --------------------------------------------------------- | -------------------------------------------------------- | | `.github/workflows/nightly-tests.yml` | `check-code-sim` > `cargo-test-sim` | Runs the nightly and manually dispatched DST smoke gate. | | `make pre-flight` | `check-code-sim` > `cargo-test-sim` > `cargo-test-extras` | Fails early on DST lint before the Rust test suites. | `check-code-sim` runs pinned stable Clippy with `--features simulation` and `cfg(madsim)` across `nautilus-common`, `nautilus-core`, `nautilus-event-store`, `nautilus-network`, `nautilus-execution`, and `nautilus-live`. A separate `--no-default-features` leg compiles and lints the OKX adapter without enabling OKX's default `high-precision` feature in the standard-precision core leg. `cargo-test-sim` uses three feature-coherent nextest invocations: | Precision | Packages | Features | Selection | | --------- | --------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------------------------------------------------------------- | | Standard | `nautilus-common`, `nautilus-core`, `nautilus-event-store`, `nautilus-network`, `nautilus-execution`, `nautilus-live` | `simulation` | All compatible common, event-store, network, and execution tests; focused live and core tests. | | Standard | `nautilus-okx` | `simulation` | Integration `dst` tests only, without OKX's default `high-precision` feature. | | High | `nautilus-common`, `nautilus-execution` | `simulation,high-precision` | All tests in both packages. | Nextest compiles the selected library and test targets, so the gate does not run a separate Cargo build. The invocations resolve each feature set once across their package sets. Together they exercise seam-routed `QuantityRaw` (`u64` / `u128`) and `PriceRaw` (`i64` / `i128`) paths at both fixed-point widths. #### Common tests The standard-precision run executes all simulation-compatible `nautilus-common` tests. Its feature graph propagates `nautilus-core/simulation`, selecting the `wall_clock_now` cfg branch throughout the suite. Plain `#[rstest]` bodies run outside a madsim runtime and use the seam's `SystemTime::now()` fallback. This is the same path madsim's libc shim takes outside a runtime. The `LiveClock` test module is cfg-gated out because its plain `#[rstest]` cases start `LiveTimer` tasks without a madsim runtime, and most wait for wall-clock progress. `live::dst::tests::test_dst_wall_clock_advances_with_virtual_time` runs under `#[madsim::test]` and asserts that `nanos_since_unix_epoch` advances with `madsim::time::sleep`. This pins virtual wall-clock behavior inside the runtime. #### Event store tests The standard-precision run executes all simulation-compatible `nautilus-event-store` tests. It exercises the synchronous event and marker writers under `cfg(madsim)`, including deterministic sequence ordering. Tests that depend on blocking OS threads retain native coverage and are gated out because those threads run outside madsim's scheduling control. The event-store smoke lane compiles and tests its existing simulation implementation without extending the static convention hook to event-store production code. #### Live startup reconciliation The focused `nautilus-live` regression runs under madsim. It verifies that a pending mass-status request reaches its configured timeout, reports the expected error, and cleans up the node without entering a real Tokio timer. #### Network tests The run executes all `nautilus-network` tests except transport-bound modules cfg-gated out at the source. Coverage includes virtual-time seam tests for sleep, timeout, and the rate limiter, plus the retry suites that exercise backoff timing. `crates/network/tests/simulation.rs` adds WebSocket reconnect, unsupported endpoint and Sockudo rejection, and jitter reset checks. Unit tests in `crates/network/src/http/simulation.rs` cover request bytes, response limits, body deadlines, cancellation, redirect policy, and HTTPS/proxy rejection. `#[madsim::test]` uses a varying seed by default and reports it on failure. Set `MADSIM_TEST_SEED` to replay a schedule. The `dst smoke (cfg madsim)` job in `.github/workflows/nightly-tests.yml` selects five consecutive seeds from `GITHUB_RUN_NUMBER * MADSIM_TEST_NUM`. The jitter runtime-reset test pins its own seed and complements the cross-seed network checks. #### Execution tests The run executes all `nautilus-execution` tests. These plain `#[rstest]` cases exercise cfg-gated branches in the matching engine, fill model, and execution engine without entering a madsim runtime. `default_std_rng()` therefore takes its host-RNG fallback in these tests. #### Core seam tests The focused `nautilus-core` selection pins `wall_clock_now` against virtual time. #### OKX adapter tests The standard-precision OKX leg runs the integration `dst` tests under `simulation` without the crate's default `high-precision` feature. Those `#[madsim::test]` cases cover public WebSocket quotes, trades, and books, business WebSocket bars, and multi-instrument quote reconnect order. #### Overall gate coverage `#[madsim::test]` cases in `nautilus-common`, `nautilus-core`, `nautilus-network`, `nautilus-live`, and `nautilus-okx` provide deterministic-scheduler coverage. The complete gate catches drift in the cfg-gated seams but does not verify end-to-end adapter determinism. ## Further reading - `.pre-commit-hooks/check_dst_conventions.sh` defines the seven enforcement rules in full and documents the `// dst-ok` marker convention. - [FoundationDB testing philosophy](https://apple.github.io/foundationdb/testing.html). - [TigerBeetle simulation testing blog posts](https://tigerbeetle.com/blog/). - [madsim repository](https://github.com/madsim-rs/madsim), the deterministic runtime. - [Turmoil repository](https://github.com/tokio-rs/turmoil), the deterministic network simulator. # Event Sourcing Source: https://nautilustrader.io/docs/latest/concepts/event_sourcing/ Event sourcing gives NautilusTrader a durable, ordered record of the messages that change engine state. The event store records those messages at the system boundary, then readers, replay tools, and verifiers use the same log to reconstruct what happened and to rebuild state. **The core philosophy**: - The event store is the durable authority for state-affecting history. - The cache is a write-through projection, not the source of truth. - Cache replay rebuilds state by applying captured history to cache-owned state. - Market data stays in the data catalog; the event store records the messages that affect state. - External I/O becomes replayable only when Nautilus captures it as commands, raw reports, or other state-affecting inputs. :::note Event-store capture, replay, verification, recovery, and retention planning have targeted test coverage, but the API surface is still evolving. Treat this page as the design contract, and the [`nautilus-event-store` README](https://github.com/nautechsystems/nautilus_trader/blob/master/crates/event_store/README.md) plus [docs.rs](https://docs.rs/nautilus-event-store) as the API reference. ::: ## Why event sourcing The cache answers "what is true now". The event store answers "how did Nautilus get here". It gives readers, replay tools, and verifiers a run-scoped history that does not require strategy logic, venue queries, or the live cache to explain past state. The event store provides Nautilus with a durable basis to: - Prove whether a sealed run is clean before replay or archive. - Inspect the exact command, report, and event sequence behind an order or component intent. - Rebuild cache state from captured history, including a snapshot anchor plus the run tail. - Trace an intent through the engine-side messages that followed from it. - Seal stale run files before the next run starts after a process exit or writer halt. ## Terms - **Run**: one kernel session for one instance, binary, and config. - **Entry**: one captured message plus replay metadata. - `seq`: the per-run sequence assigned by the writer and used as replay order. - **High-watermark**: the largest `seq` durably acknowledged by the backend. - **Snapshot anchor**: the high-watermark recorded with a cache snapshot. - **Headers**: correlation and causation metadata propagated with captured messages. ## What the store records The event store records state-affecting message bus traffic for one trading instance and one run. A run starts when the kernel starts and ends when the process stops cleanly or crashes. **Captured entries include**: - Execution commands such as submit, modify, and cancel. - Data subscription commands that define the actor or strategy observation window. - Fired time events and generated order, position, and account events. - Raw venue execution reports before reconciliation synthesizes derived events. - Reconciliation outputs produced from those raw reports. - Request and response messages, or their audit-relevant metadata, that cross the bus and affect state. - Run lifecycle entries such as `RunStarted` and `RunEnded`. Streamed market-data observations stay in the data catalog. The event store records the command stream, raw reports, generated events, and metadata needed to replay how the engine reacted to that world. Data responses are the exception: every response to an engine request is captured, including book, option-chain reference price, and custom-data responses. Only some of them, listed under [Cache replay](#cache-replay), carry a rule that applies them back to cache state; the rest are inspection records. ## Boundaries The event store is intentionally narrow: - It does not replace the data catalog. - It does not provide analytics or OLAP queries. - It does not aggregate multiple trader instances into a consensus log. - It does not yet define redaction, encryption-at-rest, or tamper evidence. ## Capture flow Capture happens at the message bus dispatch boundary, so the tap sees every state-affecting message before downstream handlers observe it. ```mermaid flowchart LR Producer["Engine, adapter, strategy, or component"] --> Bus["MessageBus publish/send"] Bus --> Tap["Capture tap"] Tap --> Adapter["BusCaptureAdapter"] Adapter --> Writer["EventStoreWriter"] Writer --> Backend["redb run file"] Bus --> Handlers["Downstream handlers"] Backend --> Reader["Reader, replay, verifier"] ``` Capture branches off the same dispatch that feeds downstream handlers, and readers only ever reach the durable backend. Capture is **asynchronous**, not an acceptance gate on dispatch. A successful capture enqueues the entry to the writer; the writer thread then assigns the next `seq`, commits a batch, and advances the high-watermark once the backend acknowledges durability. Readers scan sealed or running backends over a surface that exposes no append operations. The writer takes entries over a bounded channel. Backpressure never silently drops an accepted entry: a submit that stalls past the configured `halt_threshold` fires the halt signal instead. A backend commit failure is different, and does lose the queued batch: the writer fires the halt signal, discards the pending batch, and ends its loop, so those entries never become durable. :::warning Fail-stop does not interrupt the run. The tap logs the failure and the message still reaches its handlers, and once halted the tap stops recording, so the rest of that session runs uncaptured. No runtime component polls the halt signal to stop the trader; the recovery sweep on the next boot is what seals the run, as `CrashedRecovered` when its tail is clean. ::: Some messages legitimately cross more than one tap-visible boundary: the execution engine sends an order event to the portfolio endpoint and publishes the same event on its strategy topic, and trading commands hop from strategy to risk to execution. Duplicate dispatches of one message land within a single engine cycle, so the capture adapter deduplicates against a bounded window of recently captured message identities (event id, command id). Each logical message becomes one entry, and replay does not apply the same event twice. ## Lifecycle options `EventStoreConfig` is the serializable run policy. Process-local construction policy lives in `EventStoreLifecycleOptions`, which advanced callers pass through `EventStoreLifecycle::boot_with_options(...)`. By default the lifecycle opens `RedbBackend` and installs the default encoder and data-marker extractor registries. Lifecycle options replace any of the three: - An encoder registry, or a factory that builds one per run, applied before the bus tap starts capture. - A backend opener that returns any `EventStore` implementation for the new run. - A data-marker extractor registry factory for the configured marker classes. The backend opener is the simulation-safe path for memory capture. A DST harness or focused test can open `MemoryBackend` through the normal lifecycle, keep the same bus tap and writer semantics, and read the captured entries in-process after seal. Under `cfg(madsim)`, the writer commits each submit synchronously, so the captured `seq` order is deterministic. With a `MemoryBackend` opener, capture needs no `redb` run file. ## Entry model Each event-store entry is one captured message plus metadata: - `seq`: the per-run replay-order authority. - `ts_init`: the domain timestamp on the captured message. - `ts_publish`: the bus-accepted or writer-receive timestamp. - `topic`: the bus topic or logical endpoint. - `payload_type`: the encoded message type. - `payload`: the encoded message bytes. - `headers`: correlation and causation metadata. - `entry_hash`: the canonical hash over the entry content. `seq` orders replay. Timestamps help explain the run, but they do not override `seq`. Secondary indices cover lookup by `client_order_id` and `venue_order_id`. A `correlation_id` index can be added when a concrete inspection caller needs that lookup pattern; until then, correlation scans can walk the captured stream. ## Correlation model The target model uses three identity levels so readers can answer scope, lineage, and message identity questions. - `correlation_id`: the logical workflow or chain. - `causation_id`: the direct parent message that caused this message. - `command_id`, `event_id`, or `report_id`: the identity of this specific message. ```mermaid flowchart TD Command["SubmitOrder command_id"] --> Event["OrderAccepted event_id"] Event --> Fill["OrderFilled event_id"] Correlation["correlation_id"] --> Command Correlation --> Event Correlation --> Fill Command -. "causation_id" .-> Event Event -. "causation_id" .-> Fill ``` One `correlation_id` spans the whole workflow, while `causation_id` links each message to its direct parent. :::warning Header propagation is incomplete, so most captured entries carry empty headers today. The default encoder registry registers extractors for trading commands, data commands, and data responses, and those extractors forward whatever the message carries. Of those, only a data request (which contributes its `request_id`) and a data response (which carries a required `correlation_id`) yield a populated header in practice: in-tree trading-command producers construct their commands with `correlation_id` and `causation_id` unset, and order, position, and account events, execution reports, and time events have no extractor at all. Treat the diagram above as the design contract, not as a description of what a captured run contains. ::: Where headers are populated, this lets operators ask two common questions: - "Show everything in this workflow": filter or scan by `correlation_id`. - "Show why this event happened": walk `causation_id` back to the direct parent. ## Run files and manifests The default backend is `redb`. It stores one file per run under: ```text //.redb ``` Each run file contains: - Entries keyed by `seq`. - Secondary indices for order identifiers. - A manifest written at run start and sealed at run end. - An optional snapshot anchor for cache restore. The manifest records the run identity and reproducibility inputs: - Run identity: - `run_id` - `parent_run_id` - `instance_id` - Build identity: - `binary_hash` - `schema_version` - `crate_versions` - `feature_flags` - `adapter_versions` - Configuration identity: - `config_hash` - `registered_components` - `seed` - Lifecycle state: - `start_ts_init` - `end_ts_init` - `high_watermark` - `status` Run status is one of `Running`, `Ended`, `CrashedRecovered`, or `Quarantined`. ## Run lifecycle ```mermaid flowchart TD Start["RunStarted entry"] --> Running["Running manifest"] Running --> Capture["Capture state-affecting entries"] Capture --> Anchor["Record optional snapshot anchors"] Anchor --> Capture Capture --> RunEnded["RunEnded entry"] RunEnded --> Ended["Ended manifest"] ``` A run opens with `RunStarted` and closes with `RunEnded`; snapshot anchors are optional points recorded while the manifest stays `Running`. - `RunStarted` is the first entry of a fresh run. A repeated `open()` in the same process seals the current session before it starts a new run. - While the manifest is `Running`, the bus tap records state-affecting entries and cache snapshots can record anchors against the durable high-watermark. - A clean shutdown, kernel drop, or reset/rerun seal appends `RunEnded` and seals the manifest as `Ended`. - A fail-stopped (halted) session skips the in-process seal; the recovery sweep on the next boot owns it. The halt signal is scoped to the run that fired it: a later `open()` re-arms a fresh signal, so one halt does not poison subsequent runs in the same process. ## Recovery sealing A predecessor is an older run file for the same instance whose manifest still says `Running`. This means the previous process did not finish the normal lifecycle, or the writer halted before the manifest seal completed. ```mermaid flowchart TD Predecessor["Running predecessor"] --> Scan["Scan durable tail"] Scan --> Empty["No durable entries"] Empty --> Recovered["Seal as CrashedRecovered"] Scan --> TailEnded["Tail contains RunEnded"] TailEnded --> Ended["Seal as Ended"] Scan --> CleanTail["Clean tail without RunEnded"] CleanTail --> Recovered Scan --> BadTail["Hash, gap, or structural failure"] BadTail --> Quarantined["Seal as Quarantined"] Recovered --> Parent["Eligible parent_run_id"] Ended --> NoParent["No parent link"] Quarantined --> NoParent ``` Boot recovery scans each `Running` predecessor and chooses a final manifest status from the durable tail: | Durable tail | Sealed status | Eligible parent | | ----------------------------------------- | ------------------ | --------------- | | No entries | `CrashedRecovered` | Yes | | Clean, without `RunEnded` | `CrashedRecovered` | Yes | | Clean, ending in `RunEnded` | `Ended` | No | | Hash mismatch, gap, or structural failure | `Quarantined` | No | The sweep never leaves the trader unbootable because one run file is damaged. A hard-killed process (SIGKILL, OOM kill, power loss) leaves a file that redb refuses to open read-only; the listing falls back to a writable open, which performs redb's repair pass before recovery proceeds. A file that still cannot be opened, or that lacks a manifest, is skipped with a logged error and retried on the next boot, so recovery and retention continue over the healthy runs. Only `CrashedRecovered` predecessors become `parent_run_id`. A configured `replay_from_run_id` overrides a recovered parent after validation. The read-only verifier is separate: it can inspect a sealed run without mutating it and reports `quarantine=not-performed`. ## Replay inputs Replay follows one ordering rule: apply event-store entries in `seq` order. `ts_init` and `ts_publish` explain when messages happened, but `seq` is the durable replay order. The Rust replay-input API keeps planning separate from execution: - Event-store-only replay inputs return entries only. - Catalog-joined replay inputs add caller-selected catalog slices for context analysis. Catalog planners take explicit `CatalogSliceSelector` values and a read-only `ReplayCatalog`. Planning resolves catalog time bounds from the event-store scan unless the selector supplies explicit bounds, reports missing catalog slices, and preserves `seq` as the entry ordering authority. Loading returns `ReplayInputs`: event-store entries in `seq` order plus catalog records grouped under their selected slice. Rust callers can enable the off-by-default `persistence` feature and wrap a `ParquetDataCatalog` with `nautilus_event_store::ParquetReplayCatalog` to plan selected catalog files and filename-derived intervals. The bridge loads `quotes`, `trades`, and `bars` into typed `CatalogReplayRecord` values. :::note The persistence bridge is read-only: it uses catalog discovery and query APIs but **does not write to the catalog**. Unsupported catalog classes fail loading until replay adds a typed payload contract for that class. ::: These APIs **do not**: - Open live venue clients - Run strategies or actors - Re-run reconciliation - Delete files - Replay the clock registration/cancel lifecycle ## Cache replay Kernel-managed replay uses `EventStoreConfig::replay_from_run_id`. When set, the kernel restores cache state from the sealed run, records that run as the parent of the fresh child run, and skips live engines, clients, startup, and venue reconciliation. Quarantined runs are rejected. Replay also requires `load_state=true`: with it disabled the kernel logs an error and returns without restoring the cache or opening a child run. The cache replay loader is **state-only**. It restores the cache-owned snapshot, scans the event-store tail in `seq` order, decodes supported cache-affecting payloads, and applies them directly to `Cache`. Supported payloads include: - Synthesized account, order, and position events - Captured order lists - Complete data responses for instruments, quotes, trades, funding rates, and bars The loader **does not**: - Publish replayed entries to the live message bus - Run strategy or actor code - Query venues - Run reconciliation - Derive identifiers again - Re-arm clocks Fired `TimeEvent`s and raw venue reports are inspection records on this path; replay applies the synthesized order, position, and account events captured later in the run. ## Data marker sidecar :::note The marker sidecar is opt-in via `EventStoreConfig.data_markers` and stays off by default. ::: Exact data delivery order is not inferred from catalog timestamps. The marker sidecar records data observed at the message-bus dispatch boundary, in a file beside the event-store run at `//.markers.redb`, without writing full market-data payloads into `EventStoreEntry` rows. The sidecar supports one audit claim: when marker capture is enabled, Nautilus observed data delivery in `marker_seq` order at the bus boundary for that run, and each marker carries enough identity to join back to candidate catalog rows. It cannot: - Prove that catalog timestamps alone define bus order. - Reconstruct a data point when the catalog row is absent or changed. - Prove venue send order before Nautilus observed the message. - Say anything about runs where marker capture was disabled. - Guarantee that every observed data message produced a marker. The sidecar trades completeness for isolation from the trading path, so it does not inherit the entry writer's backpressure contract. A marker submit that finds the bounded channel full drops the marker rather than stalling the caller or halting the run, and folds its sequence into a gap record: `Overflow` when a later submit flushes it, or `WriterClosed` when the writer closes while the dropped range is still pending. Markers do not consume event-store `seq` values and do not create gaps in the entry table. Each marker has its own monotonically increasing `marker_seq` plus `event_seq_before`, the largest event-store `seq` assigned before the marker was observed. A sealed-run analyzer can derive the next event-store entry after a marker from `event_seq_before + 1`; markers that share the same `event_seq_before` are ordered by `marker_seq`. Event-store `seq` remains the replay-order authority for state-affecting entries. The sidecar has two marker kinds: - **Cursor snapshots** (`DataCursorSnapshot`): the default capture mode. Each snapshot records `marker_seq`, `event_seq_before`, `ts_init`, and the `StreamCursor` entries that advanced since the previous snapshot. A `StreamCursor` carries the stream `slot`, the highest `ts_init` seen in that slot (`ts_init_hi`), and the record `count`. A `StreamDictEntry` maps each `slot` to its `data_cls` (`BookDeltas`, `BookDepth10`, `Quote`, `Trade`, `Bar`) and instrument `identifier`. - **High-fidelity markers** (`HiFiMarker`): opt-in per instrument via `DataMarkerConfig.high_fidelity`. Each records `marker_seq`, `event_seq_before`, `slot`, `ts_event`, `ts_init`, `same_ts_ordinal`, and a 32-byte `record_fingerprint` over the canonical typed row fields. `same_ts_ordinal` and `record_fingerprint` disambiguate duplicate same-timestamp data without storing prices, quantities, sizes, or MessagePack payloads. If two catalog rows are byte-identical for the same key and timestamp, the sidecar can prove that Nautilus observed two deliveries in a specific marker order; it cannot name a unique physical catalog row after catalog compaction rewrites row order. Marker verification proves that the `marker_seq` sequence is fully accounted for, counting recorded gaps as coverage. Read the gap records to find what was dropped. The stable contract is the marker schema, opt-in capture and reader primitives, marker sequence verification, and catalog join rules. Analysis tools can build on that contract to select windows, interpret venue-specific data, rank or cluster markers, present reports, and package run bundles. With marker capture disabled, no data marker writer is installed. Cache replay and live restart do not read this sidecar: snapshot-tail replay still applies event-store entries in `seq` order, and live restart still boots from cache-owned state plus the event-store parent link. ## Snapshot-anchored recovery Cache snapshots are owned by the cache. The event store stores only the snapshot anchor: the high-watermark at snapshot time, an opaque cache-owned `blob_ref` naming the snapshot, and the cache-owned `content_hash` for that blob. ```mermaid sequenceDiagram participant Cache participant Store as Event store participant Replay Cache->>Store: Record snapshot anchor at high-watermark N Replay->>Store: Read manifest and latest anchor Replay->>Cache: Load snapshot blob from anchor Replay->>Store: Scan entries with seq > N Replay->>Replay: Apply tail in seq order ``` Recovery loads the snapshot the anchor names, then applies only the entries after the anchor's high-watermark. Recovery cases are ordered by how far the message progressed: - Before enqueue: the message never reached the writer, so producer retry policy applies. - After enqueue, before commit: the in-flight batch is not durable, so the high-watermark does not advance. - After commit, before snapshot anchor: recovery loads the prior snapshot and replays the tail. - After snapshot anchor: recovery loads the latest snapshot and replays entries after the anchor. :::info Live restart still uses snapshot-plus-reconcile. Event-store recovery becomes the live restart path only after capture coverage and replay rules cover every state-affecting path. ::: Replay correctness depends on four checks: - Entries are addressed by immutable `seq` values. - Writes reject out-of-order commits. - Readers detect gaps inside the high-watermark. - Snapshot replay plans reject anchors that point past the durable high-watermark. ## Retention planning Retention uses whole run files as the reclaim unit. The event store exposes a non-destructive planner that lists sealed run manifests, inspects their latest snapshot-anchor status, and returns candidate run files for a later supervisor or operator process to reclaim. The planner supports three modes: - `Full`: keep every sealed run and return no reclaim candidates. - `Bounded { keep_last }`: keep the newest sealed runs and also keep at least one known-good restore point. - `SnapshotAnchored`: reclaim only sealed runs older than the newest known-good restore point. A known-good restore point is a sealed, non-`Quarantined` run with a valid snapshot anchor whose high-watermark does not exceed the run's durable high-watermark. The planner compares against the last entry actually on disk rather than the manifest's recorded value, so a tail-trimmed run cannot pose as a restore point. `Running` runs are never listed as sealed runs or selected as reclaim candidates. Missing, corrupt, or invalid snapshot anchors do not count as restore points, so the planner returns no candidates when it cannot prove that at least one structurally valid restore point remains. The check stops at the anchor: the planner never loads the snapshot blob, so it cannot rule out a restore that fails on a missing or altered blob. ## Integrity and verification Every entry carries a canonical hash over its full content. Readers and verifiers recompute the hash and report mismatches. The verifier also checks manifest/high-watermark status, validates secondary indices against the entry table, and reports snapshot anchors that fail to decode or point past the durable high-watermark. :::warning A `clean` verdict proves structural integrity, not restorability or capture completeness: - The verifier checks the snapshot anchor but never loads or hashes the blob it names, so a run whose blob is missing or altered verifies clean and fails at restore. The retention planner picks restore points on the same anchor-only evidence. - Marker verification counts recorded gaps as coverage, so a run that dropped markers under backpressure verifies clean. - A run that fail-stopped mid-session verifies clean over what it did capture, and says nothing about the messages that followed the halt. ::: Run verification is process-isolated. This matters because some corrupted `redb` files can panic on open or first read, and release builds use `panic = "abort"`. The verifier runs the scan in a worker subprocess so a bad file aborts the worker, not the caller. Verify a sealed run file: ```bash cargo run -p nautilus-event-store --bin verify -- ./event_store/trader-001/1700000000-cafe0001.redb ``` Clean output looks like: ```text clean run_id=1700000000-cafe0001 status=Ended high_watermark=3 entries_scanned=3 markers=absent ``` Corrupt output includes `quarantine=not-performed`: ```text corrupt run_id=1700000000-cafe0001 status=Ended high_watermark=3 entries_scanned=3 findings=1 marker_findings=0 markers=absent quarantine=not-performed - hash mismatch at seq 2 ``` The `markers=` field reports the sidecar scan. It reads `absent` when no `.markers.redb` sits beside the run file, `clean` or `corrupt` with the scanned snapshot, high-fidelity, gap, and dictionary counts when the sidecar was read, and `error` when a sidecar is present but cannot be opened or scanned. Exit codes: - `0`: the run is clean. - `1`: the run has corrupt findings, or the worker aborted or timed out. - `2`: the verifier could not open or run against the requested file. Increase the worker timeout for a large sealed run: ```bash env NAUTILUS_EVENT_STORE_VERIFY_TIMEOUT_SECS=120 \ cargo run -p nautilus-event-store --bin verify -- ./event_store/trader-001/1700000000-cafe0001.redb ``` Read a sealed run from Rust: ```rust use nautilus_event_store::{EventStoreReader, RedbBackend, ScanDirection}; fn inspect_run() -> Result<(), Box> { let backend = RedbBackend::open_sealed_file("./event_store/trader-001/1700000000-cafe0001.redb")?; let reader = EventStoreReader::new(backend); let high_watermark = reader.high_watermark()?; for entry in reader.scan_range(1, high_watermark, ScanDirection::Forward) { let entry = entry?; println!("{} {}", entry.seq, entry.topic); } Ok(()) } ``` :::note The verifier reports corruption but does not mutate run files. Quarantine is an operator or supervisor policy. ::: ## Verification coverage The event-store test suite pins the load-bearing correctness guarantees for the current alpha surface: - The default encoder registry covers the audited state-affecting capture surface. - Fired `TimeEvent`s hit the installed event-store tap through `TimeEventHandler::run`. - The writer halts under bounded backpressure instead of dropping accepted entries. - Entry hash verification detects byte-level payload corruption. - Process-isolated verification reports truncated or zero-tailed run files as corrupt. - Cache replay reconstructs the same observed account, order, and position state as a live cache for generated captured event streams. - The same order event dispatched across multiple bus boundaries is captured once. - Snapshot anchors that fail to decode or point past the durable high-watermark surface as verifier findings instead of verifying clean. - Catalog-joined replay input planning covers selected slices, missing slices, time bounds, and event-store `seq` ordering. - Crash recovery seals `Running` predecessors as `Ended`, `CrashedRecovered`, or `Quarantined` based on the durable tail, and only `CrashedRecovered` runs become parents. - Boot recovery repairs hard-crashed run files and skips unreadable ones instead of failing the sweep. ## Relationship to DST The event store and [deterministic simulation testing](dst.md) (DST) solve different parts of replay. - The event store supplies the captured input history. - DST controls scheduling, time, seeded randomness, and other in-scope nondeterminism. Together they let a run reproduce engine behavior inside the deterministic simulation scope. The manifest records the inputs that identify such a run alongside the captured log itself: `seed`, `binary_hash`, `config_hash`, and `schema_version`. Under `cfg(madsim)`, the writer commits synchronously instead of spawning its writer thread. When a simulation harness supplies a `MemoryBackend` opener through lifecycle options, capture stays in-process and does not require `redb` files. Redb remains the default durable backend outside that advanced options path. Adapter network I/O remains outside bit-identical replay unless Nautilus captures the relevant raw inputs and routes them through deterministic interfaces. # Greeks Source: https://nautilustrader.io/docs/latest/concepts/greeks/ Nautilus provides two paths for working with option Greeks, which measure how option prices respond to changes in market variables: 1. **Venue-provided Greeks**: real-time Greeks streamed from supported venues through the `OptionGreeks` data type and the option chain aggregation system. 1. **Local Greeks calculator**: `GreeksCalculator` computes Black-Scholes Greeks from cached market data, with support for portfolio aggregation, shock scenarios, and beta weighting. Use either path independently or combine them. Venue-provided Greeks arrive through the data subscription system and require no local computation. The local calculator covers venues that do not stream Greeks, backtesting, and custom adjustments such as shocks, beta weighting, and percent Greeks. ## Venue-provided Greeks ### OptionGreeks The `OptionGreeks` type represents venue-provided sensitivities for a single option contract. It is a Rust-native type exposed to Python through PyO3. | Field | Type | Description | | ------------------ | ------------------ | --------------------------------------------------- | | `instrument_id` | `InstrumentId` | The option contract these Greeks apply to. | | `convention` | `GreeksConvention` | Numeraire convention for the Greeks. | | `delta` | `float` | Rate of change of option price per unit underlying. | | `gamma` | `float` | Rate of change of delta per unit underlying. | | `vega` | `float` | Venue-reported vega. | | `theta` | `float` | Venue-reported theta. | | `rho` | `float` | Venue-reported rho; defaults to zero. | | `mark_iv` | `float` or None | Mark implied volatility. | | `bid_iv` | `float` or None | Bid implied volatility. | | `ask_iv` | `float` or None | Ask implied volatility. | | `underlying_price` | `float` or None | Underlying price at time of calculation. | | `open_interest` | `float` or None | Open interest for the contract. | | `ts_event` | `int` | UNIX timestamp (nanoseconds) of the event. | | `ts_init` | `int` | UNIX timestamp (nanoseconds) when initialized. | Subscribe from an actor or strategy: ```python self.subscribe_option_greeks(instrument_id, client_id=ClientId("DERIBIT")) ``` Handle updates: ```python def on_option_greeks(self, greeks: OptionGreeks) -> None: self.log.info(f"delta={greeks.delta:.4f} gamma={greeks.gamma:.6f}") ``` See the [Options](options.md) guide for the full subscription API, including option chain aggregation, strike range filtering, and snapshot modes. ### Persistence and replay `OptionGreeks` is a native member of the `Data` enum, so it persists to the data catalog and replays in backtests as built-in market data rather than custom data. Use the type-specific catalog methods to write and query it: ```python catalog.write_option_greeks(greeks) # greeks: list[OptionGreeks] greeks = catalog.query_option_greeks() ``` During replay, persisted Greeks reach a subscribed actor or strategy through the same `on_option_greeks` handler used for live data. They also feed option-chain aggregation. When a strategy subscribes to an `OptionChainSlice`, the backtest data engine joins replayed `OptionGreeks` with replayed `QuoteTick` BBO updates for each option instrument. The `underlying_price` field seeds ATM selection, and `delta` supports delta-based strike selection through `StrikeRange.delta(target, tolerance)`. ### Core schema versus custom data The native `OptionGreeks` fields form the core schema: the five standard Greeks (`delta`, `gamma`, `vega`, `theta`, and `rho`) plus implied volatility, underlying price, open interest, and convention. These field names are stable. No single schema covers every Greeks use case. Put venue-specific or model-specific values such as `vanna`, `volga`, `charm`, calibration inputs, or surface metadata in [custom data](custom_data.md), not the native type. Optional venue fields are nullable. `convention` is non-nullable and defaults to `GreeksConvention.BLACK_SCHOLES` in Python. ### Underlying Rust types The core Rust implementation spans `crates/model/src/data/greeks.rs` and `crates/model/src/data/option_chain.rs`: - `OptionGreekValues`: a plain struct with `delta`, `gamma`, `vega`, `theta`, and `rho` fields. Implements `Add` and `Mul` for aggregation. - `OptionGreeks`: wraps `OptionGreekValues` with `instrument_id`, `convention`, implied volatility fields, and timestamps. Implements `Deref` so Rust callers can access Greek fields directly. - `HasGreeks` trait: provides a `greeks()` method returning `OptionGreekValues`. Implemented by `OptionGreeks`, `GreeksData`, `PortfolioGreeks`, and `BlackScholesGreeksResult`. ### Black-Scholes functions Low-level pricing functions from `crates/model/src/data/greeks.rs` are also exposed to Python: ```python from nautilus_trader.model import ( black_scholes_greeks, imply_vol, imply_vol_and_greeks, refine_vol_and_greeks, ) # Compute Greeks given known volatility result = black_scholes_greeks(s=100.0, r=0.05, b=0.0, vol=0.20, is_call=True, k=100.0, t=0.25) # result.delta, result.gamma, result.vega, result.theta, result.price, result.vol # Imply volatility from market price, then compute Greeks result = imply_vol_and_greeks(s=100.0, r=0.05, b=0.0, is_call=True, k=100.0, t=0.25, price=5.0) # Refine volatility from a starting estimate with one Halley iteration result = refine_vol_and_greeks( s=100.0, r=0.05, b=0.0, is_call=True, k=100.0, t=0.25, target_price=5.0, initial_vol=0.18 ) ``` `refine_vol_and_greeks()` performs **one refinement step**, not a full convergence loop. Use it with a good starting estimate; use `imply_vol_and_greeks()` when a full implied-volatility solve is needed. The `BlackScholesGreeksResult` returned by these functions contains: `price`, `vol`, `delta`, `gamma`, `vega`, `theta`, and `itm_prob`. Conventions: - Vega is scaled by 0.01 (sensitivity to a 1 percentage point vol change). - Theta is scaled by 1/365.25 (daily decay). - American-style options are priced as European for Greeks computation. ## Local Greeks calculator ### GreeksCalculator `GreeksCalculator` computes Black-Scholes Greeks from cached market data. It is exposed from `nautilus_trader.common`, uses the cache and clock, and is accessible from actors and strategies. ```python from nautilus_trader.common import GreeksCalculator # Typically created in on_start() calculator = GreeksCalculator(cache=self.cache, clock=self.clock) ``` #### Instrument Greeks Compute Greeks for a single instrument (option or underlying) with quantity of 1: ```python greeks = calculator.instrument_greeks( instrument_id=option_id, flat_interest_rate=0.0425, # used if no yield curve in cache ) # Returns GreeksData or None while market data is warming up. ``` For option instruments, the calculator performs these steps: 1. Look up the instrument and its underlying in the cache. 1. Retrieve prices from the cache. Standard instruments prefer `MID` and fall back to `LAST`; true index instruments prefer the cached index price. 1. Look up yield curves from the cache, falling back to `flat_interest_rate`. 1. Imply volatility from the market price with `imply_vol_and_greeks`. 1. Return a `GreeksData` object with the computed values. Missing prices return `None`, which lets strategies treat warm-up as a normal no-op path. Setup errors such as a missing instrument definition raise a Python exception instead. For non-option instruments such as futures and equities, the calculator returns `GreeksData` with `delta=1` or beta-weighted delta and zero gamma, vega, theta, and rho. Option-specific fields retain their default values. #### Shock scenarios Apply hypothetical changes to spot, volatility, or time: ```python greeks = calculator.instrument_greeks( instrument_id=option_id, spot_shock=10.0, # +10 points on underlying vol_shock=0.02, # +2 percentage points of volatility time_to_expiry_shock=1 / 365.25, # roll forward one calendar day ) ``` #### Volatility update Refine implied volatility from a cached starting point: ```python greeks = calculator.instrument_greeks( instrument_id=option_id, update_vol=True, # use cached vol as starting point cache_greeks=True, # store result for next iteration ) ``` With cached Greeks, `update_vol=True` uses the single-iteration refiner described above. If the cache has no prior Greeks for the instrument, the calculator performs a full implied-volatility solve. #### Beta-weighted Greeks Express delta and gamma in terms of an index: ```python greeks = calculator.instrument_greeks( instrument_id=option_id, index_instrument_id=InstrumentId.from_str("SPX.CBOE"), beta_weights={underlying_id: 1.15}, percent_greeks=True, ) ``` #### Time-weighted vega Normalize vega across different expirations: ```python greeks = calculator.instrument_greeks( instrument_id=option_id, vega_time_weight_base=30, # normalize to 30-day vega ) ``` #### Portfolio Greeks Aggregate Greeks across all open positions matching filter criteria: ```python portfolio = calculator.portfolio_greeks( underlyings=["AAPL", "MSFT"], venue=Venue("CBOE"), strategy_id=StrategyId("DELTA_HEDGE-001"), flat_interest_rate=0.0425, index_instrument_id=InstrumentId.from_str("SPX.CBOE"), beta_weights=beta_dict, percent_greeks=True, ) # Returns PortfolioGreeks. ``` Filters: - `underlyings`: list of symbol prefixes. For example, `["AAPL"]` matches AAPL stock and all AAPL options. - `venue`: restrict to a single venue. - `instrument_id`: restrict to a single instrument. - `strategy_id`: restrict to a single strategy. - `side`: filter by position side, such as `LONG` or `SHORT`. - `greeks_filter`: callable that receives per-position `GreeksData` after `pnl`, `price`, and the Greek values are scaled by signed position quantity; return `True` to include it. ### GreeksData `GreeksData` carries the context of a single instrument's Greeks computation and is exposed from `nautilus_trader.model`. Passing `cache_greeks=True` stores the result in the cache. The Rust `GreeksCalculator` can also publish it to the `data.GreeksData.instrument_id={symbol}` topic; the Python surface does not expose that flag. | Field | Type | Description | | ------------------ | -------------- | ------------------------------------------------------------------- | | `ts_init` | `int` | Initialization timestamp in nanoseconds. | | `ts_event` | `int` | Event timestamp in nanoseconds. | | `instrument_id` | `InstrumentId` | Instrument for the calculation. | | `is_call` | `bool` | `True` for a call or non-option result; `False` for a put. | | `strike` | `float` | Strike price. | | `expiry` | `int` | Expiry date as a `YYYYMMDD` integer. | | `expiry_in_days` | `int` | Days to expiry. | | `expiry_in_years` | `float` | Years to expiry (`expiry_in_days / 365.25`). | | `multiplier` | `float` | Contract multiplier. | | `quantity` | `float` | Quantity, set to 1 by `instrument_greeks()`. | | `underlying_price` | `float` | Underlying price used in the calculation. | | `interest_rate` | `float` | Interest rate used in the calculation. | | `cost_of_carry` | `float` | Cost of carry (`r - dividend yield` when supplied; otherwise zero). | | `vol` | `float` | Implied volatility. | | `pnl` | `float` | PnL relative to the position entry, when a position is provided. | | `price` | `float` | Option model price; non-option position PnL when supplied. | | `delta` | `float` | Delta. | | `gamma` | `float` | Gamma. | | `vega` | `float` | Vega per one percentage point of volatility. | | `theta` | `float` | Daily theta. | | `rho` | `float` | Rho, set to zero by the local calculator. | | `itm_prob` | `float` | In-the-money probability. | Internally, `portfolio_greeks()` multiplies `pnl`, `price`, and the Greek values by each position's signed quantity before adding them to the portfolio result. The intermediate `quantity` field remains `1` and is not part of `PortfolioGreeks`. The calculation **does not apply** the `multiplier` field, and the public Python types do not expose arithmetic operators for this aggregation. Rust callers can apply the same scaling with `quantity * &greeks_data`, which returns `GreeksData` with scaled `pnl`, `price`, and Greek values. ### PortfolioGreeks `PortfolioGreeks` is the aggregated result from `portfolio_greeks()`: The Rust type implements `Add` to combine portfolio results. The Python type does not expose this operator. | Field | Type | Description | | ---------- | ------- | ---------------------------------------------------- | | `ts_init` | `int` | Initialization timestamp in nanoseconds. | | `ts_event` | `int` | Event timestamp in nanoseconds. | | `pnl` | `float` | Aggregate PnL after signed-quantity scaling. | | `price` | `float` | Aggregate model value after signed-quantity scaling. | | `delta` | `float` | Portfolio delta. | | `gamma` | `float` | Portfolio gamma. | | `vega` | `float` | Portfolio vega. | | `theta` | `float` | Portfolio theta. | | `rho` | `float` | Portfolio rho, zero for local calculator results. | ### Yield curves The Python API does not expose the Rust `YieldCurveData` type. Pass `flat_interest_rate` and `flat_dividend_yield` to `GreeksCalculator` methods when Python calculations need rates that differ from the defaults. Rust callers can use `YieldCurveData` for interpolated interest rate or dividend yield curves. ## Choosing between the two paths | Criterion | Venue-provided (`OptionGreeks`) | Local calculator (`GreeksCalculator`) | | --------------------- | ----------------------------------------------------- | --------------------------------------------------------- | | Computation | Done by the venue or broker | Local Black-Scholes | | Latency | Arrives with market data | Computed on demand | | Venues | Bybit, Deribit, Derive, Interactive Brokers, and OKX | Any cached option with required prices | | Shock scenarios | Not supported | Spot, vol, and time shocks | | Portfolio aggregation | Manual, such as iterating an `OptionChainSlice` | Built-in via `portfolio_greeks()` | | Beta weighting | Not supported | Built-in | | Backtest support | Via recorded `OptionGreeks` data | From cached prices at any point in time | | Values | delta, gamma, vega, theta, rho, IV, and open interest | delta, gamma, vega, theta, itm_prob, and vol; rho is zero | | Data type | `OptionGreeks` | `GreeksData` and `PortfolioGreeks` | ## Greek definitions These terms appear across both paths. The local Black-Scholes functions scale vega and theta as described above. `OptionGreeks` retains the values reported by each venue or broker and records their `convention`. | Greek | Field | Definition | | -------- | ---------- | ----------------------------------------------------------------------------------------------------------------- | | Delta | `delta` | First derivative of option price with respect to underlying price (`dV/dS`). | | Gamma | `gamma` | Second derivative of option price with respect to underlying price (`d²V/dS²`). | | Vega | `vega` | Sensitivity to a change in implied volatility (`dV/dVol`). | | Theta | `theta` | Sensitivity to the passage of time (`dV/dt`). | | Rho | `rho` | Sensitivity to a change in the risk-free interest rate (`dV/dr`). | | ITM prob | `itm_prob` | Probability that the option finishes in the money: `P(ϕS_T > ϕK)`, where `ϕ = 1` for calls and `ϕ = -1` for puts. | ## Examples Complete working examples are available in the repository: - `examples/live/bybit/bybit_option_greeks.py`: subscribe to Bybit venue-provided Greeks. - `examples/live/deribit/deribit_option_greeks.py`: subscribe to Deribit venue-provided Greeks. - `examples/live/okx/okx_option_greeks.py`: subscribe to OKX venue-provided Greeks. ## Related guides - [Options](options.md): option instruments, chain subscriptions, and strike filtering. - [Data](data/): built-in data types, custom data, and the subscription model. - [Actors](actors.md): subscription and handler reference. - [Strategies](strategies.md): strategy implementation and handler methods. # Concepts Source: https://nautilustrader.io/docs/latest/concepts/ These guides explain the core components, architecture, and design of NautilusTrader. ## Foundations } /> } /> } /> } /> ## Domain model } /> } /> } /> } /> } /> } /> ## Data } /> } /> } /> } /> ## Execution and portfolio } /> } /> } /> } /> } /> } /> ## Components and runtime } /> } /> } /> } /> } /> } /> ## Running systems } /> } /> } /> } /> } /> } /> } /> } /> :::note If there are discrepancies between these guides and the API reference, the API reference is correct. ::: # Live Trading Source: https://nautilustrader.io/docs/latest/concepts/live/ The same strategy and execution-algorithm code can run across backtest and live environments. Live execution also introduces venue, transport, timing, persistence, external-activity, and reconciliation behavior that a simulation may not reproduce. :::warning **Live trading involves real financial risk.** Before deploying to production, understand system configuration, node operations, execution reconciliation, and the differences between backtesting and live trading. ::: ## Backtest and live differences Backtests advance a controlled clock from historical data and execute orders on simulated venues. A live node shares strategy and execution-algorithm code with a backtest, but coordinates with systems outside the process boundary. - **Venue**: Venue rules and adapter capabilities determine which order types, instructions, and events are available. See [Adapters](adapters.md). - **Transport**: A network failure can leave an order command outcome unknown. See [Command outcomes](execution/policies.md#command-outcomes). - **Timing**: Independent inputs can interleave, and the runner does not define one global FIFO order. See [Dispatch priority](#dispatch-priority-and-overload-behavior). - **Persistence**: Built-in cache backends process writes independently of venue transport, and event-store capture does not gate dispatch on durable commit. See [Persistence before transport](execution/policies.md#persistence-before-transport). - **External activity**: Venue reports can include orders created outside the node. See [External order creation](execution/reconciliation.md#external-order-creation). - **Reconciliation**: Startup and runtime checks align retained local state with venue reports. See [Execution reconciliation](execution/reconciliation.md). ## Live node lifecycle Rust `LiveNode::run()` prepares cached and venue state before starting trader components, then owns the event loop and coordinated shutdown. ```mermaid flowchart TD Build[Configure and build LiveNode] --> Cache[Restore cached state when configured] Cache --> Data[Connect data clients and cache instruments] Data --> Exec[Connect execution clients] Exec --> Recon{Startup reconciliation enabled?} Recon -->|Yes| Align[Fetch venue reports and align state] Recon -->|No| Trader[Start trader components] Align --> Trader Trader --> Run[Run event loop and periodic checks] Run -->|Stop or shutdown request| Stop[Stop trader and process residual events] Stop --> Final[Disconnect clients and finalize] ``` Live node lifecycle: instruments and execution state are prepared before strategies start trading. Cache restoration runs when a backing database is attached and cache loading is enabled. Connection, reconciliation, or trader startup failures abort startup and follow the coordinated cleanup path. ## Hosted event loops Use `run_async()` from Python to run a node on an event loop you already own, such as an ASGI server serving a dashboard beside the node. Use `run()` when the node should own the calling thread and signal handling. This lifecycle sketch leaves node configuration and request serving to the application: ```python import asyncio from nautilus_trader.live import LiveNode from nautilus_trader.live import LiveNodeHandle async def wait_until_running( handle: LiveNodeHandle, task: asyncio.Task[None], ) -> None: while not handle.is_running: if task.done(): await task raise RuntimeError("LiveNode stopped during startup") await asyncio.sleep(0.01) async def serve_with_node(node: LiveNode) -> None: cache, portfolio, handle = node.cache, node.portfolio, node.handle() run_task: asyncio.Task[None] | None = None service_task: asyncio.Task[None] | None = None try: run_task = asyncio.create_task(node.run_async()) await wait_until_running(handle, run_task) service_task = asyncio.create_task(serve_requests(cache, portfolio, handle)) done, _ = await asyncio.wait( (run_task, service_task), return_when=asyncio.FIRST_COMPLETED, ) if run_task in done: await run_task raise RuntimeError("LiveNode stopped while the service was running") await service_task finally: if service_task is not None and not service_task.done(): service_task.cancel() await asyncio.gather(service_task, return_exceptions=True) try: if run_task is not None: handle.stop() await run_task finally: node.dispose() ``` Both entry points run the same lifecycle, so a hosted node performs the same startup ordering, maintenance, reconciliation, and shutdown as an owned one. The mode decides only who owns signal handling: a hosted node installs no handlers, leaving `SIGINT` and `SIGTERM` to the host. ### Access and shutdown `run_async()` returns a coroutine and lends the node to it for the run's duration: - Capture `cache`, `portfolio`, and `handle()` **before starting**. Each stays usable while the node runs, whereas reading state through the node itself raises until the run returns it. - `handle()` works throughout, since it is how a host stops the node. `is_running` also answers throughout because it reads the same handle. - Call `dispose()` **after the run task finishes** to release the node's resources. Calling it during the run returns without doing anything; it does not defer disposal. `LiveNodeHandle` is safe to call from any thread, including a signal handler. `stop()` requests a graceful shutdown and returns immediately, so the awaiting task resolves only once shutdown finishes. Cancelling that task requests the same shutdown, waits for it, then re-raises the cancellation, which keeps `asyncio.timeout` and task groups behaving as their callers expect. ### Host integration and limits Compatibility is tested with the default asyncio loop and uvloop. An ASGI lifespan managed by Uvicorn can apply the same ownership pattern without transferring signal handling to the node. Before an ASGI lifespan reports startup complete, wait until the handle reports `Running` while checking whether the run task has failed. Keep supervising the task after startup, and treat unexpected completion as a service failure. While the node is running, it yields to the host loop periodically, so a burst of events cannot starve the host's own callbacks. Startup and shutdown drain their queues without yielding, so a large instrument load or a shutdown backlog can hold the loop for the length of that drain. :::warning[One LiveNode per process] Run one concurrent `LiveNode` per process. The runner binds its channel senders and message bus into thread-local storage, and other runtime state is process-wide. `run_async()` also rejects a second hosted node on the same event loop. Run additional nodes in separate processes. When an ASGI application lifespan constructs the node, run that application with one worker. Do not use hot reload for live trading because it restarts the worker and its node. Scale HTTP request handling with processes that do not construct a trading node. ::: A node configured with a cache database backing is rejected on a host loop. Those backings wait for their worker task by blocking the calling thread, which stalls the host loop rather than slowing it. Native-only nodes can use `run()` for database backing. Custom Python clients drive asyncio in both launch modes, so neither supports cache database backing. ## Configuration For how config structs handle defaults, `T` vs `Option` semantics, and builder patterns, see the [Configuration](configuration.md) concept guide. For node and execution engine settings, strategy configuration, cache backing, and multi-venue wiring, see the [Configure a live trading node](../how_to/configure_live_trading.md) how-to guide. ## Execution reconciliation For how submit, modify, and cancel commands resolve, see [Command outcomes](execution/policies.md#command-outcomes). At startup, reconciliation aligns cached order and position state with venue reports before trader components start. Continuous checks can then monitor in-flight orders, open orders, positions, and own order books while the node runs. When an adapter declares bounded historical reports, startup reconciliation applies their fill economics only when the report set and retained state prove a coherent position transition. Incomplete or ambiguous history can still recover exact order state without changing positions or portfolio economics. See [Execution reconciliation](execution/reconciliation.md) for configuration, recovery procedures, runtime checks, scenarios, and invariants. ## Rust live runner metrics Rust `LiveNode` exposes primitive runner metrics through `LiveNodeHandle::metrics_snapshot()`. Get the handle from the node before calling `run()`, then poll snapshots from another task and derive rates or utilization from deltas. ```rust use std::time::Duration; use nautilus_common::enums::Environment; use nautilus_live::node::{LiveNode, RunnerMetricsDelta}; let mut node = LiveNode::builder(trader_id, Environment::Live)? // Add clients, actors, and strategies here. .build()?; let metrics_handle = node.handle(); tokio::spawn(async move { let mut prev = metrics_handle.metrics_snapshot(); let mut interval = tokio::time::interval(Duration::from_secs(1)); loop { interval.tick().await; let next = metrics_handle.metrics_snapshot(); let delta = RunnerMetricsDelta::from_snapshots(prev, next); if delta.elapsed_ns == 0 { prev = next; continue; } let elapsed_s = delta.elapsed_ns as f64 / 1_000_000_000.0; let data_event_rate = delta.data_events as f64 / elapsed_s; let data_event_staleness_ns = if next.data_events.last_dispatch_at_ns == 0 { 0 } else { next.elapsed_ns .saturating_sub(next.data_events.last_dispatch_at_ns) }; log::info!( "Runner metrics: data_event_rate={data_event_rate:.0} \ data_event_staleness_ns={data_event_staleness_ns} \ dispatch_utilization={:.6} loop_utilization={:.6} \ mean_dispatch_ns={} data_queue_depth={}", delta.dispatch_utilization(), delta.loop_utilization(), delta.mean_dispatch_ns(), next.data_events.queue_depth, ); prev = next; } }); node.run().await?; ``` ### Snapshot scope and interpretation The snapshot covers `LiveNode::run` channel dispatch after startup, including residual dispatch during the shutdown grace period. It does **not** include startup buffering, startup flushes, or the final post-loop drain. - `dispatch_busy_ns` covers the five dispatch branches. `maintenance_busy_ns` and `external_msgbus_busy_ns` cover non-dispatch loop work. - Queue depths are point samples from the maintenance tick while the node is running, and can be stale during shutdown grace. - Snapshots are lock-free and may not be a consistent cross-field view. Derive rates from successive snapshots with **saturating deltas**. - Counters reset when `LiveNode::run` enters steady state. ## Dispatch priority and overload behavior The live runner's seven internal message channels are separate and unbounded. When several message channels are ready together, the runner polls in this order: 1. Time and system work. 1. Execution events. 1. Execution commands. 1. External message-bus ingress. 1. Data events. 1. Data commands. Events precede commands within the execution and data channel pairs. This polling order keeps a market-data backlog from taking priority over ready execution traffic. It does not define **one global FIFO order** across channels, adapters, or venues. Each selected branch runs to completion before the runner polls again, so a slow handler delays every channel. The runner yields to the host event loop periodically, but yielding does not change channel priority or shorten a slow handler. :::warning[Unbounded queues] Runner channels do not apply producer backpressure, coalesce messages, shed market data, or impose a maximum queue depth. Sustained input above dispatch capacity therefore increases queue depth, latency, and memory use. The runner does not automatically throttle a feed, halt trading, or shut down when a threshold is crossed. ::: Use runner metrics and queue-state events to detect this pressure. The thresholds are operational signals, not service-time guarantees. The application must decide how to alert, reduce input, halt new exposure, or stop the node when pressure persists. ## Queue pressure monitoring `LiveNode` converts runner queue samples into typed state transitions when `LiveNodeConfig.queue_monitor` is set. The monitor is **disabled by default** and publishes no queue-state events while the field is unset. ### Configure thresholds The following example sets the thresholds applied to every monitored runner channel: ```rust tab="Rust" use nautilus_live::config::{LiveNodeConfig, QueueMonitorConfig}; let config = LiveNodeConfig { queue_monitor: Some( QueueMonitorConfig::builder() .queue_depth_trigger(1_000) .queue_depth_clear(500) .mean_dispatch_ns_trigger(250_000) .mean_dispatch_ns_clear(150_000) .build(), ), ..Default::default() }; ``` ```python tab="Python" from nautilus_trader.live import LiveNodeConfig from nautilus_trader.live import QueueMonitorConfig config = LiveNodeConfig( queue_monitor=QueueMonitorConfig( queue_depth_trigger=1_000, queue_depth_clear=500, mean_dispatch_ns_trigger=250_000, mean_dispatch_ns_clear=150_000, ), ) ``` The four values apply to each monitored runner channel: - `time_events` - `exec_events` - `exec_commands` - `data_events` - `data_commands` Each clear threshold must be **lower than its trigger threshold**. Configuration validation rejects equal or inverted thresholds. ### State transitions The live runner evaluates the monitor on its 100 ms maintenance tick, after sampling current queue depths. Queue depth is a point-in-time value. Mean dispatch time uses the messages and dispatch busy time accumulated since the previous metrics snapshot. | Condition | Measure | `Triggered` | `Cleared` | | ------------ | ----------------------------------------------------- | ---------------------------------------------- | -------------------------------------------- | | `Backlogged` | Point-in-time queue depth. | `queue_depth >= queue_depth_trigger` | `queue_depth <= queue_depth_clear` | | `Slow` | Per-channel mean dispatch time for the sample window. | `mean_dispatch_ns >= mean_dispatch_ns_trigger` | `mean_dispatch_ns <= mean_dispatch_ns_clear` | Each channel tracks `Backlogged` and `Slow` independently: - A value between the clear and trigger thresholds retains the prior state, so it does not publish another event. - If both conditions cross on one tick, the node publishes two events. Each condition clears independently. - A sample window with no dispatches does not evaluate `Slow`. The condition retains its prior state until a window contains a dispatch. ### Typed delivery Each transition publishes a fresh `QueueStateChanged` value on `events.system.QueueStateChanged.`, where `` is the Rust variant name, such as `DataEvents` for Python `SystemChannel.DATA_EVENTS`. The event identifies the configured trader, runner channel, condition, and transition state. It also records the queue depth and mean dispatch time at the crossing, a fresh event ID, and event timestamps. Actors subscribe with `subscribe_queue_state(...)` and receive events through `on_queue_state(...)`. The Python API exposes `SystemChannel`, `QueueCondition`, `QueueState`, and `QueueStateChanged` from `nautilus_trader.common`. Publication stays on the in-process typed message bus, and the event has no wire representation for external message-bus streaming. See [Queue pressure state](actors.md#queue-pressure-state) for actor examples. ## Socket transport state ### Publication and routing Actors can observe transport availability for adapters that opt into socket state reporting. `LiveNode` publishes `SocketStateChanged` on `events.system.SocketStateChanged..` with the trader ID, client ID, optional venue, stable endpoint label, state, fresh event ID, and event timestamps. The endpoint label identifies one logical adapter transport without exposing its URL. `LiveNode` sets both timestamps from the kernel clock when it handles the transport's neutral state notification. Adapters send the notification through the runner's system-event channel, separately from market data. The internal channel is not part of queue-pressure monitoring. The client ID and endpoint components percent-encode bytes other than ASCII letters, digits, `-`, and `_`, so dots and wildcard characters in labels cannot change topic matching. Use `subscribe_socket_state` filters to select a client, endpoint, or both without constructing topic names. ### State semantics | Transport event or condition | Publication | | ----------------------------------- | ------------------------------------------------------ | | TCP or WebSocket becomes available. | `Connected`. | | Active transport is lost. | `Disconnected`. | | Connection or retry attempt fails. | No event. | | Deliberate shutdown. | No disconnect event. | | Reconnect attempts are exhausted. | No additional event after the reported transport loss. | `Connected` reports **transport availability**. It does not mean that authentication, subscription replay, or adapter recovery has completed. :::warning Socket state is operational evidence, not an execution-command outcome. A disconnect by itself does not reject, cancel, or resolve an in-flight command; stream updates, queries, or reconciliation provide that evidence under the [command outcome policy](execution/policies.md#command-outcomes). ::: ### Dead-peer detection A connection can stop delivering without closing: a NAT or load balancer drops it with no `FIN` and no `RST`, so writes keep succeeding into the send buffer and nothing surfaces the loss. #### Heartbeat timeout Handler-mode WebSocket and raw TCP clients reconnect on heartbeat timeout. With a configured heartbeat, the default timeout is **three heartbeat intervals**. An explicit `heartbeat_timeout_secs` overrides that window. WebSocket clients reset it on any inbound frame; raw TCP clients reset it when bytes arrive. The client expects the peer to answer heartbeats, so a transport with no heartbeat gets no default window. An explicit timeout can still enable detection without a heartbeat. For WebSocket clients, that window counts all frames, so a keepalive reply refreshes it and a quiet market does not trip it. #### Feed idle timeout An adapter that also needs to detect a feed which stopped flowing while the transport stays healthy sets a separate idle timeout, which only Text and Binary frames refresh. That second window suits a venue which pushes data on a known cadence. Where the venue answers the keepalive with a text payload, its reply refreshes the idle timeout exactly like real data does, so the window means something only when it sits below the heartbeat interval. ### Adapter and actor integration Adapter integrations construct a `SocketStateSink` and set it through the network client's `state_sink` builder option. Publication requires the `LiveNode` runner; the standalone `AsyncRunner` does not publish these events. Actors subscribe with `subscribe_socket_state(...)` and receive events through `on_socket_state(...)`. The Python API exposes `SocketState` and `SocketStateChanged` from `nautilus_trader.common`. Delivery stays on the typed in-process bus; external message-bus streaming and wire formats do not expose these events. ### Endpoint reconnect commands An actor or strategy can call `reconnect_socket(client_id, endpoint)` with an endpoint label from a state event. The runner routes the typed command through the kernel and the engine that owns the registered endpoint. The engine invokes only that transport's reconnect handle. It does not call the containing `DataClient` or `ExecutionClient` disconnect and connect lifecycle. :::note The API is **fire-and-observe**. A successful return means the command passed local validation and was queued. It does not acknowledge kernel acceptance or completed recovery. ::: An accepted request emits `SocketStateChanged` for the selected endpoint: 1. `SocketState.DISCONNECTED` as the transport enters reconnect mode. 1. `SocketState.CONNECTED` after transport recovery. The transport's normal reconnect controller preserves its authentication, subscription replay, and adapter recovery behavior. The kernel logs unknown clients, unsupported clients, unknown or ambiguous endpoints, duplicate requests, disconnecting transports, and closed transports. These rejections emit no socket state change and do not affect another endpoint. Endpoint labels use identifier characters only and never contain raw URLs. ## Shutdown on error Set `LiveNodeConfig.shutdown_on_error=True` so that a Rust error log requests a live node shutdown. The Rust logger records the first `log::error!` emitted after the kernel starts, including error logs from other threads, and the kernel publishes a `ShutdownSystem` command when the live event loop next checks for shutdown. The shutdown request follows the normal live node stop path. The node stops the trader, awaits the post-stop delay, disconnects clients, and stops the engines. It does not abort the process. ```python from nautilus_trader.config import LiveNodeConfig config = LiveNodeConfig(shutdown_on_error=True) ``` The trigger follows these rules: - Error logs suppressed by component filters or logging bypass mode still request shutdown. - A new kernel run clears and re-arms the trigger, so a process can restart a node without reinitializing the logging system. - Shutdown-on-error observes **Rust `log` records**, not Python `logging.error(...)` calls. :::note The per-engine `graceful_shutdown_on_error` option has been removed. Configure shutdown-on-error at the node/kernel level instead. ::: ## Related guides - [Execution reconciliation](execution/reconciliation.md) - State recovery and runtime consistency checks. - [Execution policies](execution/policies.md) - Command delivery, persistence, and recovery boundaries. - [Python](python.md) - Python ownership, runtime, and public API boundaries. - [Configure a live trading node](../how_to/configure_live_trading.md) - Node and engine configuration. - [Run live trading with Rust](../how_to/run_rust_live_trading.md) - Rust node setup and venue connection. - [Adapters](adapters.md) - Venue connectivity. - [Execution](execution/) - Command outcomes and order execution. - [Message bus](message_bus.md) - Typed in-process publish and subscribe behavior. - [Backtesting](backtesting/) - Testing strategies before deployment. # Logging Source: https://nautilustrader.io/docs/latest/concepts/logging/ The platform provides logging for both backtesting and live trading using a high-performance logging subsystem implemented in Rust with a standardized facade from the `log` crate. The core logger operates in a separate thread and uses a multi-producer single-consumer (MPSC) channel to receive log messages. This moves log output I/O off the calling thread. Message arguments are still formatted on the calling thread before the event is queued. Logging output is configurable and supports: - **stdout/stderr writer** for console output - **file writer** for persistent storage of logs :::tip Infrastructure such as [Vector](https://github.com/vectordotdev/vector) can be integrated to collect and aggregate events within your system. ::: ## Architecture The logging subsystem captures events from multiple sources and routes them through an MPSC channel to a dedicated logging thread: ```mermaid flowchart TB subgraph Sources["Log Sources"] PY["Python Logger"] NAUT["Nautilus Rust Components"] LOG["External Rust Libraries
(using log crate)
rustls, etc."] end subgraph Filtering["Filtering"] LF["stdout_level / fileout_level
(LoggerConfig)"] end subgraph Logger["Nautilus Logger"] NL["Logger
(implements log::Log)"] end subgraph Channel["MPSC Channel"] TX["Sender (tx)"] RX["Receiver (rx)"] end subgraph Thread["Logging Thread"] LT["Log Writer"] end subgraph Output["Output"] STDOUT["stdout/stderr"] FILE["Log Files"] end PY --> NL NAUT --> NL LOG --> LF --> NL NL --> TX --> RX --> LT LT --> STDOUT LT --> FILE subgraph Tracing["Tracing Subscriber (optional)"] TRACE["External Rust Libraries
(using tracing crate)
hyper_util, h2, tokio, etc."] EF["RUST_LOG
(EnvFilter)"] FMT["fmt::Layer"] end TRACE --> EF --> FMT --> STDOUT ``` - **Python and Nautilus components**: Log directly through the Nautilus Logger. - **External `log` crate users**: Filtered by `stdout_level`/`fileout_level` in `LoggerConfig`. - **External `tracing` crate users**: When enabled, output goes directly to stdout (separate from Nautilus logging), filtered by the `RUST_LOG` environment variable. - **Logging thread**: All Nautilus log events are sent through an MPSC channel to a dedicated thread, ensuring the main thread isn't blocked by I/O operations. ## Configuration Logging can be configured by importing the `LoggerConfig` object. By default, log events with an `INFO` `LogLevel` and higher are written to stdout/stderr. The following log levels are supported: - `OFF` - Disable logging. - `TRACE` - Most verbose level. - `DEBUG` - Detailed diagnostic information. - `INFO` - General operational messages. - `WARNING` - Potential issues that don't prevent operation. - `ERROR` - Errors that may affect functionality. See the `LoggerConfig` [API Reference](/docs/python-api-latest/common.html#nautilus_trader.common.LoggerConfig) for further details. Logging can be configured in the following ways: - Minimum `LogLevel` for stdout/stderr. - Minimum `LogLevel` for log files. - Maximum size before rotating a log file. - Maximum number of backup log files to maintain when rotating. - Automatic log file naming with date or timestamp components, or custom log file name. - Directory for writing log files. - Plain text or JSON log file formatting. - Filtering of individual components by log level. - ANSI colors in log lines. - Bypass logging entirely. - Print Rust config to stdout at initialization. - Truncate an existing log file on startup (`clear_log_file`). ### Standard output logging Log messages are written to the console via stdout/stderr writers. Set the minimum level with `stdout_level`. ### File logging Log files are written to the current working directory by default. The naming convention and rotation behavior are configurable and follow specific patterns based on your settings. Set the log directory and custom file basename with `FileWriterConfig.directory` and `FileWriterConfig.file_name`. **Log file formats:** - `None` (default) - Plain text format with `.log` extension. - `"json"` - JSON format with `.jsonl` extension, useful for log aggregation tools. For detailed information about log file naming conventions and rotation behavior, see the [Log file rotation](#log-file-rotation) and [Log file naming convention](#log-file-naming-convention) sections below. #### Log file rotation Rotation behavior depends on both the presence of a size limit and whether a custom file name is provided: - **Size-based rotation**: - Set `FileWriterConfig.file_rotate` to a `(max_file_size, max_backup_count)` tuple, such as `(100_000_000, 5)` for 100 MB and five backup files. - When writing a log entry would make the current file exceed this size, the file is closed and a new one is created. - Rotation file names have millisecond resolution. If a rotation resolves to the active path, logging continues to that file, which may briefly exceed the configured maximum size. - **Date-based rotation (default naming only)**: - Applies when `file_rotate` and `file_name` are both unset. - On the first write after each UTC date change (midnight), the current log file is closed and a new one is started, creating one file per UTC day. - **No rotation**: - When `file_name` is set without `file_rotate`, logs continue to append to the same file. - Note: Size-based rotation takes precedence: if both a custom name and size limit are provided, rotation still occurs. - **Backup file management**: - The second value in `file_rotate` limits the total number of rotated files kept. - When this limit is exceeded, the oldest backup files are automatically removed. #### Log file naming convention The default naming convention ensures log files are uniquely identifiable and timestamped. The format depends on whether file rotation is enabled: **With file rotation enabled**: - **Format**: `{trader_id}_{%Y-%m-%d_%H%M%S-%3f}_{instance_id}.{log|jsonl}` - **Example**: `TESTER-001_2025-04-09_210721-521_d7dc12c8-7008-4042-8ac4-017c3db0fc38.log` - **Components**: - `{trader_id}`: The trader identifier (e.g., `TESTER-001`). - `{%Y-%m-%d_%H%M%S-%3f}`: UTC datetime with millisecond resolution. - `{instance_id}`: A unique instance identifier. - `{log|jsonl}`: File suffix based on format setting. **Without size-based rotation (default naming)**: - **Format**: `{trader_id}_{%Y-%m-%d}_{instance_id}.{log|jsonl}` - **Example**: `TESTER-001_2025-04-09_d7dc12c8-7008-4042-8ac4-017c3db0fc38.log` - **Components**: - `{trader_id}`: The trader identifier. - `{%Y-%m-%d}`: Date only (YYYY-MM-DD). - `{instance_id}`: A unique instance identifier. - `{log|jsonl}`: File suffix based on format setting. - **Note**: With default naming and no size limit, logs rotate daily at UTC midnight. **Custom naming**: If `file_name` is set (e.g., `my_custom_log`): - With rotation disabled: The file will be named exactly as provided (e.g., `my_custom_log.log`). - With rotation enabled: The file will include the custom name and timestamp (e.g., `my_custom_log_2025-04-09_210721-521.log`). ### Component log filtering The `component_levels` parameter sets log levels for individual components. The input value should be a dictionary of component ID strings to log level strings: `dict[str, str]`. Below is an example of a trading node logging configuration that includes some of the options mentioned above: ```python from nautilus_trader.common import LogLevel from nautilus_trader.config import FileWriterConfig from nautilus_trader.config import LoggerConfig from nautilus_trader.config import LiveNodeConfig from nautilus_trader.model import TraderId config_node = LiveNodeConfig( trader_id=TraderId.from_str("TESTER-001"), logging=LoggerConfig( stdout_level=LogLevel.INFO, fileout_level=LogLevel.DEBUG, component_levels={"Portfolio": "INFO"}, file_config=FileWriterConfig(file_format="json"), ), ) ``` For backtesting, the `BacktestEngineConfig` class can be used instead of `LiveNodeConfig`, as the same options are available. ### Environment variable configuration The `NAUTILUS_LOG` environment variable provides an alternative way to configure logging using a semicolon-separated spec string. This is useful for Rust-only binaries or when you want to override logging settings without modifying code. ```bash export NAUTILUS_LOG="stdout=Info;fileout=Debug;RiskEngine=Error;is_colored" ``` **Supported keys:** | Key | Type | Description | | --------------------- | --------- | ------------------------------------------------ | | `stdout` | Log level | Maximum level for stdout output. | | `fileout` | Log level | Maximum level for file output. | | `is_colored` | Flag | Enable ANSI colors (default: true). | | `print_config` | Flag | Print config to stdout at startup. | | `log_components_only` | Flag | Only log components with explicit filters. | | `` | Log level | Component-specific level (exact match). | | `` | Log level | Module-specific level (prefix match, Rust only). | Flags are enabled by their presence in the spec string (no value needed). Log levels are case-insensitive: `Off`, `Trace`, `Debug`, `Info`, `Warn`, `Error`. :::note For Rust-only binaries, the logging subsystem initializes lazily on first use. Setting `NAUTILUS_LOG` configures it without requiring explicit `init_logging()` calls. ::: ### Components-only logging When focusing on a subset of noisy systems, enable `log_components_only` to log messages only from components listed in `component_levels`. All other components are suppressed regardless of the global stdout or file level. Example (Python configuration): ```python logging = LoggerConfig( stdout_level=LogLevel.INFO, component_levels={ "RiskEngine": "DEBUG", "Portfolio": "INFO", }, log_components_only=True, ) ``` If configuring via the environment using the Rust spec string, include `log_components_only` alongside component filters, for example: ```bash export NAUTILUS_LOG="stdout=Info;log_components_only;RiskEngine=Debug;Portfolio=Info" ``` ### Module path filtering (Rust only) When using the `NAUTILUS_LOG` environment variable, you can filter by Rust module paths in addition to component names. Keys containing `::` are treated as module path filters with prefix matching, while keys without `::` are component filters with exact matching. ```bash # Filter all OKX adapter modules to Warn, but allow Debug for the websocket modules export NAUTILUS_LOG="stdout=Info;nautilus_okx::=Warn;nautilus_okx::websocket=Debug" ``` The longest matching prefix takes precedence. In the example above, `nautilus_okx::websocket::handler` would use the `Debug` level (longer prefix), while `nautilus_okx::data` would use `Warn`. :::tip Rust log macros automatically capture the module path when no explicit component is provided. This enables module-level filtering to work with standard logging calls. ::: :::note Module path filtering is only available via the `NAUTILUS_LOG` environment variable. The Python `component_levels` configuration uses component name matching only. ::: :::warning If `log_components_only=True` (or `log_components_only` is present in the spec string) and `component_levels` is empty, no log messages will be emitted to stdout/stderr or files. Add at least one component filter or disable components-only logging. ::: ### Log colors ANSI color codes improve log readability in terminals. In environments that do not support ANSI color rendering (such as some cloud environments or text editors), these color codes may not be appropriate as they can appear as raw text. Set `LoggerConfig.is_colored=False` for these environments. ## Using a logger directly It's possible to use `Logger` objects directly, and these can be initialized anywhere (very similar to the Python built-in `logging` API). If you ***aren't*** using an object which already initializes a `NautilusKernel` (and logging) such as `BacktestEngine` or `LiveNode`, then you can activate logging in the following way: ```python from nautilus_trader.common import init_logging from nautilus_trader.common import Logger from nautilus_trader.common import LogLevel from nautilus_trader.core import UUID4 from nautilus_trader.model import TraderId log_guard = init_logging( trader_id=TraderId.from_str("TESTER-001"), instance_id=UUID4(), level_stdout=LogLevel.INFO, ) logger = Logger("MyLogger") ``` See the [`init_logging` API Reference](/docs/python-api-latest/common.html) for further details. **Keep the returned `LogGuard` alive** for as long as direct logging is needed. The logging subsystem supports up to 255 concurrent guards. ## LogGuard: managing log lifecycle `init_logging` returns a `LogGuard` that tracks one user of the process-global logging subsystem. `BacktestEngine` and `LiveNode` own their guards internally, so application code does not need to acquire a guard from an engine or node. ### Reference counting implementation The logging system uses reference counting to track active `LogGuard` instances: - **Counter increments**: When a new `LogGuard` is created, an atomic counter is incremented. - **Counter decrements**: When a `LogGuard` is dropped, the counter is decremented. - **Last guard**: When the counter reaches zero, pending file logs are flushed and synced. The process-global logging thread stays available for later guards. - **Maximum guards**: The system supports up to 255 concurrent `LogGuard` instances. Attempting to create more raises a `ValueError` from `init_logging`, or a `RuntimeError` from engine or node creation. Abrupt termination can still lose buffered logs. Dispose engines and nodes normally, and retain the guard returned by direct `init_logging` calls until the application no longer needs logging. ## Tracing subscriber for external Rust libraries External Rust crates that use the `tracing` crate can have their log output displayed by enabling the tracing subscriber. This is useful for debugging external dependencies or when integrating custom Rust components (such as feature extractors or adapters) compiled as separate PyO3 extensions. ### Enabling the subscriber Initialize the tracing subscriber directly: ```python from nautilus_trader.common import init_tracing init_tracing() ``` ### Filtering with RUST_LOG The `RUST_LOG` environment variable controls which tracing events are displayed: ```bash # Show debug logs from your crate, warn and above from hyper RUST_LOG=my_feature_extractor=debug,hyper=warn python my_script.py ``` If `RUST_LOG` is not set, the default filter level is `warn`. ### How it works The tracing subscriber uses a `tracing-subscriber` fmt layer with a custom formatter to output directly to stdout. This is separate from the Nautilus logging infrastructure - tracing output uses a Nautilus-aligned format with nanosecond timestamps. Example tracing output: ``` 2026-01-24T05:51:42.809619000Z [DEBUG] hyper_util::client::legacy::connect::http: connecting to 104.18.5.240:443 2026-01-24T05:51:42.810543000Z [DEBUG] hyper_util::client::legacy::pool: pooling idle connection for ("https", api.example.com) ``` **Differences from Nautilus logging:** - Tracing output goes directly to stdout, not through the Nautilus logging thread. - Tracing events are not written to Nautilus log files. - Filtering is controlled exclusively by `RUST_LOG`, independent of `LoggerConfig`. For external libraries that use the `log` crate (such as `rustls`), their events go through the Nautilus logger and are filtered by `stdout_level`/`fileout_level` in `LoggerConfig`. :::tip `RUST_LOG` only affects crates using `tracing`. For crates using `log`, configure verbosity through `LoggerConfig` or the `NAUTILUS_LOG` environment variable (e.g., `NAUTILUS_LOG=stdout=Debug`). ::: :::note The tracing subscriber can only be initialized once per process. A second `init_tracing()` call raises an error. ::: ## Platform-specific considerations ### Windows shutdown behavior On Windows, non-deterministic garbage collection during interpreter shutdown can occasionally delay the final `LogGuard` drop until after interpreter teardown has begun. Dropping the last guard is what flushes and syncs pending file logs, so a delayed drop can result in truncated logs. ## Related guides - [Architecture](architecture.md) - System architecture including logging infrastructure. # Message Bus Source: https://nautilustrader.io/docs/latest/concepts/message_bus/ The `MessageBus` enables communication between system components through message passing. This design creates a loosely coupled architecture where components interact without direct dependencies. The *messaging patterns* include: - Point-to-Point - Publish/Subscribe - Request/Response Messages exchanged via the `MessageBus` fall into three categories: - Data - Events - Commands ## Topic hierarchy Nautilus keeps market data topics under the `data` root. Live data publications use the direct `data....` topics, for example `data.book.deltas.XCME.ESZ24`. When requested, replayed, or workflow-generated data flows over the message bus as topic-addressable data, the `DataEngine` publishes it under `data.pipeline....`. Long requests, grouped requests, and aggregation chains can split, transform, and fan data back in before the parent request completes. These messages are still data messages, but they do not claim the same live ordering and timing semantics as normal real-time publications. For example, book deltas on the pipeline path use `data.pipeline.book.deltas.XCME.ESZ24`. Correlated request responses are delivered through response handlers keyed by correlation ID. The `data.response` topic is a capture channel for response publications, not the pipeline data path. ## Message integrity Once a message is created, **its fields must not be mutated**. This includes container fields such as `params` maps. Components can read a message and derive local state from it, but they must not rewrite the original. Immutable messages keep every consumer seeing the same input, preserve what was true at emission time, and remove a class of shared-state races. Replay, debugging, and audit all depend on messages remaining stable after dispatch. Three ownership rules follow from this: - Caller-supplied request options stay on the message. - Response metadata returned to the caller stays on the response. - Component workflow state (bounded date ranges, grouping state, replay cursors, counters, processing flags) stays in component-owned context keyed by message or request ID. When a component needs a derived message, it creates a new one with the required values instead of rewriting the original. ## Data and signal publishing While the `MessageBus` is a lower-level component that users typically interact with indirectly, `DataActor`, `Strategy`, and `ExecutionAlgorithm` provide typed methods built on top of it: ```python def publish_data(self, data_type: DataType, data: CustomData) -> None: def publish_signal(self, name: str, value, ts_event: int = 0) -> None: ``` These methods publish custom data and signals without exposing the raw message bus to Python. ## Python topic messaging Registered Python `DataActor`, `Strategy`, and `ExecutionAlgorithm` components publish arbitrary Python objects through `publish_message(topic, message)`. Use `subscribe_topic(topic, handler, priority=0)` and `unsubscribe_topic(topic, handler)` to manage callbacks on the same runtime bus. These methods do not construct, replace, or dispose the bus, and do not expose `self.msgbus`. Subscribe from a component callback such as `on_start`: ```python from nautilus_trader.common import DataActor class RiskObserver(DataActor): def on_start(self) -> None: self.subscribe_topic("app.risk.*", self.on_risk, priority=10) def on_risk(self, message: object) -> None: self.log.info(f"Risk update: {message}") ``` From another registered component, publish on a matching topic: ```python self.publish_message("app.risk.limit", {"instrument": "BTCUSDT", "limit": 3}) ``` Publication requires a non-empty concrete topic without wildcards. Subscription patterns accept `*` for zero or more characters and `?` for exactly one character. Wildcards can cross dots, so `app.risk.*` also matches `app.risk.limit.eu`. Handlers receive the original object without copying, string conversion, or serialization. Follow [message integrity](#message-integrity): treat published objects as immutable. Use application-owned topics such as `app.risk.limit`; system data and event topics carry their own payload types. These callbacks receive objects published through `publish_message`, not typed market data or signals. Use the corresponding typed subscription methods for those payloads. Patterns that also match signals or custom data log an error for each incompatible payload instead of invoking the Python callback. Python object publication does not send the object to external message-bus storage or transport. Delivery is **synchronous**. Nested publication finishes before the publishing callback continues. Higher subscription priorities run first; equal priority does not imply subscription insertion order. Use distinct priorities when callback order matters, including reproducible backtests. A handler exception is logged, and delivery continues to the remaining handlers. This delivery model does not make it safe to re-enter a component that is already executing in the publishing call stack. Subscriptions belong to the subscribing component: - Repeating the same pattern and callable is a no-op, including attempts to change its priority. Unsubscribe first to change priority. - Python-defined bound methods match by receiver and function, so repeated `self.on_risk` lookups identify the same handler. Other callables, including bound built-in methods, match by object identity. Keep a reference to these other callables for later unsubscription. Neither equality nor representation methods run. - Unsubscription removes only the exact pattern and callable owned by that component. An absent subscription is a no-op. Other components' subscriptions remain independent. - A callback already selected for a publication can still run if it is unsubscribed during that publication. Stopping retains subscriptions, and raw topic callbacks can run while a component is stopped. Resuming keeps those subscriptions. Successful reset and disposal release the component's subscriptions and retained callables; successful reset requires subscribing again. A failed disposal retains subscriptions until retirement cleanup succeeds. Fault cleanup also releases subscriptions. These methods raise `RuntimeError` before runtime registration, after successful disposal, from a foreign thread, while releasing subscriptions, or when the registered bus is unavailable or replaced. Invalid topics and patterns raise `ValueError`; a non-callable handler raises `TypeError`. Priority must be an integer in `[0, 4294967295]`; values outside this range raise `OverflowError`. ## Messaging styles NautilusTrader is an **event-driven** framework where components communicate by sending and receiving messages. Understanding the different messaging styles helps when building trading systems. This guide explains the four primary messaging patterns available in NautilusTrader: | **Messaging style** | **Purpose** | **Best for** | | :------------------------------------------------------------- | :----------------------------------- | :---------------------------------------------------- | | **Custom data publish/subscribe** | Structured trading data exchange | Trading metrics, indicators, data needing persistence | | **Signal publish/subscribe** | Lightweight notifications | Simple alerts, flags, and status updates | | [**Python object publish/subscribe**](#python-topic-messaging) | In-process Python object exchange | Application topics shared by Python components | | **Rust MessageBus publish/subscribe** | Low-level, typed topic communication | Native runtime components | Each approach serves different purposes. Use this guide to decide which pattern to use. ### Rust MessageBus publish/subscribe to topics #### Concept The `MessageBus` is the central hub for all messages in NautilusTrader. Rust components can publish typed messages to named topics and subscribe handlers to those topics. This low-level interface is not part of the Python actor or strategy surface. #### Benefits and use cases Direct message-bus access is for native components that need: - **Cross-component communication** within the system. - **Flexibility** to define typed topics and payloads. - **Decoupling** between publishers and subscribers who don't need to know about each other. - **Global reach** where messages can be received by multiple subscribers. - Working with events that do not fit the data actor model. - Advanced scenarios requiring full control over messaging. #### Considerations - You must track topic names manually (typos could result in missed messages). - You must define handlers manually. ### Custom data publish/subscribe #### Concept Custom data exchanges structured values between data actors and strategies. A `CustomData` value carries a `DataType`, payload, event timestamp, and initialization timestamp for routing and event ordering. #### Benefits and use cases The Data publish/subscribe approach works well when you need: - **Exchange of structured trading data** like market data, indicators, custom metrics, or option greeks. - **Event timestamps** (`ts_event`, `ts_init`) for ordering inputs during backtests; publication itself does not sort messages. - **Data persistence and serialization** through registered custom data classes, integrating with NautilusTrader's data catalog system. - **Standardized trading data exchange** between system components. #### Considerations - The payload must expose `ts_event` and `ts_init`. - Persistence requires registering a serializable custom data class. #### Quick overview code ```python from dataclasses import dataclass from nautilus_trader.model import CustomData from nautilus_trader.model import DataType @dataclass class GreeksData: delta: float gamma: float ts_event: int ts_init: int data_type = DataType("GreeksData") data = CustomData( data_type, GreeksData( delta=0.75, gamma=0.1, ts_event=1_630_000_000_000_000_000, ts_init=1_630_000_000_000_000_000, ), ) self.publish_data(data_type, data) self.subscribe_data(data_type) def on_data(self, data: CustomData) -> None: if data.data_type == data_type: greeks = data.data self.log.info(f"Delta: {greeks.delta}, Gamma: {greeks.gamma}") ``` See [Custom data](custom_data.md) for registration and persistence. ### Signal publish/subscribe #### Concept **Signals** are a lightweight way to publish and subscribe to simple notifications within the actor framework. This is the simplest messaging approach, requiring no custom class definitions. #### Benefits and use cases The Signal messaging approach works well when you need: - **Simple, lightweight notifications/alerts** like "RiskThresholdExceeded" or "TrendUp". - **Quick, on-the-fly messaging** without defining custom classes. - **Broadcasting alerts or flags** as simple primitive values. - **Easy API integration** with straightforward methods (`publish_signal`, `subscribe_signal`). - **Multiple subscriber communication** where all subscribers receive signals when published. - **Minimal setup overhead** with no class definitions required. #### Considerations - Each signal carries a single value. The value is converted to a string on publish, so handlers always receive `signal.value` as a `str`; complex data structures are not preserved. - Differentiate between signals in the `on_signal` handler with `signal.name`. #### Quick overview code ```python # Define signal constants for better organization (optional but recommended) import types from nautilus_trader.common import LogColor from nautilus_trader.core.datetime import unix_nanos_to_dt signals = types.SimpleNamespace() signals.NEW_HIGHEST_PRICE = "NewHighestPriceReached" signals.NEW_LOWEST_PRICE = "NewLowestPriceReached" # Subscribe from a DataActor or Strategy self.subscribe_signal(signals.NEW_HIGHEST_PRICE) self.subscribe_signal(signals.NEW_LOWEST_PRICE) # Publish from a DataActor or Strategy self.publish_signal( name=signals.NEW_HIGHEST_PRICE, value=signals.NEW_HIGHEST_PRICE, # value can be the same as name for simplicity ts_event=bar.ts_event, # timestamp from triggering event ) # Handler (fixed callback name) def on_signal(self, signal): match signal.name: case signals.NEW_HIGHEST_PRICE: self.log.info( f"New highest price was reached. | " f"Signal value: {signal.value} | " f"Signal time: {unix_nanos_to_dt(signal.ts_event)}", color=LogColor.GREEN, ) case signals.NEW_LOWEST_PRICE: self.log.info( f"New lowest price was reached. | " f"Signal value: {signal.value} | " f"Signal time: {unix_nanos_to_dt(signal.ts_event)}", color=LogColor.RED, ) ``` ### Summary and decision guide #### Decision guide: Which style to choose? | **Use case** | **Recommended approach** | **Setup required** | | :------------------------------------- | :---------------------------------- | :---------------------------------------- | | Native system-level communication | Rust `MessageBus` publish/subscribe | Typed topic and handler | | Structured Python component data | `DataActor` custom data methods | `DataType`, `CustomData`, and `on_data()` | | Arbitrary in-process Python objects | Component topic methods | Application topic and callable | | Simple Python alerts and notifications | `DataActor` signal methods | Signal name and `on_signal()` | ## External egress and ingress The `MessageBus` can write serialized messages to external streams. This section describes the external egress and ingress sides of the external bus. Rust-native live nodes use injected `MessageBusExternalEgress` and `MessageBusExternalIngress` surfaces, so the core node does not depend on Redis, a broker, shared-memory implementation, or socket protocol. :::info Redis is the built-in external backing for serializable messages. The minimum supported Redis version is 6.2, required for the `MINID` stream trimming used by autotrim. ::: When external egress is configured, outgoing publish messages are first dispatched to in-process subscribers, then serialized into the existing `BusMessage` wire record: - `topic`: the exact message bus topic used by the internal publish call, for example `data.quotes.BINANCE.BTCUSDT` or `events.order.S-001`. - `type`: the canonical payload type name, for example `QuoteTick` or `OrderEventAny`. - `payload_kind`: present with the value `typed` for control, execution, and reconciliation payloads whose fixed type names could also identify custom payloads. - `encoding`: the payload encoding selected from the message bus encoding policy. - `payload`: serialized bytes encoded with the selected encoding. An external producer that writes directly to a Redis stream must include `topic`, `type`, and `payload`. The `topic` must be a valid publish topic and cannot contain `*` or `?`. The `encoding` field is optional and defaults to JSON when omitted. The receiving node skips entries without `type` because it cannot select a payload decoder. Set `payload_kind` to `typed` for a fixed typed payload; without this discriminator, the receiving node treats the same type name as custom data. External egress receives that record as `publish(BusMessage)`. This outbound call must not block the node's bus thread. Bounded egress implementations drop on a full queue instead of applying back-pressure to the trading loop. Closing the message bus closes the configured egress. Inbound external streams are exposed through the separate Rust `MessageBusExternalIngress` trait. Ingress yields the same `BusMessage { topic, payload_type, encoding, payload }` shape. `republish_external_message` decodes supported inbound messages and republishes them internally without forwarding the message back out. Normal internal republishing requires the inbound payload type to be registered for streaming and skips unregistered types without decoding. `LiveNode::add_stream_processor` in Rust and `LiveNode.add_stream_processor` in Python register callbacks for typed JSON or MessagePack payloads. Processors run in registration order before the internal streaming registration check, so they receive supported typed messages even when internal republishing is disabled. Python processors receive a mapping with an added `payload_type` field. A processor error stops later processors and skips internal republishing for that message. For custom data, egress writes and ingress expects an envelope in the Redis `payload` field, not the bare custom object. The canonical JSON envelope is: ```json { "type": "MyData", "data_type": { "type_name": "MyData", "metadata": { "source": "external" }, "identifier": "optional-storage-key" }, "payload": { "value": 42, "ts_event": 0, "ts_init": 0 } } ``` The envelope requires `type` and `payload`; `data_type` is optional on ingress and defaults to the message type with no metadata or identifier. In the canonical emitted form, `data_type.type_name` uses the same custom type name, `metadata` is an object that may be empty, and `identifier` is present only when assigned. The envelope `type` must match the Redis stream `type`. The envelope `payload` is the bare object passed to the registered class's `from_json(...)` method. MessagePack uses the same map fields encoded as MessagePack bytes. For Python custom data, register the class before starting the node: ```python from nautilus_trader.model import register_custom_data_class register_custom_data_class(MyData) ``` The external-client subscription registers the payload type for streaming, while `register_custom_data_class(...)` installs the process-wide JSON decoder. Both registrations are required. See [Custom data](custom_data.md#registration-architecture) for the class requirements. For Redis, messages are transmitted via a Multiple-Producer Single-Consumer (MPSC) channel to a separate Rust task. That task writes the message to Redis streams. Offloading I/O to a separate task keeps the publishing thread unblocked. With MessagePack or JSON, Rust-native external egress forwards serializable typed publications. This includes instruments, quotes, trades, bars, book deltas, depth-10 snapshots, mark/index/funding updates, option greeks (`OptionGreeks`), account state, portfolio snapshots, order events, position events, and custom data. With the `defi` feature this also includes DeFi blocks, pools, liquidity updates, fee collects, and flash events. Full order book snapshots, `GreeksData` records, option chain slices, and DeFi pool swaps are not forwarded because those types do not implement Serde serialization. When `external_streams` is non-empty, JSON or MessagePack egress also forwards subscription commands, trading commands, execution mass-status requests, order status reports, fill reports, position status reports, and execution mass-status reports. These payloads remain local when no external streams are configured. With SBE or Cap'n Proto, Rust-native external egress forwards the built-in market data payloads with schema codecs: quotes, trades, bars, book deltas, depth-10 snapshots, mark price updates, index price updates, funding rate updates, and option greeks. Other payload types are dropped with a debug log when those schema encodings are selected. ## Configuration The message bus external backing technology uses a behavior config plus a technology-owned backing config. `MessageBusConfig` controls message bus behavior. `RedisMessageBusConfig` owns Redis connection settings and implements `MessageBusBackingFactory`. ```rust use nautilus_common::{ enums::SerializationEncoding, msgbus::{MessageBusBackingFactory, MessageBusConfig}, }; use nautilus_infrastructure::redis::msgbus::RedisMessageBusConfig; let config = MessageBusConfig { encoding: SerializationEncoding::Json, encoding_market_data: Some(SerializationEncoding::Sbe), timestamps_as_iso8601: true, buffer_interval_ms: Some(100), autotrim_mins: Some(30), use_trader_prefix: true, use_trader_id: true, use_instance_id: false, streams_prefix: "streams".to_string(), types_filter: Some(vec!["QuoteTick".to_string(), "TradeTick".to_string()]), ..Default::default() }; let redis_config = RedisMessageBusConfig::default(); let backing = redis_config.create(trader_id, instance_id, config.clone())?; ``` Existing Rust callers can continue using `RedisMessageBusFactory::new(redis_config)`, which delegates to the config implementation. ### Backing config A `RedisMessageBusConfig` is required when using the built-in Redis backing. For a default Redis setup on the local loopback you can pass `RedisMessageBusConfig::default()`. Redis selection is explicit in the Rust type. The config does not use a user-facing selector such as `type = "redis"` or `backing_type = "redis"`. Rust-native callers that inject `MessageBusExternalEgress` with `LiveNodeBuilder::with_external_msgbus_egress` pass concrete connection details when they construct that egress surface. The core message bus does not require a `RedisMessageBusConfig` for injected egress. The Rust live runtime accepts `external_streams` in `MessageBusConfig`, and consumes inbound `BusMessage`s when callers inject a `MessageBusExternalIngress` with `LiveNodeBuilder::with_external_ingress`. The config names the external stream keys; the injected ingress is the concrete runtime source. Rust callers can install `RedisMessageBusConfig` with `LiveNodeBuilder::with_external_msgbus_factory`. Building fails when a factory is combined with separately injected egress or ingress. A factory always installs egress and creates ingress only when `external_streams` is non-empty. Python exposes the same builder method for built-in backing configs, currently `RedisMessageBusConfig`. The existing `RedisMessageBusFactory` wrapper remains supported. Python does not accept arbitrary factory classes. :::warning The built-in Redis ingress starts each configured stream at the current timestamp, so entries that already exist when the node starts are not replayed. After startup it advances the last-seen ID for each stream and preserves those IDs across connection retries. Use cache recovery or the event store when durable pre-start replay is required; `external_streams` provides live forwarding, not a consumer-group backlog. ::: ### Encoding Rust-native external message bus egress supports these encoding names: - JSON (`json`) - MessagePack (`msgpack`) - Cap'n Proto (`capnp`, with the Rust `capnp` feature) - SBE (`sbe`, with the Rust `sbe` feature) Use the `encoding` config option to control the message writing encoding. Use `encoding_market_data` to override the encoding for market data payloads backed by the external bus binary codecs. Use `encoding_builtin` to override account state, portfolio snapshot, order event, and position event payloads. Custom and unmapped payload types always use `encoding`. `MessageBusConfig::validate` requires the default `encoding` to support custom payloads, so it must be JSON or MessagePack. Category overrides must be supported by every published payload type in that category. SBE and Cap'n Proto can currently be used only for `encoding_market_data`, and only when the matching Rust feature is enabled. `encoding_builtin = "sbe"` and `encoding_builtin = "capnp"` fail validation until those schema codecs cover the built-in event category. The Redis cache payload path supports MessagePack and JSON only. SBE and Cap'n Proto are schema payload encodings for Rust-native external message bus egress, not Redis cache encodings, and selecting either for a Redis cache payload is an error. :::tip The `json` encoding is used by default for human readability and interoperability. Use `msgpack` when payload size and serialization performance are a primary concern. ::: ### Timestamp formatting By default timestamps are formatted as UNIX epoch nanosecond integers. Alternatively you can configure ISO 8601 string formatting by setting `timestamps_as_iso8601` to `true`. ### Message stream keys Message stream keys identify individual trader nodes and organize messages within streams. The `trader-` prefix, trader ID, and instance ID segments are optional and controlled by the options below; the streams prefix is always included. With every segment enabled, a trader key has the following structure: ``` trader-{trader_id}:{instance_id}:{streams_prefix} ``` With the default options (`use_trader_prefix` and `use_trader_id` enabled, `use_instance_id` disabled) the base stream key is `trader-{trader_id}:{streams_prefix}`. These options control Redis stream keys. They do not rewrite the `topic` passed to an injected `MessageBusExternalEgress`; that topic remains the internal message bus publish topic. When `stream_per_topic` is `True`, Redis egress appends the topic to the stream key. When it is `False`, Redis stores all messages on the base stream key and keeps the topic as a message field. The following options are available for configuring message stream keys: #### Trader prefix If the key should begin with the `trader-` prefix. #### Trader ID If the key should include the trader ID for the node. #### Instance ID Each trader node is assigned a unique instance ID, which is a UUIDv4. This instance ID helps distinguish individual traders when messages are distributed across multiple streams. You can include the instance ID in the trader key by setting the `use_instance_id` configuration option to `True`. This is particularly useful when you need to track and identify traders across various streams in a multi-node trading system. #### Streams prefix The `streams_prefix` string enables you to group all streams for a single trader instance or organize messages for multiple instances. Configure this by passing a string to the `streams_prefix` configuration option, ensuring other prefixes are set to false. #### Stream per topic Indicates whether the producer will write a separate stream for each topic. This is particularly useful for Redis backings, which do not support wildcard topics when listening to streams. If set to False, all messages will be written to the same stream. :::info Redis does not support wildcard stream topics. For better compatibility with Redis, it is recommended to set this option to False. ::: ### Types filtering When messages are published on the message bus, they are serialized and written to a stream if a backing for the message bus is configured and enabled. To prevent flooding the stream with data like high-frequency quotes, you may filter out certain types of messages from external publication. To enable this filtering mechanism, pass a list of payload type names to the `types_filter` parameter in the message bus configuration. Listed types are excluded from external publication. A name shared by a custom payload and a fixed typed payload excludes both. ```python from nautilus_trader.config import MessageBusConfig # Create a MessageBusConfig instance with types filtering message_bus = MessageBusConfig(types_filter=["QuoteTick", "TradeTick"]) ``` ### Stream auto-trimming Use `autotrim_mins` to set a lookback window in minutes and `autotrim_maxlen` to set an approximate maximum number of entries for each Redis stream. You can configure either policy or both. When both are set, the message bus removes entries that exceed either the time window or the entry-count threshold. Redis applies `autotrim_maxlen` with approximate trimming for better write performance, so a stream may contain slightly more entries than the configured threshold. :::info The Redis implementation trims each stream at most once per minute, so entries can remain up to roughly a minute longer than the `autotrim_mins` window. ::: ## External streams The message bus within a `LiveNode` (node) is referred to as the "internal message bus". A producer node is one which publishes messages onto an external stream (see [external egress and ingress](#external-egress-and-ingress)). The consumer node listens to external streams to receive and publish deserialized message payloads on its internal message bus. ```mermaid flowchart TB producer[Producer Node] stream[Stream] consumer1[Consumer Node 1] consumer2[Consumer Node 2] producer --> stream stream --> consumer1 stream --> consumer2 ``` :::info Set the `LiveDataEngineConfig.external_clients` with the list of `client_id`s intended to represent the external streaming clients. The `DataEngine` will filter out subscription commands for these clients, ensuring that the external streaming provides the necessary data for any subscriptions to these clients. When the Rust `DataEngine` skips an external-client subscription, it registers the corresponding streaming payload type for inbound republishing on the message bus. ::: ### Example configuration The following example details a streaming setup where a producer node publishes Binance data externally, and a downstream consumer node publishes these data messages onto its internal message bus. #### Producer node We configure the `MessageBus` of the producer node to publish to a `"binance"` stream. The settings `use_trader_id`, `use_trader_prefix`, and `use_instance_id` are all set to `false` to ensure a simple and predictable stream key that the consumer nodes can register for. ```rust let message_bus = MessageBusConfig { use_trader_id: false, use_trader_prefix: false, use_instance_id: false, streams_prefix: "binance".to_string(), // <--- stream_per_topic: false, autotrim_mins: Some(30), ..Default::default() }; let redis_config = RedisMessageBusConfig { connection_timeout: 2, response_timeout: 2, ..Default::default() }; let mut node = LiveNode::builder(trader_id, Environment::Live)? .with_msgbus_config(message_bus) .with_external_msgbus_factory(Box::new(redis_config)) .build()?; node.run().await?; ``` #### Consumer node We configure the `MessageBus` of the consumer node to receive messages from the same `"binance"` stream. A `RedisMessageBusConfig` creates ingress from `external_streams`, and `LiveNode::run` publishes the received messages onto the node's internal message bus. We declare the client ID `"BINANCE_EXT"` as an external client so the `DataEngine` does not attempt to send data commands to this client ID. ```rust let data_engine = LiveDataEngineConfig { external_clients: Some(vec![ClientId::from("BINANCE_EXT")]), ..Default::default() }; let message_bus = MessageBusConfig { external_streams: Some(vec!["binance".to_string()]), // <--- ..Default::default() }; let redis_config = RedisMessageBusConfig { connection_timeout: 2, response_timeout: 2, ..Default::default() }; let mut node = LiveNode::builder(trader_id, Environment::Live)? .with_data_engine_config(data_engine) .with_msgbus_config(message_bus) .with_external_msgbus_factory(Box::new(redis_config)) .build()?; node.run().await?; ``` ## Related guides - [Actors](actors.md) - Actors use the message bus for event handling. - [Architecture](architecture.md) - Message bus role in system architecture. # Networking Source: https://nautilustrader.io/docs/latest/concepts/networking/ NautilusTrader adapters use the shared `nautilus-network` clients for HTTP request/response APIs, WebSocket streams, and suffix-framed TCP protocols. These clients add trading-system policy around the underlying Rust transports: rate limits, connection reuse, liveness checks, reconnect control, replay coordination, and bounded reads. | Client | Underlying transport | Use when | Added policy | | -------------- | ----------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------- | | HTTP | Hyper | Finite request/response operations | Layered quotas, pooled connections, keepalive, timeouts, proxy routing, and bounded bodies | | WebSocket | `tokio-tungstenite` or `sockudo-ws` | Long-lived framed streams | Runtime backend selection, quotas, heartbeats, liveness checks, reconnects, and session fencing | | Raw TCP socket | Tokio and `rustls` | Suffix-framed byte streams | Framing, initial retries, heartbeats, liveness checks, reconnects, and ordered replay | The [Adapters](adapters.md) guide explains how venue clients translate these transports into Nautilus domain messages. This page covers the shared transport behavior beneath that boundary. ## HTTP client [`HttpClient`](../../crates/network/src/http/client.rs) wraps one reusable Hyper client and one or more shared rate limiters. A request waits for every applicable quota before the inner client builds and executes it. ```mermaid flowchart LR adapter[Adapter HTTP client] subgraph network[nautilus-network] client[HttpClient] limiter[RateLimiter] inner[InnerHttpClient] end hyper["Hyper client
pool and keepalive"] endpoint[HTTP endpoint] adapter --> client client -->|await quotas| limiter client -->|execute| inner inner <--> hyper hyper <--> endpoint ``` The outer client applies quota policy; the reusable inner client owns connection and response policy. The Rust API exposes `http::Method`, `http::StatusCode`, and `url::Url`. Requests return `HttpResponse` or `HttpResponseStream`, and failures return `HttpClientError`. ### Rate limiting and requests The rate limiter uses the generic cell rate algorithm (GCRA) with a default quota and optional per-key overrides. A request can carry several keys, such as an endpoint and an order scope, and waits for them together. Multiple limiters let one request consume independent budgets, such as per-IP and per-account limits. Sharing their `Arc` values across HTTP clients keeps those budgets process-wide instead of creating one allowance per connection. The client accepts default and per-request headers, query parameters with repeated values, raw request bodies, and `GET`, `POST`, `PUT`, `PATCH`, and `DELETE` methods. A client-level timeout applies to all requests unless a request supplies its own timeout. An optional proxy applies to both HTTP and HTTPS traffic. **HTTP status errors remain normal `HttpResponse` values** so each adapter can interpret the venue's body and retry rules. The transport retries requests canceled before transmission on reused connections, and allows two retries for remote HTTP/2 `GOAWAY(NO_ERROR)` or `REFUSED_STREAM` errors. Other transport failures and HTTP status codes do not trigger retries. Adapters can wrap retryable operations with [`RetryManager`](../../crates/network/src/retry.rs), but the adapter must decide which venue errors and operations are safe to retry. ### Connection reuse and response bounds Each production `HttpClient` enables `TCP_NODELAY`, keeps up to 32 idle connections per host, and retains an idle connection for up to 60 seconds. HTTP/2 connections send keepalive probes every 30 seconds even while idle and use adaptive flow-control windows. Reusing a client preserves the pool and avoids a new TCP and TLS handshake for each request. Buffered responses contain the status, only the header names selected when the client was built, and the raw body bytes. The client rejects a declared body larger than 100 MiB before reading it. For chunked or unbounded responses, it stops as soon as accumulated bytes would cross the same limit. Endpoints whose path or query can contain credentials can use the redacted request path, which removes the URL from transport errors and logs. `HttpClient::get_stream` returns status and body chunks without accumulating the complete response or applying the buffered size limit. One absolute deadline covers headers and the whole body, including time spent processing chunks. No rate-limit keys are supplied, so no quota is consumed. Dropping an unfinished response releases the exchange, including its owned connection task [under simulation](dst.md#simulated-http-and-websocket-transport). Dataset downloads use this path to stream to a temporary file before renaming it. Downloads configure a separate timeout for response headers and for each body read, allowing a progressing transfer to exceed that duration. ### HTTP transport benchmarks The [HTTP comparison](../../crates/network/benches/BENCHMARKS.md#http-transport-comparison), measured 2026-09-09 on an AMD Ryzen Threadripper 9980X, compares the previous Reqwest 0.13.4 client with the direct Hyper implementation. Both run in the same `bench-lto` binary with fat LTO and one codegen unit. The CPU governor is `performance`, ASLR is disabled per process, and client and server threads are pinned to separate physical cores. Accepted sessions have no sampled Cargo or compiler activity. Five independent sessions provide 60 paired samples per workload. The following 64 KiB cases summarize GET and POST at concurrency 1 and 16; the full report includes 1 KiB and 1 MiB responses, p99 values, uncertainty intervals, and resource measurements. | 64 KiB workload | Reqwest req/s | Hyper req/s | Paired throughput change | Paired p99 change | | -------------------- | ------------- | ----------- | ------------------------ | ----------------- | | GET, concurrency 1 | 29,516 | 30,645 | +3.6% | -3.1% | | POST, concurrency 1 | 27,054 | 28,223 | +4.1% | -3.4% | | GET, concurrency 16 | 55,870 | 55,460 | -0.5% | +0.4% | | POST, concurrency 16 | 47,779 | 47,631 | -0.3% | +0.2% | Throughput columns are medians of sample summaries. Changes are medians of paired within-round ratios; positive throughput changes and negative p99 changes favor Hyper. Serial throughput for 1 KiB and 64 KiB responses improves by 3.6% to 5.0%. Concurrent cases range from -2.6% to +0.2%, and the 1 KiB concurrent POST case has a paired p99 increase of 1.8%. These results support a modest serial improvement, with regressions in some concurrent workloads. The benchmark exercises complete requests and validates response bodies, status, headers, and connection reuse over loopback HTTP/1.1. It excludes TLS, HTTP/2, proxies, WAN latency, and adapter parsing, so the results do not establish a production-wide speedup. ## WebSocket client [`WebSocketClient`](../../crates/network/src/websocket/client.rs) separates connection lifecycle from frame transport. A controller owns reconnect and shutdown transitions, one writer serializes all sink access, and handler mode assigns each connection to one reader task. An optional heartbeat task sends liveness traffic through the same writer. ```mermaid flowchart LR adapter[Adapter] subgraph client[WebSocketClient] limiter[RateLimiter] controller[Controller] reconnect[Reconnect handle] reader[Reader task] writer[Writer task] heartbeat[Heartbeat task] state[SocketStateSink] end transport["WsTransport
Message and TransportError"] tungstenite[tokio-tungstenite] sockudo[sockudo-ws] endpoint[WebSocket endpoint] adapter -->|send| limiter --> writer adapter -->|request reconnect| reconnect --> controller reader -->|messages| adapter controller -->|availability edges| state --> adapter controller -->|replace| reader controller -->|replace| writer heartbeat --> writer writer <--> transport --> reader transport -. runtime backend .-> tungstenite transport -. runtime backend .-> sockudo tungstenite <--> endpoint sockudo <--> endpoint ``` The lifecycle tasks use one neutral transport interface, so adapters do not depend on a concrete WebSocket library. ### Connection modes | Mode | Reader ownership | Automatic reconnect | Liveness behavior | | ------- | ---------------------- | ------------------------------ | ---------------------------------------------- | | Handler | Internal callback task | Exponential backoff and jitter | Heartbeat and application-data idle timeouts | | Stream | Caller-owned reader | Disabled | Caller reports failure and replaces the client | Handler mode is the usual choice for long-lived adapter connections. Stream mode suits adapters that need direct stream backpressure or own a protocol-specific reconnect sequence. ### Transport backends The `WsTransport` abstraction normalizes text, binary, Ping, Pong, and Close frames together with transport errors. `WebSocketConfig.backend` selects either backend at runtime: | Backend | Availability | Upgrade headers | Proxy behavior | | ----------------------------------------------------------------- | -------------------------------------------- | ----------------------------------------- | -------------------------------- | | [`tokio-tungstenite`](https://crates.io/crates/tokio-tungstenite) | Always compiled | Passed through the WebSocket handshake | HTTP and HTTPS `CONNECT` tunnels | | [`sockudo-ws`](https://crates.io/crates/sockudo-ws) | Default with the `transport-sockudo` feature | Passed through a local HTTP/1.1 handshake | HTTP and HTTPS `CONNECT` tunnels | Disabling default Cargo features removes `sockudo-ws` and makes Tungstenite the default. Both backends use `rustls` for `wss://` connections and set `TCP_NODELAY` on paths where Nautilus creates the TCP stream. :::warning A recognized SOCKS proxy URL logs a warning and **connects directly** because WebSocket SOCKS tunneling is not implemented. Malformed proxy URLs and other unsupported schemes return an error. ::: ### Liveness and recovery The configured heartbeat sends either an RFC 6455 Ping or a venue-specific text message at a fixed interval. Configuring one also arms a response deadline: the client expects the peer to answer, so an unset `heartbeat_timeout_secs` defaults to three intervals. Set the field to choose a different window. A transport with no heartbeat gets no default, because nothing would guarantee the inbound frames needed to keep the window open. The **heartbeat timeout** resets on every inbound frame, including Ping and Pong, so it detects a peer that has stopped sending anything. The separate **idle timeout** resets only on text or binary application data, so control traffic cannot hide a silent market-data stream. A venue that answers the keepalive with a text payload refreshes the idle timeout exactly like real data does, so that window means something only when it sits below the heartbeat interval. An unset timeout leaves that detection off, except that an unset `heartbeat_timeout_secs` still derives three intervals when a heartbeat is configured. A zero timeout is rejected. Adapters that expose a non-optional integer map zero to unset rather than passing it through. A read failure, write failure, Close frame, heartbeat timeout, idle timeout, or explicit reconnect request moves a handler-mode client into reconnecting state. Reconnect uses exponential backoff with bounded jitter and allows unlimited attempts by default. A replacement connection that remains active for at least 10 seconds resets the attempt count and backoff. A configured maximum closes the client after that many consecutive failed or short-lived attempts. ```mermaid stateDiagram-v2 [*] --> Active: initial connection succeeds Active --> Reconnecting: I/O failure, Close, timeout, or explicit request Reconnecting --> Active: replacement succeeds Reconnecting --> Reconnecting: attempt fails Reconnecting --> Closed: configured attempt limit reached Active --> Disconnecting: deliberate disconnect Reconnecting --> Disconnecting: deliberate disconnect Disconnecting --> Closed: shutdown completes ``` Handler mode publishes `Disconnected` on entry to `Reconnecting` and `Connected` on recovery; individual attempts and deliberate disconnects add no state-sink edges. The writer installs the replacement sink before the controller starts its reader and publishes the reconnect notification. A **connection epoch** advances with each sink replacement. Reader fences drop frames from retired sessions, while epoch-aware handlers and sends let an adapter bind work to the transport that produced it. Mutable reconnect headers apply to later handshakes without interrupting the active connection. Adapters can register an `AuthTracker` so a disconnect invalidates authentication. They can also gate the reconnect buffer on that tracker, making messages wait for the new session to authenticate and discarding the remaining buffer if authentication fails. `SubscriptionState` separately records confirmed, pending subscribe, and pending unsubscribe intent for adapter-driven resubscription; it never sends protocol messages itself. ### Reconnect throttling Once three reconnect attempts occur inside a rolling two-minute window, each further attempt waits at least one second, regardless of the configured backoff. The window is purely time-based: a replacement connection that survives the stability threshold still resets the backoff and attempt count, but has no effect on the floor. Throttling lifts by itself once fewer than three attempts remain inside the window. Venues rate-limit new connections per IP (Binance permits 300 connections per five minutes, OKX three per second), so an unthrottled reconnect loop can otherwise escalate a transient drop into an IP-level throttle or ban affecting every client behind that address. The first three attempts in any window incur no additional throttling delay; configured backoff still applies. A single drop can still trigger an immediate first reconnect. ### State reporting and explicit reconnect Clients configured with a `SocketStateSink` publish ordered `Connected` and `Disconnected` availability edges. A successful initial connection publishes `Connected`; transport loss or an accepted explicit reconnect publishes `Disconnected`; and a successful replacement publishes `Connected`. Initial connection failure, individual retry attempts, retry exhaustion, deliberate disconnect, and client drop do not add events. The sink therefore describes transport availability, not every internal `ConnectionMode` transition. Its callback runs synchronously and serializes edges, so it must return promptly and must not request another transition through the same sink. `request_reconnect()` atomically asks a handler-mode controller to replace its active transport. A cloneable `WebSocketReconnectHandle` gives adapter tasks the same capability without ownership of the client and distinguishes accepted, already reconnecting, disconnecting, closed, and unsupported requests. An accepted request invalidates registered authentication state and publishes `Disconnected` before the replacement can become active. Stream mode reports `Unsupported` because its reader is caller-owned. ### Send semantics Application text and binary sends wait for their rate-limit keys and for an active connection. The ordinary send methods return after enqueueing the frame, so **success does not prove delivery**. The writer keeps FIFO order for application messages buffered during reconnect or after a failed write and replays them on a replacement connection. A control frame belongs to the connection it was issued on, so a failed Ping, Pong, or Close is dropped rather than replayed. This in-memory buffer provides reconnect continuity, not durable or exactly-once delivery. Ownership-bound text sends take an expected connection epoch and wait for the writer result. They fail if ownership changes and never replay on another connection. Connection-bound Pong sends use the same epoch check so a response cannot leak onto the connection after the one that received its Ping. :::warning If a bound write times out after it starts, **delivery is undetermined** and the caller must not retry blindly. ::: ### Backend benchmarks The [WebSocket benchmark](../../crates/network/benches/BENCHMARKS.md) was measured on 2026-07-29. The following 512 B results are the median of three back-to-back runs on the same AMD Ryzen Threadripper 9980X host: | Metric | `tokio-tungstenite 0.30.0` | `sockudo-ws 2.0.1` | | --------------------------------- | -------------------------: | -----------------------: | | Round-trip text latency, p99 | 3.305 us | 0.651 us | | One-way binary burst latency, p99 | 17.647 us | 15.053 us | | Text receive throughput | 7.187 million messages/s | 8.504 million messages/s | | Text send throughput | 6.400 million messages/s | 7.207 million messages/s | | Text round-trip throughput | 0.530 million messages/s | 1.852 million messages/s | Across the measured 64 B, 512 B, and 4,096 B payloads, `sockudo-ws 2.0.1` reduced round-trip p99 latency by 73% to 82%. At 512 B it processed 18% more receives, 13% more sends, and 250% more round trips. These are backend frame-transport microbenchmarks over established, uncompressed 1 MiB in-memory Tokio duplex streams. They exclude DNS, TCP connect, TLS, HTTP upgrade, kernel network I/O, external latency, keepalive traffic, and the reconnecting client lifecycle. These WebSocket measurements do not cover HTTP or raw TCP clients, and their absolute values should only be compared on the same machine. ## Raw TCP socket client [`SocketClient`](../../crates/network/src/socket/client.rs) supports plain and TLS byte streams for protocols that delimit messages with a fixed suffix. A controller coordinates the connection, one reader splits inbound frames, and one writer appends the suffix while serializing concurrent sends. ```mermaid flowchart LR adapter[Adapter] subgraph client[SocketClient] controller[Controller] reconnect[Reconnect handle] reader["Reader task
split and strip suffix"] writer["Writer task
append suffix"] heartbeat[Heartbeat task] state[SocketStateSink] end replay[Reconnect replay] stream[Plain or TLS TCP stream] endpoint[TCP endpoint] adapter -->|send| writer adapter -->|request reconnect| reconnect --> controller reader -->|complete message| adapter controller -->|availability edges| state --> adapter controller -->|replace| reader controller -->|replace| writer heartbeat --> writer replay -->|before buffered sends| writer writer <--> stream --> reader stream <--> endpoint ``` The writer owns framing and replay order; the adapter receives complete messages without the configured suffix. ### Framing and liveness The suffix must contain at least one byte and applies in both directions. The reader retains a partial frame across reads and strips the suffix before invoking the callback. While the session remains active, it emits complete messages in arrival order. If an unterminated frame grows past 10 MiB, the reader stops and the controller reconnects instead of allowing unchecked memory growth. An optional heartbeat task sends a configured byte payload at a fixed interval; the writer appends the same suffix as it does for application messages. A raw socket has no Ping frames, so the payload is required. `heartbeat_timeout_secs` stops the reader when no bytes arrive within the window. Unset, it defaults to three intervals when a heartbeat is configured and leaves detection off otherwise. A zero timeout is rejected. The socket enables `TCP_NODELAY` to avoid Nagle delays for small protocol messages. ### Connection and TLS policy The client accepts `host:port` or URL input and supports plain or TLS mode. TLS uses the standard web PKI roots. A certificate directory can add trusted roots and, when it contains a matching certificate and private key, supply a client identity for mutual TLS. Initial connection establishment makes up to five attempts by default, with a 10-second bound per attempt and exponential backoff. Once connected, transport loss uses the configurable reconnect timeout, exponential backoff, bounded jitter, and unlimited attempts by default. As with the WebSocket client, 10 seconds of stable uptime resets the reconnect cycle, and the same reconnect throttling bounds its attempt rate once reconnects flap. An optional state sink reports semantic connection loss and recovery. ### State reporting and explicit reconnect The optional `SocketStateSink` has the same availability contract as the WebSocket client. It publishes `Connected` after successful initial connection, `Disconnected` when an active transport enters reconnect, and `Connected` after recovery. It omits initial failures, individual attempts, retry exhaustion, deliberate disconnect, and drop. Its synchronous callback must return promptly and must not request another transition through the same sink. `request_reconnect()` atomically asks the controller to replace an active plain or TLS transport. The cloneable `SocketReconnectHandle` lets adapter tasks make that request without owning the client and reports whether it was accepted or rejected because the client is already reconnecting, disconnecting, or closed. An accepted request publishes `Disconnected` before waking the controller; normal reconnect replay and buffer ordering then apply to the replacement. ### Replay and delivery boundaries During reconnect, the writer buffers application messages in FIFO order. After installing a replacement writer, it can first send protocol replay messages supplied by the adapter, such as a logon or session setup sequence, and then drain the buffered application messages. The replacement reader starts only after that drain succeeds. A post-reconnection callback runs after the writer, buffer, and reader are ready. ```mermaid sequenceDiagram participant C as Controller participant W as Writer task participant P as Replacement peer participant R as Reader task participant A as Adapter C->>C: Establish replacement connection C->>W: Install writer and optional replay batch opt replay is configured W->>P: Send protocol replay end W->>P: Drain buffered sends in FIFO order alt drain succeeds W-->>C: Confirm completion C->>R: Retire previous reader C->>C: Enter Active and publish Connected C->>R: Start replacement reader opt callback is configured C-->>A: Run post-reconnection callback end else send fails or times out W-->>C: Report failure C->>C: Keep reconnecting and retry end ``` A raw TCP replacement becomes active only after optional protocol replay and buffered application messages drain successfully; the reader and post-reconnection callback start afterward. :::warning `send_bytes` returns when the message enters the writer channel, not when the peer receives it. A concurrent disconnect can still prevent delivery. Reconnect replay and buffering are process memory, so protocols that require durable or exactly-once delivery must enforce those guarantees above the socket client. ::: ## TCP socket options The WebSocket and raw TCP socket clients apply the same options to every outbound connection, including the hop to an HTTP `CONNECT` proxy. The HTTP client uses a separate Hyper connector with `TCP_NODELAY`, keepalive after 15 seconds idle with 15 seconds between probes and three retries, and a 30-second `TCP_USER_TIMEOUT` on Linux, Android, and Fuchsia. The table below applies to WebSocket and raw TCP clients. | Option | Value | Detects or prevents | | ------------------ | ------------------------------- | --------------------------------------------------------- | | `TCP_NODELAY` | Enabled | Nagle delaying a small frame behind an unacknowledged one | | Keepalive | 20 s idle, 10 s apart, 3 probes | An idle peer that has gone away without closing | | `TCP_USER_TIMEOUT` | 1 minute, Linux only | Outbound data that is never acknowledged | These catch a connection that stops delivering without closing, which a NAT or load balancer produces when it drops state with no `FIN` and no `RST`. Writes keep succeeding into the send buffer and return `Ok` for messages the peer will never receive. Kernel defaults take roughly 15 minutes to give up; these bound that at about a minute. `TCP_USER_TIMEOUT` is sized to exceed the keepalive probe budget. On Linux it also overrides `TCP_KEEPCNT`, so detection there follows the timeout rather than the probe count, which applies on macOS and Windows. Treat them as a backstop. The heartbeat timeout usually fires first, and unlike it these need no configuration and still bound a connection whose reader task has stopped making progress. A socket that rejects an option is still usable, so failures are logged and the connection proceeds. ## Testing The network crate separates algorithm checks from operating-system I/O and simulated failure topologies. This keeps a failure local: a state-machine invariant should fail without a socket, wire behavior should fail against a small loopback peer, and reconnect races should fail under a reproducible network schedule. ### Unit and component tests Tests beside the implementation cover configuration validation, state transitions, rate limits, backoff, retry budgets, framing, transport conversion, authentication, subscription state, and reconnect buffer policy. Pure logic uses fake clocks and direct state models. Async task tests use paused Tokio time, in-memory duplex streams, injected transports, or an ephemeral loopback server so they can exercise the real reader, writer, heartbeat, and controller tasks without an external service. The client suites then test their own protocol boundary. HTTP tests cover request serialization, response headers and body limits, timeouts, proxy behavior, and URL redaction. WebSocket and raw TCP tests cover concurrent sends, liveness timeouts, framing, connection epochs, state sinks, explicit reconnect, replay order, and shutdown races. The shared TLS tests cover certificate loading and a complete mutual-TLS handshake. A separate loopback integration suite exercises the WebSocket HTTP `CONNECT` proxy path for plain `ws://` upstreams. ### Property tests [`proptest`](https://github.com/proptest-rs/proptest) generates values and operation traces for invariants that example cases cannot enumerate. The suites compare GCRA decisions with a reference model, check backoff and retry bounds, round-trip transport messages through both WebSocket backends, and exercise authentication, subscription, and reconnect-buffer state machines. Selected suites persist minimized failures in `crates/network/proptest-regressions` so a discovered case becomes a permanent regression test. ### Deterministic network simulation [`turmoil`](https://crates.io/crates/turmoil) tests compile the production raw TCP and WebSocket clients against simulated TCP types through the crate's `net` seam. Fixed seeds make failures reproducible. Stressed runs vary task order and message latency, while scenarios inject connection drops, partitions and repairs, stalled peers, handshake failures, and disconnects during backoff or recovery. Assertions cover eventual state, attempt limits, heartbeat behavior, message ordering, authentication gating, and clean shutdown. Separate suites exercise the Tungstenite and Sockudo backends over the same simulated protocol. Default tests keep real Tokio loopback networking and exclude the simulation-only suites. Enabling the `turmoil` feature swaps the TCP layer and includes those suites: ```bash cargo nextest run -p nautilus-network cargo nextest run -p nautilus-network --features turmoil ``` # Options Source: https://nautilustrader.io/docs/latest/concepts/options/ Nautilus provides first-class support for options trading across traditional and crypto markets. This includes option-specific instrument types, venue-provided Greeks streaming, option chain aggregation, and a local Black-Scholes Greeks calculator for risk management. ## Option instrument types The platform defines several option instrument types: | Instrument | Description | | -------------------- | ---------------------------------------------------------------------------- | | `OptionContract` | Exchange-traded option on an underlying with strike and expiry. | | `OptionSpread` | Exchange-defined multi-leg option strategy as one line. | | `CryptoOption` | Crypto option with crypto quote/settlement; inverse or quanto style. | | `CryptoOptionSpread` | Crypto option spread with inverse, settlement currency, and fractional size. | | `BinaryOption` | Fixed-payout option that settles to 0 or 1. | Greeks-relevant metadata varies by instrument type: - `OptionContract`, `CryptoOption`: full Greeks inputs including `strike_price`, `option_kind` (CALL/PUT), `expiration_ns`, `underlying`, `multiplier`. - `OptionSpread`, `CryptoOptionSpread`: an exchange-defined multi-leg strategy published as a single tradable instrument. Has `underlying`, `expiration_ns`, and `strategy_type` (a venue-defined code). The spread itself carries no `strike_price` or `option_kind`; venue-provided leg details are stored in `info` when the adapter supplies them. Orders execute against the spread as one line. `CryptoOptionSpread` additionally carries `is_inverse` and `settlement_currency` for venues like Deribit. - `BinaryOption`: has `expiration_ns` and `outcome`/`description`, but no `strike_price`, `option_kind`, or `underlying`. ## Subscribing to Greeks Venues like Deribit, Bybit, and OKX publish real-time Greeks alongside their options markets. Nautilus provides two subscription levels: - **Per-instrument Greeks**: subscribe to individual option contracts. - **Option chain slices**: subscribe to an aggregated view of an entire option series. ### Per-instrument Greeks Subscribe to venue-provided Greeks for a single option contract from an actor or strategy: ```python from nautilus_trader.model import ClientId client_id = ClientId("DERIBIT") self.subscribe_option_greeks(instrument_id, client_id=client_id) ``` Handle incoming updates by implementing the `on_option_greeks` handler: ```python def on_option_greeks(self, greeks) -> None: self.log.info( f"{greeks.instrument_id}: " f"delta={greeks.delta:.4f} gamma={greeks.gamma:.6f} " f"vega={greeks.vega:.4f} theta={greeks.theta:.4f} " f"mark_iv={greeks.mark_iv} underlying={greeks.underlying_price}" ) ``` To stop receiving updates: ```python self.unsubscribe_option_greeks(instrument_id, client_id=client_id) ``` ### Option chain subscriptions An option chain subscription aggregates quotes and Greeks across all strikes in an option series into `OptionChainSlice` snapshots. The `DataEngine` creates one Rust `OptionChainManager` per series and owns the lifecycle: creating the manager, routing incoming data, running snapshot timers, and draining wire subscription changes. ```python from nautilus_trader.model import OptionSeriesId from nautilus_trader.model import StrikeRange series_id = OptionSeriesId(...) # venue, underlying, settlement currency, expiry # Subscribe to 5 strikes above and below ATM, snapshot every 1000ms strike_range = StrikeRange.atm_relative(strikes_above=5, strikes_below=5) self.subscribe_option_chain( series_id, strike_range=strike_range, snapshot_interval_ms=1000, ) ``` Handle snapshots by implementing the `on_option_chain` handler: ```python def on_option_chain(self, chain) -> None: for strike in chain.strikes(): call = chain.get_call(strike) put = chain.get_put(strike) if call and call.greeks: self.log.info(f"Call {strike}: delta={call.greeks.delta:.4f}") ``` ### Strike range filtering `StrikeRange` controls which strikes are active in a chain subscription: | Variant | Description | Example | | ------------- | --------------------------------------------------- | -------------------------------- | | `Fixed` | Subscribe to an explicit set of strikes. | `StrikeRange.fixed([...])` | | `AtmRelative` | N strikes above and N below the current ATM strike. | `StrikeRange.atm_relative(5, 5)` | | `AtmPercent` | All strikes within a percentage band around ATM. | `StrikeRange.atm_percent(0.10)` | | `Delta` | Strikes whose call or put delta is near a target. | `StrikeRange.delta(0.25, 0.05)` | For dynamic strike ranges, subscriptions are **deferred until the ATM price is determined**. ATM is derived from the venue reference price in `OptionGreeks.underlying_price`. It can also be seeded from a reference price fetched for the option series via HTTP, allowing instant bootstrap before live WebSocket ticks arrive. As ATM shifts, the active strike set rebalances automatically. `Delta` resolves from venue-provided Greeks: a strike is active when its call or put delta magnitude (calls positive, puts negative, compared by absolute value) falls within `tolerance` of `target`. A typical out-of-the-money target such as `0.25` selects a strike on each side of ATM. Before the ATM reference price is known, `Delta` is deferred like other dynamic ranges. After ATM is known, when no active strike's Greeks match the band (including before any Greeks arrive), `Delta` falls back to an ATM-relative window of five strikes either side of ATM. Before switching from the fallback window to selected delta strikes, the aggregator waits until every fallback leg has Greeks so partial early updates do not drop neighboring strikes. ### Snapshot vs. raw mode The `snapshot_interval_ms` parameter controls publishing behavior: - **Snapshot mode** (`snapshot_interval_ms=1000`): Quotes and Greeks accumulate in a buffer and publish as an `OptionChainSlice` on a timer. Suitable for periodic portfolio rebalancing or UI display. - **Raw mode** (`snapshot_interval_ms=None`): Each quote or Greeks update for an active instrument publishes a slice immediately. Suitable for latency-sensitive strategies that react to individual updates. ## Backtesting option chains Option-chain backtests use the same `OptionChainManager` and `OptionChainAggregator` path as live subscriptions. The prerequisite is a Nautilus Parquet catalog that already contains the option instruments and the per-instrument data needed for the chain: - `QuoteTick` records for each option contract, carrying the replayed best bid and offer. - `OptionGreeks` records for each option contract, carrying delta, implied volatility, convention, and the `underlying_price` used to seed ATM. - `CryptoOption` or `OptionContract` instruments for the same instrument IDs. Tardis replays satisfy this contract when option book snapshots or quotes are written as `QuoteTick` and `option_summary` messages are written as `OptionGreeks`. The backtest does not download or request missing catalog data during the run. Configure a `BacktestNode` run with both data streams for the option instruments in the series: ```python data = [ BacktestDataConfig( data_type="QuoteTick", catalog_path="/path/to/catalog", instrument_ids=option_instrument_ids, ), BacktestDataConfig( data_type="OptionGreeks", catalog_path="/path/to/catalog", instrument_ids=option_instrument_ids, ), ] ``` Then subscribe from the strategy: ```python strike_range = StrikeRange.delta(0.25, 0.05) self.subscribe_option_chain( series_id, strike_range=strike_range, snapshot_interval_ms=1000, ) ``` Use `snapshot_interval_ms=None` for raw mode. Raw mode publishes a slice after each quote or Greeks update for an active instrument. Use an integer interval for thinned snapshots. Thinned mode accumulates the latest BBO and Greeks per instrument and publishes the chain on the timer cadence, reducing event volume for large chains. Each `OptionChainSlice` joins the latest BBO and Greeks by instrument, then groups the result by strike and option kind. A quote can arrive before Greeks, and Greeks can arrive before a quote; the aggregator keeps latest state and attaches both when available. The `underlying_price` in `OptionGreeks` drives ATM detection. Selection can happen either in the subscription range or inside the strategy: - Moneyness: use `StrikeRange.atm_relative(...)` or `StrikeRange.atm_percent(...)`. - Delta: use `StrikeRange.delta(target, tolerance)`, or inspect `entry.greeks.delta` in `on_option_chain`. - Strike: use `StrikeRange.fixed([...])`, or read `chain.get_call(strike)` and `chain.get_put(strike)`. Matching is quote-driven for options. Market orders and marketable limits fill as takers against the opposing replayed BBO. Passive limit orders rest on the simulated book and can fill as makers when later BBO updates trade through the limit price. The model does not simulate L2 queue position for options. Structural option fee models are configured on the simulated venue, not inferred from the venue name: ```python from decimal import Decimal from nautilus_trader.execution import CappedOptionFeeModel from nautilus_trader.execution import TieredNotionalOptionFeeModel deribit_like = CappedOptionFeeModel( maker_rate=Decimal("0.0003"), taker_rate=Decimal("0.0003"), ) okx_like = TieredNotionalOptionFeeModel( maker_rate=Decimal("0.0002"), taker_rate=Decimal("0.0005"), ) ``` Pass one of these objects as `fee_model` on `BacktestVenueConfig`. The Rust surface uses `FeeModelAny::CappedOption(CappedOptionFeeModel::new(...))` and `FeeModelAny::TieredNotionalOption(TieredNotionalOptionFeeModel::new(...))`. See `examples/backtest/tardis_option_chain.py` and the Rust `tardis-option-chain` example in `crates/backtest/examples/`. ## Option chain architecture The option chain system is event-driven and built around per-series isolation. The `DataEngine` creates one Rust `OptionChainManager` per subscribed option series. The manager wraps `OptionChainAggregator` and `AtmTracker`, registers message bus handlers, publishes snapshots, and queues wire subscription changes for the engine to drain. ```mermaid flowchart TD subgraph DataEngine DE[DataEngine] end subgraph "OptionChainManager (per series)" MGR[OptionChainManager] AGG[OptionChainAggregator] ATM[AtmTracker] TMR[SnapshotTimer] end DC[DataClient] -- QuoteTick --> DE DC -- OptionGreeks --> DE DE -- "handle_quote()" --> MGR DE -- "handle_greeks()" --> MGR MGR --> AGG MGR --> ATM ATM -- "reference price" --> AGG TMR -- "timer tick" --> MGR MGR -- "OptionChainSlice" --> MB((MessageBus)) MB -- "on_option_chain" --> S[DataActor / Strategy] DE -- "sub/unsub" --> DC ``` ### Component responsibilities #### DataEngine Holds one `OptionChainManager` per active `OptionSeriesId`. On `SubscribeOptionChain`, it resolves instruments from the cache, requests a series reference price for dynamic strike ranges, creates the manager, subscribes active instruments to the data client, and sets up the snapshot timer. On each timer tick, the manager checks for rebalances, publishes a snapshot, and queues any wire subscription changes for the engine to drain. On `UnsubscribeOptionChain` or when all instruments expire, it tears down the manager, cancels the timer, and unsubscribes wire-level feeds. #### OptionChainManager A per-series Rust manager around `OptionChainAggregator` and `AtmTracker`. The `DataEngine` feeds it market data through `handle_quote()` and `handle_greeks()`. In snapshot mode, timer callbacks call `publish_slice()`. In raw mode, each active quote or Greeks update calls `publish_slice()` immediately. The manager bootstraps the active instrument set internally on the first ATM price. #### OptionChainAggregator Accumulates quotes and Greeks into call/put buffers using **keep-latest semantics**. Instruments that did not update since the last snapshot are still included. Greeks that arrive before any quote for an instrument are held in a `pending_greeks` buffer and attached when the first quote arrives. On each `snapshot()` call, the aggregator produces an immutable `OptionChainSlice`. #### AtmTracker Derives the ATM price reactively from the `underlying_price` field in incoming `OptionGreeks` events. It can be pre-seeded from an HTTP reference price for the option series, allowing instant bootstrap without waiting for WebSocket ticks. ### Bootstrap and rebalancing For dynamic strike ranges (`AtmRelative`, `AtmPercent`, and `Delta`), the active instrument set cannot be determined until the ATM price is known. There are two bootstrap paths: **Instant bootstrap (reference price available):** 1. `DataEngine` receives `SubscribeOptionChain`, resolves all instruments for the series from the cache, and requests a reference price from the data client. 2. When the reference price response arrives, the engine creates the manager with the ATM price pre-seeded. The manager computes the active strike set during construction. 3. The engine subscribes the active instruments immediately. **Deferred bootstrap (no reference price):** 1. The engine has no matching client or cached option instrument, the client reports no reference price, the request fails, or the request times out after 30 seconds. 2. The engine creates the manager with no initial ATM price. The active set is empty. When the request reached a client with a cached sample option, the engine subscribes that sample's Greeks as the bootstrap source. Without a client or sample, bootstrap still depends on relevant Greeks data already flowing from another subscription. 3. When the engine feeds an `OptionGreeks` event with `underlying_price` through `handle_greeks()`, the manager bootstraps the active instrument set, registers message bus handlers, and queues the new wire subscriptions for the engine to drain. The sample subscription becomes part of the active set or is released. Once bootstrapped, the aggregator monitors ATM drift. On each snapshot timer tick, the manager calls the aggregator's `check_rebalance()` which returns any instruments to add or remove. A hysteresis threshold and cooldown period prevent thrashing near strike boundaries. ## OptionGreeks data type `OptionGreeks` carries venue-provided sensitivities and implied volatility for a single option contract: | Field | Type | Description | | ------------------ | ------------------ | --------------------------------------------------- | | `instrument_id` | `InstrumentId` | The option contract these Greeks apply to. | | `convention` | `GreeksConvention` | Numeraire convention for the Greeks. | | `delta` | `float` | Rate of change of option price per unit underlying. | | `gamma` | `float` | Rate of change of delta per unit underlying. | | `vega` | `float` | Venue-reported vega. | | `theta` | `float` | Venue-reported theta. | | `rho` | `float` | Venue-reported rho; defaults to zero. | | `mark_iv` | `float` or None | Mark implied volatility. | | `bid_iv` | `float` or None | Bid implied volatility. | | `ask_iv` | `float` or None | Ask implied volatility. | | `underlying_price` | `float` or None | Underlying price at time of calculation. | | `open_interest` | `float` or None | Open interest for the contract. | | `ts_event` | `int` | UNIX timestamp (nanoseconds) of the event. | | `ts_init` | `int` | UNIX timestamp (nanoseconds) when initialized. | ## OptionChainSlice data type `OptionChainSlice` is a point-in-time snapshot of an entire option series. Properties: | Property | Type | Description | | ------------ | ---------------- | ----------------------------------- | | `series_id` | `OptionSeriesId` | The option series identifier. | | `atm_strike` | `Price` or None | Current ATM strike (if determined). | | `ts_event` | `int` | UNIX timestamp (nanoseconds). | | `ts_init` | `int` | UNIX timestamp (nanoseconds). | Call and put data are accessed through methods, not as direct properties. Each `OptionStrikeData` returned by these methods contains a `quote` (`QuoteTick`) and an optional `greeks` (`OptionGreeks`) for that strike. Methods: - `strikes()`: all unique strike prices in the chain. - `strike_count()`, `call_count()`, `put_count()`: counts. - `get_call(strike)`, `get_put(strike)`: full `OptionStrikeData`. - `get_call_greeks(strike)`, `get_put_greeks(strike)`: Greeks only. - `get_call_quote(strike)`, `get_put_quote(strike)`: quote only. - `is_empty()`: true if the chain has no data. ## Adapter support The following adapters support option Greeks subscriptions: | Adapter | Per-instrument Greeks | Option chains | | ------------------- | --------------------- | ------------- | | Deribit | Yes | Yes | | Bybit | Yes | Yes | | Derive | Yes | Yes | | Interactive Brokers | Yes | Yes | | OKX | Yes | Yes | ## See also - [Greeks](greeks.md) - Local Greeks calculation and portfolio risk management. - [Data](data/) - Built-in data types and the subscription model. - [Actors](actors.md) - Subscription and handler reference table. # Order Book Source: https://nautilustrader.io/docs/latest/concepts/order_book/ NautilusTrader implements its order books in Rust. `OrderBook` maintains public market depth for an instrument. `OwnOrderBook` tracks your own orders separately so filtered views can subtract them from public liquidity. :::note This guide uses the Rust model API for book operations. Subscription and handler examples use the Python strategy and actor API. Python exposes the book types as `nautilus_trader.model.OrderBook` and `nautilus_trader.model.OwnOrderBook`; see the [model API reference](/docs/python-api-latest/model/book.html) for the Python interface. ::: ## Book types `OrderBook` instances are maintained per instrument for both backtesting and live trading: - `L3_MBO`: Level 3 market-by-order (MBO) data. Tracks every order at every price level, keyed by order ID. On each book side, an order ID maps to exactly one price level: re-adding an ID at a different price moves the order to the new level. MBP-style input uses a price-derived ID. A zero order ID likewise signals missing identity, except that top-of-book input uses the order side as its ID. - `L2_MBP`: Level 2 market-by-price (MBP) data. Aggregates orders by price level (one entry per price). - `L1_MBP`: Level 1 market-by-price (MBP) top-of-book data, also known as best bid and offer (BBO). Captures only the best prices. :::note Quote, trade, and bar data (`QuoteTick`, `TradeTick`, and `Bar`) can also drive `L1_MBP` books. ::: ## Subscribing to book data Strategies and actors subscribe to order book updates through the following methods. Subscriptions and handlers are part of the Python strategy/actor layer: ```python from nautilus_trader.model import BookType from nautilus_trader.model import OrderBook from nautilus_trader.model import OrderBookDeltas from nautilus_trader.model import OrderBookDepth10 # Incremental book deltas self.subscribe_book_deltas(instrument_id, BookType.L2_MBP) # Aggregated depth snapshots (up to 10 levels) self.subscribe_book_depth10(instrument_id, BookType.L2_MBP) # Full book snapshots at a timed interval self.subscribe_book_at_interval(instrument_id, BookType.L2_MBP, interval_ms=1000) ``` Each subscription type delivers data to the corresponding handler: ```python def on_book_deltas(self, deltas: OrderBookDeltas) -> None: ... def on_book_depth(self, depth: OrderBookDepth10) -> None: ... def on_book(self, order_book: OrderBook) -> None: ... ``` ## Accessing the book The `OrderBook` exposes top-of-book accessors: ```rust let best_bid: Option = book.best_bid_price(); let best_ask: Option = book.best_ask_price(); let spread: Option = book.spread(); let midpoint: Option = book.midpoint(); ``` ## Analysis methods The `OrderBook` supports market depth analysis and execution simulation: ```rust // Average fill price for a given quantity let avg_fill_px = book.get_avg_px_for_quantity(quantity, OrderSide::Buy); // Average price, filled quantity, and worst price for a target exposure let (avg_px, filled_qty, worst_px) = book.get_avg_px_qty_for_exposure(target_exposure, OrderSide::Buy); // Cumulative quantity available at or better than a price let qty = book.get_quantity_for_price(price, OrderSide::Buy); // Quantity at a specific price level only let qty = book.get_quantity_at_level(price, OrderSide::Buy, 2); // Simulate fills against the book let fills: Vec<(Price, Quantity)> = book.simulate_fills(&order); // All crossed levels regardless of order quantity let levels = book.get_all_crossed_levels(OrderSide::Buy, price, 2); ``` ## Integrity checks Call `book_check_integrity` to validate that the book state is consistent with its type: - **L1_MBP**: No more than one level per side. - **L2_MBP**: No more than one order per price level. - **L3_MBO**: No additional per-level constraint; multiple orders may share a price. - **All types**: Best bid must not exceed best ask (crossed book). Locked markets (bid == ask) are considered valid. This is an **explicit check**: applying a delta does not call it. The Rust `apply_delta` and `apply_deltas` methods separately validate the incoming instrument ID against the book and return `BookIntegrityError::InstrumentMismatch` on mismatch. For a nonzero order ID, a delta whose side is `None` first tries to resolve the side from the ladder cache. If no side is cached, an `Add` returns `BookIntegrityError::NoOrderSide`, while an `Update` or `Delete` is skipped. If the ID exists on both sides, an `Add` returns `BookIntegrityError::AmbiguousOrderSide`, while an `Update` or `Delete` is skipped with a warning. Out-of-order deltas and depth snapshots are **applied rather than rejected**, so a venue that replays or reorders events still reaches the state those events describe. Only the book metadata is protected: `sequence` and `ts_last` are high-water marks and never regress. A stale update logs one warning for each field that regressed, `sequence` and `ts_event` independently, and how often it logs depends on how the update arrives: - **Incremental deltas**: Once per stale delta. - **Snapshot deltas**: Once per snapshot, whether it arrives as an `F_SNAPSHOT` batch or as a single `F_SNAPSHOT` delta, since every delta in a rebuild shares the snapshot's sequence and timestamp. - **Depth snapshots**: Once, since an `OrderBookDepth10` replaces the book in a single update. A snapshot report describes the incoming snapshot, so it does not depend on whether each of its deltas reaches the book. An `L1_MBP` book driven by quotes or trades is the exception to all of this: a stale `QuoteTick` or `TradeTick` is skipped with a warning and leaves the book unchanged. ## Pretty printing Both `OrderBook` and `OwnOrderBook` provide a `pprint` method that returns the book as a human-readable table: ```rust println!("{}", book.pprint(5, None)); println!("{}", book.pprint(5, Some(Decimal::new(1, 2)))); // group_size = 0.01 ``` The `group_size` parameter buckets price levels into coarser groups for instruments with fine tick sizes. The output is a formatted table with bids on the left, prices in the center, and asks on the right. ## Own order book The `OwnOrderBook` tracks your own working orders separately from the public book. Market making and other quoting strategies use it to estimate available liquidity at each price level after subtracting their own orders. Execution engines maintain own books when `manage_own_order_books` is enabled. The cache updates an existing own book as order events change state. Eligible orders have a price and do not use `IOC` or `FOK` time in force. Terminal events may still clean up an existing own book entry, even when the order would not otherwise be eligible for tracking. ### Order lifecycle The `OwnOrderBook` tracks orders through their lifecycle. Orders are added during submission or materialized from reconciliation. Nonterminal states such as `OrderStatus::Accepted`, `OrderStatus::PendingUpdate`, `OrderStatus::PendingCancel`, and `OrderStatus::PartiallyFilled` update the entry. The closed states `OrderStatus::Denied`, `OrderStatus::Rejected`, `OrderStatus::Canceled`, `OrderStatus::Expired`, `OrderStatus::Filled`, and `OrderStatus::Voided` remove it. Each `OwnBookOrder` carries: - `trader_id`: Trader ID that owns the order. - `client_order_id`: Client order ID used to reconcile the own book with cache state. - `venue_order_id`: Venue order ID when one has been assigned. - `side`, `price`, and `size`: Order side, price, and remaining (leaves) quantity. - `order_type` and `time_in_force`: Order metadata retained for inspection. - `status`: Current order status, such as `SUBMITTED`, `ACCEPTED`, or `PENDING_CANCEL`. - `ts_last`: Timestamp of the latest order event applied to this own-book order. - `ts_accepted`: Timestamp when the venue accepted the order, or zero before acceptance. - `ts_submitted`: Timestamp when the order was submitted, or zero before submission. - `ts_init`: Timestamp when the order was initialized. The `status` and `ts_accepted` fields drive the optional filters described in [Status and time filtering](#status-and-time-filtering). ### Auditing The `audit_open_orders` method reconciles an own book against a set of valid client order IDs. Any own-book order not in the provided set is removed and logged as an audit error. `Cache::audit_own_order_books` builds this set from open, in-flight, and active-local orders so non-terminal entries remain during normal event-processing and venue-latency windows. Live systems can run this audit periodically through the own-books audit interval. ### Querying ```rust // Check if a specific order is tracked let in_book = own_book.is_order_in_book(&client_order_id); // Get all tracked order IDs per side let bid_ids = own_book.bid_client_order_ids(); let ask_ids = own_book.ask_client_order_ids(); // Aggregated quantities per price level let bid_qty = own_book.bid_quantity(None, None, None, None, None); let ask_qty = own_book.ask_quantity(None, None, None, None, None); // Pretty print println!("{}", own_book.pprint(5, None)); ``` ### Filtered views Subtract your own orders from the public book to see net available liquidity: ```rust // Filtered maps of price -> quantity (own orders subtracted) let net_bids = book.bids_filtered_as_map(Some(10), Some(&own_book), None, None, None); let net_asks = book.asks_filtered_as_map(Some(10), Some(&own_book), None, None, None); // Full filtered OrderBook with all analysis methods available let filtered = book.filtered_view(Some(&own_book), Some(10), None, None, None); let avg_px = filtered.get_avg_px_for_quantity(quantity, OrderSide::Buy); ``` The `filtered_view` method returns a new `OrderBook` with your own sizes subtracted, giving access to the full set of analysis methods (`spread`, `midpoint`, `get_avg_px_for_quantity`, etc.) on the net book. ### Status and time filtering Filtered views support optional status and time-based filtering for own orders: ```rust let statuses = AHashSet::from([OrderStatus::Accepted]); // Only subtract ACCEPTED orders (ignore SUBMITTED, PENDING_CANCEL, etc.) let filtered = book.filtered_view(Some(&own_book), None, Some(&statuses), None, None); ``` The `accepted_buffer_ns` parameter provides a grace period. When `ts_now` is set, the view includes an own order only when `ts_accepted + accepted_buffer_ns <= ts_now`. This excludes recently accepted orders that may not yet appear in the public book feed. The time check applies regardless of order status, so combine it with a status filter to exclude non-accepted orders. Omitting `ts_now` disables acceptance-time filtering, and a positive `accepted_buffer_ns` requires `ts_now`. ```rust // Only subtract orders accepted at least 500ms ago let filtered = book.filtered_view( Some(&own_book), None, None, Some(500_000_000), Some(clock.timestamp_ns().as_u64()), ); ``` ## Binary markets Binary markets can expose complementary outcome instruments, such as Polymarket YES and NO tokens. For a known complementary pair, the parity transform maps a price `p` on one outcome to `1 - p` on the other. Under this transform, a NO bid at 0.40 becomes a YES ask at 0.60. The `OwnOrderBook::combined_with_opposite` method handles this transformation, merging orders from both outcome instruments into a view for the first book: ```rust let yes_own = own_yes_book .cloned() .unwrap_or_else(|| OwnOrderBook::new(yes_instrument_id)); let no_own = own_no_book .cloned() .unwrap_or_else(|| OwnOrderBook::new(no_instrument_id)); // Merge NO orders with the parity price transform (1 - price) let combined = yes_own.combined_with_opposite(&no_own).unwrap(); // Filter the public YES book using the combined own book let filtered = book.filtered_view(Some(&combined), None, None, None, None); ``` The transformation works as follows: - NO asks at price `p` become bids at price `1 - p` in the combined book. - NO bids at price `p` become asks at price `1 - p` in the combined book. :::warning The method rejects matching instrument IDs, but it cannot verify that the two instruments are complementary. The caller must supply the actual opposite instrument. The resulting own book can filter the public YES book against your orders in either outcome instrument. ::: # Overview Source: https://nautilustrader.io/docs/latest/concepts/overview/ ## Introduction NautilusTrader is an open-source, production-grade, Rust-native engine for multi-asset, multi-venue trading systems. The system spans research, deterministic simulation, and live execution within a single event-driven architecture, with Python serving as the control plane for strategy logic, configuration, and orchestration. This separation provides the performance and safety of a compiled trading engine with the flexibility of Python for system composition and strategy development. Trading systems can also be written entirely in Rust for mission-critical workloads. The same strategy and execution-algorithm code can run across backtest and live systems, reducing deployment divergence. Live execution still introduces venue, transport, timing, persistence, external-activity, and reconciliation behavior that a simulation may not reproduce. See [Backtest and live differences](live.md#backtest-and-live-differences). NautilusTrader is asset-class-agnostic. Any venue with a REST API or WebSocket feed can be integrated through modular adapters. Integrations span centralized and decentralized crypto exchanges (CEX and DEX), foreign exchange (FX), equities, futures, options, and betting exchanges. ## Features - **Fast**: Rust core with asynchronous networking using [tokio](https://crates.io/crates/tokio). - **Reliable**: Rust provides type safety and thread safety, with optional state persistence backed by Redis or PostgreSQL. - **Portable**: Runs on Linux, macOS, and Windows. Deploy using Docker. - **Flexible**: Modular adapters integrate any REST API or WebSocket feed. - **Advanced orders**: Time-in-force options include `IOC`, `FOK`, `GTC`, `GTD`, `DAY`, `AT_THE_OPEN`, and `AT_THE_CLOSE`. The domain model also supports conditional triggers, `post-only`, `reduce-only`, iceberg, and `OCO`, `OUO`, and `OTO` contingency orders. Venue support varies by adapter. - **Customizable**: User-defined components, or assemble entire systems from scratch using the [cache](cache.md) and [message bus](message_bus.md). - **Backtesting**: Run multiple venues, instruments, and strategies simultaneously using historical quotes, trades, bars, order books, and custom data with nanosecond resolution. - **Live**: Identical strategy implementations between research and live deployment. - **Multi-venue**: Run market-making and cross-venue strategies across multiple venues simultaneously. - **AI training**: High-throughput simulation supports workloads such as training AI trading agents with reinforcement learning (RL) or evolutionary strategies (ES). ## Why NautilusTrader? Trading strategy research typically happens in Python using vectorized approaches, while production trading systems are built separately using event-driven architectures in compiled languages. NautilusTrader removes this separation. A Rust-native core provides a deterministic event-driven runtime for both research and live execution, while Python serves as the control plane. The same architecture, execution semantics, and time model operate across both environments, allowing strategies to move from research to production without reimplementation. Python bindings for the Rust-native runtime are provided via [PyO3](https://pyo3.rs). Installing an official prebuilt Python wheel does not require a Rust toolchain. ## Use cases NautilusTrader supports three main use cases: - Backtest trading systems on historical data (`backtest`). - Simulate trading systems with real-time data and virtual execution (`sandbox`). - Deploy trading systems live on real or paper accounts (`live`). NautilusTrader provides backtest and live node implementations for both Python and Rust. The sandbox adapter supplies simulated execution for a `sandbox` environment. :::note - Examples use these node implementations unless stated otherwise. - A trading strategy is one component of an end-to-end trading system, which also includes application and infrastructure layers. ::: ## Distributed The platform integrates into larger distributed systems. The [external message bus](message_bus.md#encoding) supports JSON and MessagePack payloads, plus Cap'n Proto and Simple Binary Encoding (SBE) for schema-covered market data. Apache Arrow and Parquet provide columnar interchange and persistence through the [data catalog](data/index.md#data-catalog). Format support varies by payload type. ## Common core The common system core is used by all node [environment contexts](architecture.md#environment-contexts): `backtest`, `sandbox`, and `live`. User-defined actors, strategies, and execution algorithms use the same lifecycle across these contexts. ## Backtesting Feed data to a `BacktestEngine` either directly or through a higher-level `BacktestNode` and `ParquetDataCatalog`, then run the data through the system with nanosecond resolution. See [Backtesting](backtesting/) for the APIs and execution model. ## Live trading A `LiveNode` ingests data and events from multiple data and execution clients, supporting demo, paper, and real accounts. The Rust-native node, including its PyO3 interface, runs the kernel event loop on the calling thread, while asynchronous I/O and background tasks use a shared multi-threaded Tokio runtime. See [Live trading](live.md) for the node lifecycle and risk considerations, and [Execution reconciliation](execution/reconciliation.md) for state recovery. ## Domain model The trading domain model includes [value types](value_types.md) such as `Price` and `Quantity`, plus [orders](orders/) and [positions](positions.md) that aggregate events to determine state. ## Timestamps NautilusTrader represents system timestamps as **UNIX nanoseconds**. Its standard ISO 8601 (RFC 3339) formatter uses UTC and preserves all nine fractional digits. A millisecond formatter preserves three fractional digits for selected displays, such as good-till-date (GTD) expiry times. A timestamp string consists of: - Full date component always present: `YYYY-MM-DD`. - `T` separator between date and time components. - Nine fractional digits for nanosecond output, or three for millisecond output. - UTC timezone designated by the `Z` suffix. Example: `2024-01-05T15:30:45.123456789Z` For the complete specification, refer to [RFC 3339: Date and Time on the Internet](https://datatracker.ietf.org/doc/html/rfc3339). ## UUIDs The `UUID4` value type provides random Universally Unique Identifier (UUID) version 4 values for events, commands, reports, and other internal messages. It uses the `uuid` crate to validate version and variant bits when parsing strings. A valid UUID v4 under RFC 9562 consists of: - 32 hexadecimal digits displayed in 5 groups. - Groups separated by hyphens: `8-4-4-4-12` format. - Version 4 designation (indicated by the third group starting with "4"). - IETF variant designation (indicated by the fourth group starting with "8", "9", "a", or "b"). Example: `2d89666b-1a1e-4a75-b193-4eb3b454c757` For the complete specification, see [RFC 9562: Universally Unique Identifiers (UUIDs)](https://www.rfc-editor.org/rfc/rfc9562.html). ## Data types NautilusTrader defines the following built-in market and reference data types. Availability for historical requests and live subscriptions depends on the provider and adapter. See [Data](data/index.md) for their fields and behavior. - `OrderBookDelta` (single order book change) - `OrderBookDeltas` (container type) - `OrderBookDepth10` (fixed depth of 10 levels per side) - `QuoteTick` - `TradeTick` - `Bar` - `MarkPriceUpdate` - `IndexPriceUpdate` - `FundingRateUpdate` - `OptionGreeks` - `Instrument` - `InstrumentStatus` - `InstrumentClose` Use [custom data](custom_data.md) for application-specific types. The following `PriceType` options select granular data for internal bar aggregation: - `BID` - `ASK` - `MID` - `LAST` `BID`, `ASK`, and `MID` use `QuoteTick` data, while `LAST` uses `TradeTick` data. Composite bar types aggregate smaller bars instead. ## Bar aggregations The following `BarAggregation` methods are available: - `MILLISECOND` - `SECOND` - `MINUTE` - `HOUR` - `DAY` - `WEEK` - `MONTH` - `YEAR` - `TICK` - `VOLUME` - `VALUE` (also known as dollar bars) - `RENKO` (price-based bricks) - `TICK_IMBALANCE` - `TICK_RUNS` - `VOLUME_IMBALANCE` - `VOLUME_RUNS` - `VALUE_IMBALANCE` - `VALUE_RUNS` All listed aggregations are implemented for internal aggregation. Information-driven aggregations require `TradeTick` data. A `BarSpecification` combines a price type, aggregation method, and positive step size. Fixed-subunit time bars have divisibility limits; see [Bar types](data/index.md#bar-types) for the validation rules. Internal aggregation can run during live trading when the required input data is available. ## Account types The [accounting engine](accounting.md#account-types) supports the following configurations in both live and backtest environments: - `Cash` single-currency (base currency) - `Cash` multi-currency - `Margin` single-currency (base currency) - `Margin` multi-currency - `Betting` single-currency - `Wallet` multi-currency (blockchain wallets; execution client in development) ## Order types The [order model](orders/) supports the following types, subject to venue adapter support: - `MARKET` - `LIMIT` - `STOP_MARKET` - `STOP_LIMIT` - `MARKET_TO_LIMIT` - `MARKET_IF_TOUCHED` - `LIMIT_IF_TOUCHED` - `TRAILING_STOP_MARKET` - `TRAILING_STOP_LIMIT` ## Value types The following fixed-point value types are backed by either 128-bit or 64-bit raw integers, depending on the [precision mode](../getting_started/installation.md#precision-mode) used during compilation. - `Price` - `Quantity` - `Money` Official Python wheels use high-precision mode on all supported platforms. Pure Rust builds default to standard precision unless the `high-precision` feature is enabled. ### High-precision mode (128-bit) When the `high-precision` feature flag is **enabled**, values use the specification: | Type | Raw backing | Max precision | Min value | Max value | | :--------- | :---------- | :------------ | :------------------ | :----------------- | | `Price` | `i128` | 16 | -17,014,118,346,046 | 17,014,118,346,046 | | `Money` | `i128` | 16 | -17,014,118,346,046 | 17,014,118,346,046 | | `Quantity` | `u128` | 16 | 0 | 34,028,236,692,093 | ### Standard-precision mode (64-bit) When the `high-precision` feature flag is **disabled**, values use the specification: | Type | Raw backing | Max precision | Min value | Max value | | :--------- | :---------- | :------------ | :------------- | :------------- | | `Price` | `i64` | 9 | -9,223,372,036 | 9,223,372,036 | | `Money` | `i64` | 9 | -9,223,372,036 | 9,223,372,036 | | `Quantity` | `u64` | 9 | 0 | 18,446,744,073 | # Portfolio Source: https://nautilustrader.io/docs/latest/concepts/portfolio/ The Portfolio maintains account and position-derived state for a trading node or backtest. Strategies use it to query accounts, PnL, exposure, margin, equity, and performance statistics. ## Currency conversion The Python `Portfolio` can convert PnL and exposure from native cost currencies to an account base currency or an explicit target currency. This supports instruments with different cost currencies and accounts with different base currencies. ### Supported conversions Currency conversion is available for the following portfolio queries: - `realized_pnl()` and `realized_pnls()` convert realized PnL. - `unrealized_pnl()` and `unrealized_pnls()` convert unrealized PnL. - `total_pnl()` and `total_pnls()` convert total PnL. - `net_exposure()` and `net_exposures()` convert net exposure. All eight methods accept an optional `target_currency`. A successful targeted query contains only that currency. The Portfolio converts each native value directly to the target, even when an account has a different base currency. ### Single account behavior With `PortfolioConfig.convert_to_account_base_currency=true` (the default), a query for one account without `target_currency` converts values to the account's base currency when it has one. Otherwise, the result remains in its native cost currency. ```python # Returns exposure in the account's base currency (e.g., USD) exposure = portfolio.net_exposures(venue=BINANCE, account_id=account_id) ``` ### Multi-account behavior When querying multiple accounts without `target_currency`, the output depends on whether the method returns a currency map or one `Money` value: - Collection methods can return a dictionary with one entry per output currency, using account base currencies where configured and native cost currencies otherwise. Provide `target_currency` to aggregate the complete result into one currency. - Single-value methods return `None` if values from different accounts cannot resolve to one output currency. `net_exposures()` also returns `None` if one instrument spans accounts with different base currencies because its per-instrument exposure cannot resolve to one output currency. ```python # Multiple accounts with the same base currency exposures = portfolio.net_exposures(venue=BINANCE) # Returns {USD: Money(...)} # Accounts with different base currencies and separately resolvable instruments exposures = portfolio.net_exposures(venue=BINANCE) # Returns {USD: Money(...), EUR: Money(...)} # Force single currency across accounts exposures = portfolio.net_exposures(venue=BINANCE, target_currency=USD) # Returns {USD: Money(...)} ``` ### Calculation failures PnL and exposure queries fail closed when any required price, xrate, or exact arithmetic operation is unavailable. Their Python behavior depends on the method type: - Single-value methods (`realized_pnl`, `unrealized_pnl`, `total_pnl`, and `net_exposure`) return `None`. - `realized_pnls`, `unrealized_pnls`, and `total_pnls` raise `RuntimeError`. - `net_exposures` returns `None`. Collection queries **fail as one unit**. They never return a partial result or combine target and source currencies. For example, one unpriced instrument invalidates the whole `unrealized_pnls` or `total_pnls` result. A valid all-scope `net_exposures()` query returns `{}` when the portfolio is flat. :::warning Exchange rate data must be available when using `target_currency` for cross-currency aggregation. ::: ### Conversion price types Position valuation prefers a current `MARK` price when `use_mark_prices` is enabled. Otherwise, it uses `BID` for a long position and `ASK` for a short position before trying the remaining price fallbacks. Currency conversion uses a current `MID` xrate from the cache. If `use_mark_xrates` is enabled, a current `MARK` xrate takes precedence and `MID` remains the fallback. Explicit target-currency queries do not reuse a carried stale xrate. ### Exposure aggregation `net_exposure()` values each open position for one instrument before adding long notional and subtracting short notional. It returns the magnitude of that net valued notional, so the result does not retain direction. A caller-supplied `price` values every selected position at the same price. Without an override or common mark price, side-specific bid and ask prices can leave a valuation residual for equal opposing quantities. `net_exposures()` groups and sums nonzero per-instrument magnitudes by output currency. It does not net directional exposure between different instruments. ### Price overrides The Python methods `unrealized_pnl`, `total_pnl`, and `net_exposure` accept an optional `price`. When supplied, the Portfolio values the selected instrument at that price instead of reading a cached market price. The calculation is fresh: it does not replace the cached PnL, exposure, or market price used by later queries. ## Equity and mark-to-market The Portfolio exposes pull-style queries for continuous portfolio valuation and recorded snapshots. Per-currency results use the relevant account base currency or native cost currency. | Method | Returns | | ---------------------------------------------- | ------------------------------------------------------ | | `mark_values(venue, account_id)` | Signed MTM totals for open positions. | | `equity(venue, account_id)` | Total equity combining balance and position valuation. | | `build_snapshot(account_id)` | Account-wide MTM totals and valuation metadata. | | `snapshots(account_id)` | Recorded account snapshots in emission order. | | `missing_price_instruments(venue, account_id)` | Instruments currently flagged as unpriceable. | Longs contribute positive notional, shorts contribute negative notional. Flat positions are skipped. An account-scoped `equity()` query returns `{}` for an unknown account. For a known account, it raises `RuntimeError` if exact snapshot valuation fails instead of presenting the failure as empty equity. ### Equity formula Equity combines the account balance with open-position valuation, using a different second term depending on account type: - **Cash accounts without a base currency and Wallet accounts**: Start with `balances_total`. For positions owned by that account, do not add a base-asset mark value when the balance already holds that asset and the instrument's cost currency differs from its base currency. Add mark values for inverse instruments and positions not represented by a credited balance asset. - **Cash accounts with a base currency and betting accounts**: `balances_total + Σ mark_value(open positions)`. - **Margin accounts**: `balances_total + Σ unrealized_pnl(open positions)`. `mark_values()` always returns gross open-position values, including assets already present in a multi-currency Cash or Wallet balance. The value-once rule means `equity()` and equity snapshots count each non-inverse base asset either as a balance or a mark value, not both. The margin path uses the same cached unrealized PnL pipeline that powers `unrealized_pnls()`. ### Price fallback Valuation asks `Cache` for a price in this order, stopping at the first match: 1. Mark price, if `use_mark_prices=true` (the default) in `PortfolioConfig` and a mark price is cached. 2. Side-appropriate quote: `BID` for longs, `ASK` for shorts. 3. Last trade price. 4. Most recent cached bar close (populated when `bar_updates=true`). Set `use_mark_prices=false` to skip the mark tier and begin with the side-appropriate quote. If none of the four yield a current price, the Portfolio carries the last valid price for that instrument and position side. The next snapshot lists the instrument in `stale_instruments`. If the position has never had a valid price, it goes into the missing-price tracker, is listed in `unpriced_instruments`, and is excluded from the sum. ### Base currency conversion When `convert_to_account_base_currency=true` (the default) and the account has a `base_currency` set, cost-currency values are converted to the base currency using `MID` xrates from `Cache.get_xrate()`. With `use_mark_xrates=true`, the cached mark xrate from `Cache.get_mark_xrate()` is used first and falls back to `MID` if unavailable. The output dictionary then has a single key matching the base currency. When `convert_to_account_base_currency=false`, or the account has no `base_currency`, results are keyed by each position's native cost currency and no xrate conversion is applied. If no current xrate is available for a required conversion, the Portfolio carries the last valid rate and lists its source currency in `stale_currencies`. If no valid rate has ever been available, the position is treated as unpriceable and flagged through the missing-price tracker rather than silently valued at a 1.0 rate. ### Snapshot valuation metadata `PortfolioSnapshot.total_equity` always provides the per-currency MTM breakdown. When base-currency conversion is enabled and the account has a base currency, `base_currency_equity` provides the headline scalar in that currency. It is `None` when conversion is disabled or the account has no base currency. `is_stale` is true when the snapshot uses a carried price or xrate, or excludes a position that has never had all required valuation inputs. The related fields identify the cause: - `stale_instruments`: Instruments valued with carried prices. - `stale_currencies`: Source currencies converted with carried xrates. - `unpriced_instruments`: Instruments excluded because no complete valid valuation has ever been available. Call `build_snapshot(account_id)` for an on-demand sample. Call `snapshots(account_id)` to read the bounded recorded sequence. The methods are available from the Rust Portfolio and Strategy API and from the Python Portfolio binding. Building a snapshot does not add it to the recorded sequence; only the configured lifecycle emission records snapshots. ### Automatic equity curve `PortfolioConfig.equity_curve=true` (the default) records and publishes a mark-to-market snapshot when each account registers, at every UTC midnight even while the account is flat, and when the backtest or live node shuts down. Set `equity_curve=false` for workloads such as optimizer runs that do not consume an equity curve. On-demand `equity()` and `build_snapshot()` calculations remain available. The separate `snapshot_interval_ms` setting remains opt-in. When set, it adds fine-grained snapshots only while the account has an open position. ### Missing-price tracking The tracker keeps the latest missing set for each account-filtered query scope and the unfiltered venue scope. `missing_price_instruments(venue)` returns their venue-wide union. Pass `account_id` to return only that account's current set. Each observation remains authoritative until the same scope runs again; a filtered result does not declare an earlier unfiltered result resolved. It has two observable behaviors: - A warning log fires once per instrument on the transition from no scope reporting it to at least one scope reporting it, not on every subsequent call. Once every reporting scope observes recovery, a future drop re-warns. - When a venue goes flat (no open positions), its tracker entry is cleared so stale instruments do not remain flagged. Call `missing_price_instruments(venue)` to inspect the current set. :::tip If `equity()` understates what you expect, check `missing_price_instruments(venue)` before investigating the math. An instrument without a usable mark, quote, trade, or bar price is excluded from the total and appears in the missing-price tracker. ::: ### Venue and account scope Python collection queries accept optional `venue` and `account_id` scopes. If both are provided, they must resolve to the same account or the query raises `ValueError`. With `account_id=None`, a venue query aggregates across every account on that venue. An account-filtered valuation reconciles only that account's observation, so flags raised by other accounts on the same venue survive. ### Python query boundary The Python Portfolio is a read-only query facade for portfolio state. It does not expose initialization, reset, or update commands; the Rust engine remains responsible for authoritative mutation. Statistic registration is the exception, because it configures analysis rather than portfolio state. The facade also does not expose the internal recorded realized-PnL cache. `account()` returns a detached, point-in-time copy. The copy **does not reflect later account updates**, and changing it does not affect the Portfolio. Call `account()` again to obtain the latest account state. ## Portfolio statistics `Portfolio.statistics()` computes a new `PortfolioStatistics` value from all accounts, cached positions, position snapshots, recorded close-time PnLs, and portfolio snapshots. It recomputes the statistics on every call, so invoke it sparingly on hot paths. The result contains: - PnL statistics for each currency. - Return statistics from the preferred return series described below. - General statistics derived from positions. The default set includes `WinRate`, `ProfitFactor`, `SharpeRatio`, and `LongRatio`. See the [Analysis API Reference](/docs/python-api-latest/analysis.html) for all built-in statistic types. Pass another built-in type, such as `MaxDrawdown`, to `Portfolio.register_statistic()` to add it to `Portfolio.statistics()`, backtest results, and post-run logs. A standalone `PortfolioAnalyzer` keeps its own registrations, which do not reach the Portfolio. After a backtest, `engine.get_result()` exposes these categories through `stats_pnls`, `stats_returns`, and `stats_general`, plus the selected `returns_series`. When `run_analysis=true`, the engine also logs the three statistic categories under `PORTFOLIO PERFORMANCE` after the run. ### Custom statistics Subclass `PortfolioStatistic` and override the calculation methods for the input categories the statistic supports. Each category receives the same data the built-in statistics use, and a statistic contributes a value only where it overrides the matching method. `Portfolio.statistics()` feeds the first three categories; benchmark-relative results come from `PortfolioAnalyzer.get_performance_stats_returns_vs_benchmark()`, which takes the benchmark series from the caller. | Method | Input | Result category | | --------------------------------------- | -------------------------------------------- | ------------------------ | | `calculate_from_returns` | Returns keyed by UNIX nanoseconds | `returns` | | `calculate_from_realized_pnls` | Realized PnLs for one currency, oldest first | `pnls` | | `calculate_from_positions` | Positions and position snapshots | `general` | | `calculate_from_returns_with_benchmark` | Strategy returns and benchmark returns | `PortfolioAnalyzer` only | ```python from nautilus_trader.analysis import PortfolioStatistic class TradeCount(PortfolioStatistic): def calculate_from_realized_pnls(self, realized_pnls: list[float]) -> float | None: return float(len(realized_pnls)) engine.portfolio.register_statistic(TradeCount()) ``` The name defaults to the class name split on word boundaries, so `TradeCount` registers as "Trade Count". Override the `name` property to choose the name directly. Registering a statistic whose name matches an existing one replaces it, and `Portfolio.deregister_statistic()` removes one by name. A registration persists across `Portfolio.statistics()` calls and analyzer state resets, so it reaches backtest results and post-run logs for the whole run. Register before the run to cover every query. Return a `float`, or `None` when the metric is undefined for the given data. Define the result for empty or insufficient data: return `None` when the metric is unknown, or use a domain-appropriate default such as `0.0`. A statistic that raises, or that returns a non-numeric value, logs an error and contributes no value for that category; the remaining statistics still calculate. Every category is called for a registered statistic, including on a run that closed no trades, where the PnL category receives an empty list. :::warning A calculation method runs while the Portfolio holds its internal state borrowed, so calling a Portfolio method that mutates state from inside one panics. Keep a statistic a pure function of the data it is given. ::: ## Returns: position vs portfolio The analyzer tracks two distinct return series: - **Position returns** (`analyzer.position_returns()`) measure realized return per position as a side-aware price return relative to the average open price. This reflects the instrument's price movement between entry and exit, independent of account size or leverage. - **Portfolio returns** (`analyzer.portfolio_returns()`) measure daily percentage change in mark-to-market account equity. A $900 gain on a $100,000 account reports roughly 0.9% for that day. When complete portfolio snapshots span at least two distinct UTC dates, the analyzer uses the final snapshot from each date and computes portfolio returns automatically. It uses them as the primary series for statistics, tearsheets, and the monthly returns heatmap. A snapshot emitted exactly at UTC midnight closes the preceding date, keeping the daily tier consistent with fine-grained samples. The first valid registration sample anchors the opening value for a partial first date. Missing or unpriced account dates are forward-filled after every account has an initial valid sample. Multiple snapshots on the same date count as one date, so intra-day trading alone does not produce portfolio returns. When portfolio returns are unavailable, the analyzer falls back to position returns; Python tearsheets can also fall back to account reports. The convenience accessor `analyzer.returns()` resolves this preference: portfolio returns when present, position returns otherwise. ### Multi-currency accounts Portfolio returns require every account's snapshot equity to resolve to one common currency. Base-currency conversion normally provides that scalar. When snapshots expose multiple currencies or accounts resolve to different currencies, the analyzer falls back to position returns. An explicit tearsheet currency can select matching per-currency equity where available. If you need portfolio-level returns for a multi-currency account, compute them externally by converting balances to a common currency before calculating percentage changes. ### Multi-account calculation Backtest analysis aggregates all cached accounts after resolving them to a common currency. The tearsheet follows the same account-wide aggregation rule for multi-venue backtests. ## Related guides - [Positions](positions.md): Position tracking within portfolios. - [Reports](reports.md): Generate portfolio analysis reports. - [Visualization](visualization.md): Visualize portfolio performance. # Positions Source: https://nautilustrader.io/docs/latest/concepts/positions/ This guide explains how NautilusTrader creates and updates positions from order fills, calculates profit and loss (PnL), and preserves closed cycles under `NETTING` order management system (OMS) configurations. ## Overview A position records exposure to an instrument during an open-close cycle. It aggregates the fills assigned to one position ID and tracks its quantity, average prices, realized PnL, commissions, and related identifiers. Use a market price with the position's methods to calculate unrealized PnL and notional value. The execution engine creates positions when orders fill and tracks them from open to close. OMS configuration determines whether fills share a net position or remain in separate hedged positions. ## Position lifecycle ### Creation The system opens a position on the first fill: - **NETTING OMS**: Opens on the first fill for an instrument and strategy. The position uses the deterministic ID `{instrument_id}-{strategy_id}`. - **HEDGING OMS**: Opens on first fill for a new `position_id` (multiple positions per instrument). A position tracks: - Opening order and fill details. - Entry side (`BUY` or `SELL`). - Quantity and average entry price after applying the opening fill. - Timestamps for initialization and opening. :::tip You can access positions through the Cache using `self.cache.position(position_id)` or `self.cache.positions(instrument_id=instrument_id)` from within your actors/strategies. ::: ### Updates As additional fills occur, the position: - Aggregates quantities from buy and sell fills. - Recalculates average entry and exit prices. - Updates peak quantity for the current cycle. - Tracks the current cycle's order IDs and trade IDs. - Accumulates commissions by currency. ### Closure A position closes when the **net quantity becomes zero** (`FLAT`). At closure: - The closing order ID is recorded. - Duration is calculated from open to close. - Final realized PnL is computed. - In either OMS type, when the position later reopens under the same ID, the engine snapshots the closed state to preserve historical PnL (see [Position snapshotting](#position-snapshotting)). ## Order fill aggregation Positions aggregate order fills to maintain an accurate view of market exposure. The aggregation process handles both sides of trading activity: ### Buy fills When a BUY order fills: - Increases long exposure or reduces short exposure. - Updates average entry price for opening trades. - Updates average exit price for closing trades. - Calculates realized PnL for any closed portion. ### Sell fills When a SELL order fills: - Increases short exposure or reduces long exposure. - Updates average entry price for opening trades. - Updates average exit price for closing trades. - Calculates realized PnL for any closed portion. ### Net position calculation The position maintains a `signed_qty` field representing the net exposure: - Positive values indicate `LONG` positions. - Negative values indicate `SHORT` positions. - Zero indicates a `FLAT` (closed) position. ```python # Example: Position aggregation # Initial BUY 100 units at $50 signed_qty = +100 # LONG position # Subsequent SELL 150 units at $55 signed_qty = -50 # Closes the LONG cycle and opens a SHORT cycle # Final BUY 50 units at $52 signed_qty = 0 # Position FLAT (closed) ``` ### Reversal accounting An opposite-side fill larger than the open quantity closes the existing exposure and opens the residual in the other direction. The execution engine splits this fill into a close and a new opening, with closed-state retention governed by [Position snapshotting](#position-snapshotting). When an unsplit reversal fill is applied directly to one `Position`, including during fill-void replay, the position starts a new accounting episode for the residual exposure: - `avg_px_open` becomes the reversal fill price. - `avg_px_close` becomes `None`, and `realized_return` becomes zero until a subsequent closing fill. - `buy_qty` and `sell_qty` restart with only the opening residual on the new entry side and zero on the other side. Later close averages therefore exclude volume from the previous direction. **Only the closing portion** realizes PnL against the previous entry price. The object's `realized_pnl` and commission totals remain cumulative across this reversal, with the fill's commission counted once. The reset does not clear fill history, opening timestamps, or peak quantity. If the position instead reaches `FLAT` and a later fill reopens it, the full cycle resets, including realized PnL and commissions. These episode resets apply to fill-driven reversals; a quantity adjustment that changes the position's side does not perform the same reset. ## Position adjustments Position adjustments record quantity or PnL changes that occur outside normal order fills. The system represents these changes as `PositionAdjusted` events. ### Base-currency commissions When trading spot currency pairs (for example, BTC/USDT) or FX spot, commissions paid in the base currency directly affect the net quantity received or delivered: - **Opening fills**: Commission is deducted from the traded quantity. A buy of 1.0 BTC with 0.001 BTC commission results in a net long position of 0.999 BTC. - **Closing fills**: Commission is applied to `signed_qty` because it affects actual inventory. Selling a 0.999 BTC LONG position with 0.000999 BTC commission leaves you SHORT 0.000999 BTC, not FLAT, because you gave up 0.999999 BTC total. - **Flips**: Commission affects the final position size on both sides of the flip. :::note Base-currency commissions only apply to spot currency pairs and FX spot instruments where the commission currency matches `instrument.base_currency`. For other instruments, commissions are tracked separately and do not affect position quantity. ::: ### Funding payments Funding adjustments track periodic payments for perpetual futures without affecting position quantity. They use `quantity_change = None` and can include a PnL change. ### Adjustment tracking The position exposes its retained adjustments: - `position.adjustments()` returns the list of all `PositionAdjusted` events. - Each adjustment includes its type (`COMMISSION` or `FUNDING`), quantity or PnL change, reason, event ID, and timestamps. - The current adjustment history is cleared when a closed position reopens. - If fills remain after `purge_events_for_order()`, the position regenerates commission adjustments from the surviving fills and reapplies non-commission adjustments. If no fills remain, the position becomes an empty `FLAT` shell and clears its adjustment history. ## OMS types and position management NautilusTrader supports two position management modes. A strategy configured with `OmsType.UNSPECIFIED` uses the venue's OMS type. For configuration details and position ID rules, see the [Execution guide](execution/index.md#order-management-system-oms). ### `NETTING` In `NETTING` mode, fills for each instrument and strategy are aggregated into a single position: - One position per instrument and strategy. - All fills contribute to the same position. - A fill that crosses zero closes the current cycle and opens a new cycle on the opposite side. - Historical snapshots preserve closed position states. ### `HEDGING` In `HEDGING` mode, multiple positions can exist for the same instrument: - Multiple simultaneous `LONG` and `SHORT` positions. - Each position has a unique position ID. - Positions are tracked independently. - No automatic netting across positions. - A fill with a new position ID creates a separate position. If a later fill reuses a closed position ID, the engine archives the closed cycle before replacing the cached state. - A virtual position flip creates a new ID and keeps the original closed position in the cache, so that path does not need a closed-cycle snapshot. :::warning `HEDGING` can increase margin requirements when a venue maintains long and short positions independently. A venue with a `NETTING` OMS exposes only its net position, even when NautilusTrader tracks multiple virtual positions. Check the venue's position mode and margin rules. ::: ### Strategy vs venue OMS Strategy and venue OMS types can differ: | Strategy OMS | Venue OMS | Result | | ------------ | --------- | ------------------------------------------------------------------- | | `NETTING` | `NETTING` | One position per instrument and strategy. | | `HEDGING` | `HEDGING` | Multiple positions per instrument and strategy. | | `NETTING` | `HEDGING` | One virtual position across the venue positions. | | `HEDGING` | `NETTING` | Multiple virtual positions against the venue's single net position. | :::tip Align the strategy and venue OMS types unless the strategy requires virtual positions. See the integration guide for the venue's position-mode configuration. ::: ## Position snapshotting Position snapshotting preserves closed cycles for PnL tracking and reporting when a later fill reopens a closed position. ### Why snapshotting matters When a position closes (becomes `FLAT`) and then reopens under the same ID with a new trade, the position object is reset to track the new exposure. Without snapshotting, the historical realized PnL from the previous position cycle would be lost. ### How it works When a fill reopens a closed position under the same ID, the execution engine archives the closed state before opening the next cycle. This applies to both `NETTING` and `HEDGING` OMS. A `HEDGING` flip using a non-virtual ID follows a separate path: it reuses the ID without archiving the closed cycle. The snapshot preserves: - Final quantities and prices. - Realized PnL. - All fill events. - Commission totals. The cache stores snapshots by position ID. The active cache entry then represents the new cycle, while previous snapshots remain accessible. The Portfolio includes their realized PnL in instrument totals. A fill void that corrects a fill from an earlier cycle is the one exception. The correction moves the cycle boundaries the stored snapshots describe, so the engine replaces them with the cycles the corrected history actually closes, keeping each counted once. See [Position replay across NETTING cycles](execution/index.md#position-replay-across-netting-cycles). :::note This closed-cycle archive differs from optional position state snapshots. Setting `snapshot_positions=True` publishes state when a position opens, changes, or closes, while `snapshot_positions_interval_secs` periodically publishes all open positions. A cache with a Redis or Postgres backing also persists these snapshots. Without cache backing, both paths publish snapshots on the in-process message bus without persisting them. See [`LiveExecutionEngineConfig`](/docs/python-api-latest/live.html#nautilus_trader.live.LiveExecutionEngineConfig) for these settings. ::: ### Example scenario ```text # NETTING OMS Example # Cycle 1: Open LONG position BUY 100 units at $50 # Position opens SELL 100 units at $55 # Position closes, PnL = $500 # Snapshot taken preserving $500 realized PnL # Cycle 2: Open SHORT position SELL 50 units at $54 # Position reopens (SHORT) BUY 50 units at $52 # Position closes, PnL = $100 # Snapshot taken preserving $100 realized PnL # Total realized PnL = $500 + $100 = $600 (from snapshots) ``` ## PnL calculations Position PnL calculations account for instrument specifications and market conventions. ### Realized PnL The price component of realized PnL is calculated when fills partially or fully close a position. Commissions in the position's cost currency affect realized PnL as each fill arrives. ```python # For standard instruments # LONG: realized_pnl = (exit_price - entry_price) * closed_quantity * multiplier # SHORT: realized_pnl = (entry_price - exit_price) * closed_quantity * multiplier # For inverse instruments (side-aware) # LONG: realized_pnl = closed_quantity * multiplier * (1/entry_price - 1/exit_price) # SHORT: realized_pnl = closed_quantity * multiplier * (1/exit_price - 1/entry_price) ``` The position side selects the formula. ### Unrealized PnL `unrealized_pnl()` calculates PnL for an open position from the supplied `price`. You can pass a bid, ask, mid, last, or mark price: ```python position.unrealized_pnl(last_price) # Using last traded price position.unrealized_pnl(bid_price) # Conservative for LONG positions position.unrealized_pnl(ask_price) # Conservative for SHORT positions ``` For a `FLAT` position, it returns `Money(0, cost_currency)` regardless of the supplied price. ### Total PnL `total_pnl()` combines the realized and unrealized components: ```python total_pnl = position.total_pnl(current_price) # Returns realized_pnl + unrealized_pnl ``` ### Currency considerations - PnL is calculated in the instrument's cost currency: quote for linear contracts, base for inverse contracts, and settlement for quanto contracts. - For Forex, the cost currency is typically the quote currency. - Portfolio aggregates realized PnL per instrument in cost currency. - Multi-currency totals require conversion outside the Position class. ## Commissions and costs Positions track fill commissions: - Commissions are accumulated by currency. - Each fill's commission is added to the running total. - Multiple commission currencies are supported. - Realized PnL includes commissions only when denominated in the position's cost currency. - Other commissions are tracked separately and may require conversion. ```python commissions = position.commissions() # Returns list[Money] with aggregated commission totals per currency notional = position.notional_value(current_price) # Returns Money in quote (linear), base (inverse), or settlement currency (quanto) ``` :::warning In Python, `notional_value()` raises `ValueError` if an inverse position lacks a base currency, the supplied inverse price is not positive, or the result cannot be represented as `Money`. Rust callers can use `try_notional_value()` to handle these calculation errors; `notional_value()` panics if the calculation fails. ::: ## Position properties and state ### Identifiers - `id`: Unique position identifier. - `instrument_id`: The traded instrument. - `account_id`: Account where position is held. - `trader_id`: The trader who owns the position. - `strategy_id`: The strategy managing the position. - `opening_order_id`: Client order ID that opened the position. - `closing_order_id`: Client order ID that closed the position, if closed. ### Position state - `side`: Current position side (`LONG`, `SHORT`, or `FLAT`). - `entry`: Opening side for the current cycle (`BUY` for `LONG`, `SELL` for `SHORT`). Updates when the position reverses direction. - `quantity`: Current absolute position size. - `signed_qty`: Signed position size (positive for `LONG`, negative for `SHORT`). - `peak_qty`: Maximum quantity reached during the current open-close cycle. - `is_open`: Whether position is currently open. - `is_closed`: Whether position is closed (`FLAT`). - `is_long`: Whether position side is `LONG`. - `is_short`: Whether position side is `SHORT`. ### Pricing and valuation - `avg_px_open`: Average entry price. - `avg_px_close`: Average exit price when closing. - `realized_pnl`: Realized profit/loss. - `realized_return`: Realized return as decimal (e.g., 0.05 for 5%). - `quote_currency`: Quote currency of the instrument. - `base_currency`: Base currency if applicable. - `settlement_currency`: Currency for PnL settlement. See [Reversal accounting](#reversal-accounting) for how price averages and returns reset while realized PnL remains cumulative when an unsplit fill reverses one position. ### Instrument specifications - `multiplier`: Contract multiplier. - `price_precision`: Decimal precision for prices. - `size_precision`: Decimal precision for quantities. - `is_inverse`: Whether instrument is inverse. ### Timestamps - `ts_init`: When position was initialized. - `ts_opened`: When position was opened. - `ts_last`: Last update timestamp. - `ts_closed`: When the position was closed, if closed. - `duration_ns`: Duration from open to close in nanoseconds, or zero while open. ### Associated data - `symbol`: The instrument's ticker symbol. - `venue`: The trading venue. - `client_order_ids`: Unique client order IDs for retained fills in the current cycle. - `venue_order_ids`: Unique venue order IDs for retained fills in the current cycle. - `trade_ids`: Unique trade IDs for retained fills in the current cycle. - `events`: Retained order fill events in the current cycle. - `adjustments`: Retained position adjustments in the current cycle. - `event_count`: Number of retained fill events in the current cycle. - `last_event`: Most recently retained fill event. - `last_trade_id`: Trade ID of the most recently retained fill. :::info For complete type information and detailed property documentation, see the Position [API Reference](/docs/python-api-latest/model/position.html#nautilus_trader.model.Position). ::: ## Events and tracking Each `Position` object records the fills and adjustments for its current open-close cycle: - Fill events remain in application order. - Client order, venue order, and trade ID accessors return sorted, unique values. - `event_count` reports the number of retained fill events. - Closed `NETTING` cycles retain their event history in the cache snapshots described above. This data supports: - Detailed position analysis. - Trade reconciliation. - Performance attribution. - Audit trails. :::tip Use `position.events()` to access the current cycle's retained fills for reconciliation. The `position.trade_ids()` result helps match against broker statements. See the [Execution guide](execution/) for reconciliation best practices. ::: ## Numerical precision `Position` uses `f64` for signed quantity, average prices, realized returns, and PnL intermediates. `Price`, `Quantity`, and `Money` retain their fixed-point representations at the API boundary, but conversions between these types and `f64` can introduce rounding. `f64` represents every integer exactly only through `2^53`; above that boundary, conversion can lose low-order bits. It provides roughly 15 to 16 significant decimal digits rather than a fixed number of exact decimal places. The design avoids the higher computational cost of arbitrary-precision arithmetic. The average-price calculation also avoids multiplying raw fixed-point values because those products can overflow their integer representation. Average prices reuse the prior `f64` average, and realized PnL is converted between `Money` and `f64` as fills accumulate. The resulting precision depends on the values, settlement-currency precision, and sequence of fills. `quantity` is derived from `signed_qty` at the instrument's `size_precision`. If that conversion rounds a residual quantity to zero, the position becomes `FLAT` and normalizes `signed_qty` to zero. Inverse PnL calculations reject nonpositive open or close prices and positive prices below `1e-15`. With the `defi` feature, converting a `Price` or `Quantity` with more than 16 decimal places to `f64` panics, so `Position` does not support 17- or 18-decimal fill values. Tests in `crates/model/src/position.rs` cover a `0.01` USD commission, nine-decimal price inputs, 100 sequential fills, prices from `0.00001` to `99999.99999`, and same-price round trips. These cases do not establish a universal precision bound. :::warning If a workflow requires exact decimal arithmetic for regulatory reporting or audit records, perform and retain a separate decimal calculation from the original fills and adjustments. Converting `Position` float outputs back to decimal, including through `signed_decimal_qty()`, cannot restore discarded precision. `Position` does not provide an exact-decimal guarantee. Validate the instruments, currencies, amount ranges, and fill sequences used by the application. ::: ## Integration with other components Positions interact with several key components: - **Portfolio**: Aggregates position exposure and PnL across instruments and strategies. - **ExecutionEngine**: Creates and updates positions from fills. - **Cache**: Stores current position state and closed-cycle snapshots. - **RiskEngine**: Reads open positions when it checks whether an order reduces exposure. :::info Positions are not created for spread instruments. Contingent orders can still trigger for spreads, but they operate without position linkage. The engine handles spread instruments separately from regular positions. ::: ## Related guides - [Events](events/): How fills produce position events. - [Orders](orders/): Orders that create and modify positions. - [Execution](execution/): Fill handling that updates positions. - [Portfolio](portfolio.md): Portfolio-level position aggregation. # Python Source: https://nautilustrader.io/docs/latest/concepts/python/ NautilusTrader provides a Python control surface over the Rust core through PyO3. Use this guide to understand which runtime owns each part of the system, where Python code executes, and which Python interfaces form the supported public contract. For native Rust applications, see [Rust](rust.md). For installation and supported Python versions, see [Installation](../getting_started/installation.md). ## Runtime model The Python package combines Python facades under `nautilus_trader` with the compiled `nautilus_trader._libnautilus` extension. Prebuilt wheels contain the extension and do not require a Rust toolchain at runtime. | Layer | Responsibility | | ------------------ | ---------------------------------------------------------------------------------------------- | | Python application | Configuration, composition, user components, analysis, and integration with Python services. | | PyO3 bindings | Type conversion, argument validation, exceptions, and ownership-safe wrappers over Rust state. | | Rust core | Domain types, engines, nodes, cache, portfolio, message bus, adapters, and persistence. | Python objects such as `Cache` and `Portfolio` are wrappers over Rust-owned state. Nodes and engines keep their internal runtime objects private and expose bounded inspection and control methods. This preserves one source of state while allowing Python code to configure the system and inspect its results. ## User components Python user components subclass the public PyO3 base classes and override their documented callbacks: | Component | Use | | -------------------- | ----------------------------------------------------------------------------- | | `DataActor` | Subscribe to data, handle events, and run non-trading workflows. | | `Strategy` | Implement trading decisions and submit orders. | | `ExecutionAlgorithm` | Split or schedule routed orders through the execution engine. | | `Controller` | Create and manage actors and strategies through `ImportableControllerConfig`. | Application code constructs configs, registers official adapter factories, and adds components to `BacktestNode`, `BacktestEngine`, or `LiveNode`. Rust remains responsible for routing, engine state, order management, accounting, and venue clients. Callbacks execute **synchronously** on the event-processing thread and must return promptly. Blocking I/O, model inference, or long calculations delay market-data handling and order execution. Offload that work to an executor or another process. See [Configure a live trading node](../how_to/configure_live_trading.md) for the live-trading rule. ## Async execution Rust adapter networking runs on Tokio. Python async libraries run on an asyncio event loop; PyO3 does not turn Python coroutines into Tokio tasks. `LiveNode` supports two execution modes: | Method | Execution context | Signal owner | Completion | | ------------- | ---------------------- | ---------------- | ----------------------------------------------------------- | | `run()` | Calling thread; blocks | `LiveNode` | Returns after coordinated shutdown finishes. | | `run_async()` | Python host loop | Host application | Resolves after the same coordinated shutdown path finishes. | `run_async()` lets an asyncio or ASGI application host the node on its existing loop. It drives the same Rust lifecycle as `run()` and leaves `SIGINT` and `SIGTERM` handling to the host. Compatibility is tested with the default asyncio loop, uvloop, and an ASGI lifespan managed by Uvicorn. The Python wheel does not install uvloop or Uvicorn; applications supply their chosen loop and server. An ASGI application whose lifespan constructs a node must run with one worker and without hot reload. Capture `node.cache`, `node.portfolio`, and `node.handle()` before starting `run_async()`. The coroutine owns the node until it finishes, while the captured objects remain usable. Stop a hosted run through `LiveNodeHandle.stop()`, then await the run task for complete shutdown. Cancellation requests the same graceful shutdown before it propagates. Call `node.dispose()` after the task finishes to release the node's resources. A host must wait for the handle to report `Running` before reporting startup complete, supervise the run task for its lifetime, and fail the service if the task completes unexpectedly. See [Hosted event loops](live.md#hosted-event-loops) for the lifecycle, cancellation, fairness, and cache-backing contract. ## Public API contract The generated type stubs under `python/nautilus_trader/` define the supported Rust-bound Python surface. They record public classes, methods, properties, parameters, and return types from the Rust binding sources. The [Python API reference](../api_reference/index.md) renders the same public modules and their documentation. A runtime attribute on a Rust-bound class absent from the generated stubs is not part of the supported contract. The documented Python client, provider, and importable-config classes also form a public interface. PyO3 validates bound arguments before Rust code runs and maps fallible operations to Python exceptions. Code should handle the documented exception type instead of depending on an internal Rust error representation. Generated stubs are source-derived artifacts. Binding changes update the Rust source and regenerate the stubs; the checked-in `.pyi` files are not independent API definitions. :::warning[Side enum compatibility aliases] `OrderSide.NO_ORDER_SIDE` and `PositionSide.NO_POSITION_SIDE` remain available as compatibility aliases for `None`. They are not enum members and may be removed in a future version. Use `None` for optional side values. ::: ## Ownership and lifecycle Rust ownership remains visible at node boundaries: - `BacktestNode` keeps its engines internal. Preserve an engine after a run with `dispose_on_completion=False`, then inspect it through the node's cache, portfolio, statistics, and report methods. - `LiveNode.run_async()` lends the node to its coroutine. State access through the node raises during the run, while `is_running` and `handle()` remain available. A `dispose()` call during the run is a no-op, not a deferred request. Call it again after the run finishes; objects captured before the run remain available until then. - Concurrent `LiveNode` or `BacktestNode` instances in one process are not supported because their runtime state is not isolated. Dispose one node before starting the next, or use separate processes for parallel execution. These boundaries prevent Python references from exposing mutable engine internals or creating multiple owners for the same runtime state. ## Support boundaries Official adapters are implemented in Rust and exposed through Python configs, factories, clients, and data types under `nautilus_trader.adapters`. Their integration guides define the supported venue capabilities. Custom live adapters subclass the Python client bases and register through `LiveNodeBuilder` or `LiveNode.build`. Their async work runs on the node's bound Python event loop. They receive a read-only cache view and emit typed data, events, and reports through queued output. Independent Rust/PyO3 packages can use the same Python protocol with model objects from the installed wheel. See the [Python adapter interface](../developer_guide/python_adapters.md) for the supported hooks, factory/config contract, lifecycle rules, and v1 migration limits. ## Choosing Python or Rust Use Python when application composition, rapid strategy development, analysis tools, or integration with the Python ecosystem matters. Use Rust when the application must run without a Python runtime or needs native traits and direct crate-level control. Both paths use the same Rust domain model and engines. The [Rust capability matrix](rust.md#capability-matrix) shows which components and official adapters are exposed through each path. ## Related guides - [Architecture](architecture.md) - Core components, threading, and dependency flow. - [Rust](rust.md) - Native Rust APIs and runtime use. - [Live trading](live.md) - LiveNode lifecycle and hosted event loops. - [Backtesting](backtesting/) - Backtest engines, nodes, data, and venues. - [Adapters](adapters.md) - Official adapter configuration and routing. - [Migration from v1](../../MIGRATION_V2.md) - Python API changes and migration boundaries. # Reports Source: https://nautilustrader.io/docs/latest/concepts/reports/ This guide explains the portfolio analysis and reporting capabilities provided by the `ReportProvider` class, and how these reports are used for PnL accounting and backtest post-run analysis. ## Overview `ReportProvider` turns cached orders, fills, positions, and account states into pandas DataFrames for analysis and visualization. These reports help you evaluate strategy performance, analyze execution quality, and verify PnL accounting. The same reports are available in backtesting and live trading, which keeps performance evaluation and strategy comparison consistent across both. Reports can be generated using two approaches: - **Backtest methods**: `BacktestEngine.generate_orders_report()` and its siblings read the engine's own cache. `BacktestNode` exposes the same methods, taking the run config ID as the first argument. - **`ReportProvider` directly**: pass any collection of orders or positions, such as a live node's cache or a filtered cache query. Every method returns an empty DataFrame when no matching data exists. Report generation requires pandas, which `nautilus_trader.analysis` imports lazily: the module imports without pandas installed, and the `ImportError` surfaces when you generate a report. The `visualization` extra installs it. ## Available reports The `ReportProvider` class offers several static methods to generate reports from trading data. Each report returns a pandas DataFrame with specific columns and indexing for easy analysis. ### Orders report Generates a full view of all orders: ```python from nautilus_trader.analysis import ReportProvider # From a completed backtest run orders_report = engine.generate_orders_report() # Or from any cache, such as a live node's orders_report = ReportProvider.generate_orders_report(cache.orders()) ``` **Returns `pd.DataFrame`. Columns include:** | Column | Description | | ----------------- | -------------------------------------------------- | | `client_order_id` | Index - unique order identifier. | | `instrument_id` | Trading instrument. | | `strategy_id` | Strategy that created the order. | | `trader_id` | Trader identifier. | | `account_id` | Account identifier (if assigned). | | `venue_order_id` | Venue-assigned order ID (if accepted). | | `side` | BUY or SELL. | | `type` | MARKET, LIMIT, etc. | | `status` | Current order status. | | `quantity` | Original order quantity (string). | | `filled_qty` | Amount filled (string). | | `price` | Limit price (string, order-type dependent). | | `avg_px` | Average fill price (string, if filled). | | `time_in_force` | Time-in-force instruction. | | `ts_init` | Order initialization timestamp (Unix nanoseconds). | | `ts_last` | Last update timestamp (Unix nanoseconds). | Additional columns vary by order type, such as `trigger_price` for stop orders and `expire_time_ns` for GTD orders. See `Order.to_dict()` for the complete field list. ### Order fills report Provides a summary of filled orders (**one row per order**): ```python # From a completed backtest run fills_report = engine.generate_order_fills_report() # Or from any cache fills_report = ReportProvider.generate_order_fills_report(cache.orders()) ``` This report includes only orders with `filled_qty > 0` and contains the same columns as the orders report, but filtered to executed orders only. Note that `ts_init` and `ts_last` are converted to datetime objects in this report for easier analysis. ### Fills report Details individual fill events (**one row per fill**): ```python # From a completed backtest run fills_report = engine.generate_fills_report() # Or from any cache fills_report = ReportProvider.generate_fills_report(cache.orders()) ``` **Returns `pd.DataFrame`. Columns include:** | Column | Description | | ----------------- | ---------------------------------------- | | `client_order_id` | Index - order identifier. | | `trade_id` | Unique trade/fill identifier. | | `venue_order_id` | Venue-assigned order ID. | | `instrument_id` | Trading instrument. | | `strategy_id` | Strategy that created the order. | | `account_id` | Account identifier. | | `position_id` | Associated position ID (if applicable). | | `order_side` | BUY or SELL. | | `order_type` | Order type (MARKET, LIMIT, etc.). | | `last_px` | Fill execution price (string). | | `last_qty` | Fill execution quantity (string). | | `currency` | Currency of the fill. | | `liquidity_side` | MAKER or TAKER. | | `commission` | Commission amount and currency (string). | | `ts_event` | Fill timestamp (datetime). | | `ts_init` | Initialization timestamp (datetime). | See `OrderFilled.to_dict()` for the complete field list; the report drops its `type` column. ### Positions report Position analysis including snapshots: ```python # From a completed backtest run, which includes snapshots automatically positions_report = engine.generate_positions_report() # Or from any cache positions_report = ReportProvider.generate_positions_report( positions=cache.positions(), snapshots=cache.position_snapshots(), # Needed for NETTING OMS totals ) ``` **Returns `pd.DataFrame`. Columns include:** | Column | Description | | ------------------ | ----------------------------------------------------- | | `position_id` | Index - unique position identifier. | | `instrument_id` | Trading instrument. | | `strategy_id` | Strategy that managed the position. | | `trader_id` | Trader identifier. | | `account_id` | Account identifier. | | `opening_order_id` | Order ID that opened the position. | | `closing_order_id` | Order ID that closed the position. | | `entry` | Entry side (BUY or SELL). | | `side` | Position side (LONG, SHORT, or FLAT). | | `quantity` | Current position size (string). | | `peak_qty` | Maximum size reached (string). | | `avg_px_open` | Average entry price (float). | | `avg_px_close` | Average exit price (float, if closed). | | `commissions` | Commissions paid, one entry per currency (list). | | `realized_pnl` | Realized profit/loss in the cost currency (string). | | `realized_return` | Realized return as a ratio (float), so `0.05` is 5%. | | `ts_init` | Position initialization timestamp (Unix nanoseconds). | | `ts_opened` | Opening timestamp (datetime). | | `ts_last` | Last update timestamp (Unix nanoseconds). | | `ts_closed` | Closing timestamp (datetime or NA). | | `duration_ns` | Position duration in nanoseconds. | | `is_snapshot` | Whether this is a historical snapshot. | Snapshot rows are indexed by a generated ID derived from the original position ID, so use `is_snapshot` rather than the index to separate archived cycles from live positions. See `Position.to_dict()` for the complete field list; the report drops `signed_qty`, `base_currency`, `quote_currency`, and `settlement_currency`. ### Account report Tracks account balance and margin changes over time: ```python from nautilus_trader.model import Venue venue = Venue("BINANCE") # From a completed backtest run account_report = engine.generate_account_report(venue=venue) # Or from any cache account_report = ReportProvider.generate_account_report(cache.account_for_venue(venue)) ``` `BacktestEngine.generate_account_report()` requires `venue` or `account_id` and raises `ValueError` when both are omitted. `account_id` takes precedence when both are supplied, and an unknown account yields an empty DataFrame. **Returns `pd.DataFrame`. Columns include:** | Column | Description | | --------------- | ------------------------------------------ | | `ts_event` | Index - timestamp of account state change. | | `account_id` | Account identifier. | | `account_type` | Type of account (e.g., CASH, MARGIN). | | `base_currency` | Base currency for the account. | | `total` | Total balance amount (string). | | `free` | Available balance (string). | | `locked` | Balance locked in orders (string). | | `currency` | Currency of the balance. | | `reported` | Whether balance was reported by venue. | | `margins` | Margin information (list, if applicable). | | `info` | Additional venue-specific information. | Each row represents a balance entry; accounts with multiple currencies produce multiple rows per account state event. ## PnL accounting considerations Accurate PnL accounting requires careful consideration of several factors: ### Position-based PnL - **Realized PnL**: Calculated when positions are partially or fully closed. - **Unrealized PnL**: Marked-to-market using current prices. `Position.unrealized_pnl(last)` marks an open position at a given `Price`. - **Commission impact**: Only included when in the position's cost currency. See [Positions](positions.md) for how base-currency commissions on spot pairs adjust position size instead. :::warning Position snapshots preserve historical PnL when a closed position reopens under the same ID, in either `NETTING` or `HEDGING` OMS. **Include snapshots in reports** for accurate total PnL calculation. See [Position snapshotting](positions.md#position-snapshotting) for reopening and flip behavior. ::: ### Multi-currency accounting When dealing with multiple currencies: - Each position tracks PnL in its cost currency: quote for linear contracts, base for inverse contracts, and settlement for quanto contracts. - Portfolio aggregation requires currency conversion. `Portfolio.realized_pnls(target_currency=...)` does this with cached exchange rates; see [Supported conversions](portfolio.md#supported-conversions). - Commission currencies may differ from the position's cost currency. ```python from decimal import Decimal # Accessing PnL across positions for position in cache.positions_closed(): realized = position.realized_pnl # Money in the position's cost currency, or None if realized is None or realized.currency == base_currency: continue # Converting by hand: cache.get_xrate() returns a float, so wrap rates as Decimal rate = Decimal(str(my_fx_rates[(realized.currency, base_currency)])) converted = realized.as_decimal() * rate ``` ### Snapshot considerations For `NETTING` OMS, an accurate instrument total adds the realized PnL of every archived cycle to the live position. See [Position snapshotting](positions.md#position-snapshotting) for how the execution engine archives a closed cycle. ```python from decimal import Decimal from nautilus_trader.model import Money pnl_by_currency = {} for position in cache.positions(instrument_id=instrument_id): # Archived cycles are stored under the live position's ID snapshots = cache.position_snapshots(position_id=position.id) for pnl in (position.realized_pnl, *(s.realized_pnl for s in snapshots)): if pnl is None: continue running = pnl_by_currency.get(pnl.currency, Decimal(0)) pnl_by_currency[pnl.currency] = running + pnl.as_decimal() # Create Money objects for each currency total_pnls = [Money.from_decimal(amount, currency) for currency, amount in pnl_by_currency.items()] ``` ## Backtest post-run analysis After a backtest completes, analysis is available through result statistics and generated reports. ### Accessing backtest results ```python # After backtest run engine.run() # Access result statistics result = engine.get_result() # Generate reports from the backtest engine fills_report = engine.generate_fills_report() venue = engine.list_venues()[0] account_report = engine.generate_account_report(venue=venue) # Or access data directly for custom analysis orders = engine.cache.orders() positions = engine.cache.positions() snapshots = engine.cache.position_snapshots() ``` ### Portfolio statistics The backtest result provides performance metrics: ```python # Access backtest result statistics result = engine.get_result() # Get different categories of statistics stats_pnls = result.stats_pnls # Keyed by currency code, then statistic name stats_returns = result.stats_returns # Keyed by statistic name stats_general = result.stats_general # Keyed by statistic name ``` Each statistic contributes to the categories for which it implements a calculation: realized PnLs, returns, or positions. A custom statistic can contribute to more than one category. :::info See the [Portfolio guide](portfolio.md#portfolio-statistics) for the default statistic set, how each category is derived, and the difference between position returns and portfolio returns. ::: ### Visualization NautilusTrader provides interactive tearsheets and plots via Plotly: ```python from nautilus_trader.analysis import create_tearsheet # After backtest run engine.run() # Generate interactive HTML tearsheet create_tearsheet(engine, output_path="tearsheet.html") ``` This creates an interactive HTML report with: - Equity curve - Drawdown analysis - Monthly returns heatmap - Performance statistics table - Returns distribution For more control, generate individual plots: ```python import pandas as pd from nautilus_trader.analysis import create_equity_curve returns = pd.Series( [0.01, -0.005, 0.002], index=pd.date_range("2024-01-01", periods=3, tz="UTC"), ) fig = create_equity_curve(returns, title="My Strategy Equity") fig.show() # Display in browser fig.write_image("equity.png") # Export to PNG (requires kaleido) ``` Install visualization dependencies: ```bash uv pip install --pre "nautilus_trader[visualization]" ``` ## Report generation patterns ### Live trading During live trading, generate reports periodically: ```python from datetime import timedelta from nautilus_trader.analysis import ReportProvider from nautilus_trader.common import DataActor from nautilus_trader.common import TimeEvent class ReportingActor(DataActor): def on_start(self) -> None: # Schedule periodic reporting self.clock.set_timer( name="generate_reports", interval=timedelta(minutes=30), callback=self.generate_reports, ) def generate_reports(self, event: TimeEvent) -> None: # Generate and log reports positions_report = ReportProvider.generate_positions_report( positions=self.cache.positions(), snapshots=self.cache.position_snapshots(), ) # Save or transmit report positions_report.to_csv(f"positions_{event.ts_event}.csv") ``` ### Performance analysis For backtest analysis: ```python import pandas as pd # Run the backtest engine.run() # Collect results positions_closed = engine.cache.positions_closed() result = engine.get_result() stats_pnls = result.stats_pnls stats_returns = result.stats_returns stats_general = result.stats_general # Create summary dictionary results = { "total_positions": len(positions_closed), "pnl_total": stats_pnls.get("USD", {}).get("PnL (total)"), "win_rate": stats_pnls.get("USD", {}).get("Win Rate"), "sharpe_ratio": stats_returns.get("Sharpe Ratio (252 days)"), "profit_factor": stats_returns.get("Profit Factor"), "long_ratio": stats_general.get("Long Ratio"), } # Display results results_df = pd.DataFrame([results]) print(results_df.T) # Transpose for vertical display ``` :::info Reports are generated from in-memory data structures. For large-scale analysis or long-running systems, consider persisting reports to a database for efficient querying. See the [Cache guide](cache.md) for persistence options. ::: ## Integration with other components The `ReportProvider` works with several system components: - **Cache**: Source of all trading data (orders, positions, accounts) for reports. - **Portfolio**: Computes its statistics from the same cache data independently, not from these reports. - **BacktestEngine**: Exposes the report methods used for post-run analysis and visualization. - **Position snapshots**: Required for accurate PnL reporting in `NETTING` OMS. ## Related guides - [Visualization](visualization.md) - Interactive tearsheets and charts from backtest results. - [Portfolio](portfolio.md) - Portfolio statistics and performance metrics. - [Backtesting](backtesting/) - Running backtests that generate reports. - [Cache](cache.md) - Cache system that stores data for reports. # Rust Source: https://nautilustrader.io/docs/latest/concepts/rust/ Nautilus has a complete Rust implementation under the `crates/` directory. You can write actors, strategies, run backtests, and trade live without Python. The domain model is shared with the Python package, which runs user components on the same Rust engine through PyO3. :::warning The Rust API is under active development. Method signatures and trait requirements may change between releases. ::: ## System implementations Nautilus has two paths. Choose the one that matches how you want to author and deploy the system. - **Rust**: Pure Rust under `crates/`. Runs without Python. - **Python**: [Python user components](python.md) running on the Rust core through PyO3 bindings under `python/nautilus_trader/`. ### Capability matrix | Component | Rust | Python | | --------------- | ---- | ------ | | Strategy | ✓ | ✓ | | Actor | ✓ | ✓ | | DataEngine | ✓ | ✓ | | ExecutionEngine | ✓ | ✓ | | RiskEngine | ✓ | ✓ | | BacktestEngine | ✓ | ✓ | | BacktestNode | ✓ | ✓ | | LiveNode | ✓ | ✓ | | OrderEmulator | ✓ | ✓ | | Matching engine | ✓ | ✓ | | Portfolio | ✓ | ✓ | | Accounts | ✓ | ✓ | | Cache | ✓ | ✓ | | MessageBus | ✓ | ✓ | | Data catalog | ✓ | ✓ | | Indicators | ✓ | ✓ | | Exec algorithms | TWAP | TWAP | | Controller | - | ✓ | | Tearsheets | - | ✓ | :::note The Controller runtime is implemented in Rust and powers the Python `Controller` base class. The matrix marks it absent for Rust because the supported registration path (importable controller configs) is Python-only. ::: ### Adapters | Adapter | Rust | Python | | ------------------- | ---- | ------ | | Architect AX | ✓ | ✓ | | Betfair | ✓ | ✓ | | Binance | ✓ | ✓ | | BitMEX | ✓ | ✓ | | Blockchain | ✓ | ✓ | | Bybit | ✓ | ✓ | | Coinbase | ✓ | ✓ | | Databento | ✓ | ✓ | | Deribit | ✓ | ✓ | | Derive | ✓ | ✓ | | dYdX | ✓ | ✓ | | Hyperliquid | ✓ | ✓ | | Interactive Brokers | ✓ | ✓ | | Kraken | ✓ | ✓ | | Lighter | ✓ | ✓ | | OKX | ✓ | ✓ | | Polymarket | ✓ | ✓ | | Sandbox | ✓ | ✓ | | Tardis | ✓ | ✓ | ### Choosing a path - **Rust** gives native performance without a Python runtime. All core trading functionality is available. Use it for latency-sensitive deployments or teams that prefer a compiled language. - **Python** keeps the Python authoring experience. User components (actors, strategies) run on the Rust core for data processing and execution. ## Project setup The Nautilus crates are published to [crates.io](https://crates.io/crates/nautilus-backtest). Add them to your `Cargo.toml`: ```toml [dependencies] nautilus-backtest = "0.63" nautilus-common = "0.63" nautilus-execution = "0.63" nautilus-model = { version = "0.63", features = ["test-support"] } nautilus-trading = { version = "0.63", features = ["examples"] } anyhow = "1" log = "0.4" ``` For live trading, add the live crate and the adapter for your venue: ```toml [dependencies] nautilus-live = "0.63" nautilus-okx = "0.63" ``` To track the latest development branch, point all Nautilus dependencies at the same git source to avoid type mismatches between crates.io and git versions: ```toml [dependencies] nautilus-backtest = { git = "https://github.com/nautechsystems/nautilus_trader.git", branch = "develop" } nautilus-common = { git = "https://github.com/nautechsystems/nautilus_trader.git", branch = "develop" } nautilus-execution = { git = "https://github.com/nautechsystems/nautilus_trader.git", branch = "develop" } nautilus-model = { git = "https://github.com/nautechsystems/nautilus_trader.git", branch = "develop", features = ["test-support"] } nautilus-trading = { git = "https://github.com/nautechsystems/nautilus_trader.git", branch = "develop", features = ["examples"] } ``` The minimum supported Rust version (MSRV) is **1.98.1**. ### Feature flags | Flag | Crate | Effect | | ---------------- | ------------------- | --------------------------------------------------- | | `high-precision` | `nautilus-model` | 16-digit fixed precision (default is 9). | | `test-support` | `nautilus-model` | Test fixtures, builders, specs, and defaults. | | `examples` | `nautilus-trading` | Example strategies (`EmaCross`, `GridMarketMaker`). | | `streaming` | `nautilus-backtest` | Catalog-based data streaming via `BacktestNode`. | | `defi` | `nautilus-model` | DeFi data types. Implies `high-precision`. | :::tip Standard 9-digit precision handles most traditional finance instruments. Enable `high-precision` for crypto venues where prices or quantities need more than nine decimal places. ::: ### Memory allocator The `nautilus` CLI and Python wheels use [mimalloc](https://crates.io/crates/mimalloc) for Rust allocations. A Rust binary chooses its own allocator, so add mimalloc to yours to match: ```toml [dependencies] mimalloc = "0.1" ``` ```rust use mimalloc::MiMalloc; use nautilus_common::logging::headers::register_allocator_mimalloc; #[global_allocator] static GLOBAL: MiMalloc = MiMalloc; fn main() { register_allocator_mimalloc(); } ``` Declaring `GLOBAL` selects mimalloc. Call `register_allocator_mimalloc` at the start of `main`, before constructing a Nautilus node, so the version header reports `allocator: mimalloc `. Registration **only updates the header metadata**; it does not select the allocator. The default system allocator also works. Measure throughput and resident memory on your workload and platform when comparing allocator choices. See the [architecture guide](architecture.md#memory-allocation) for background. ## Actors An actor receives market data, custom data/signals, and system events but does not manage orders. Implement the `DataActor` trait and use `nautilus_actor!` to wire your `DataActorCore` field into the runtime contract. Your type implements or derives `Debug`; the macro supplies the native runtime wiring. User code normally uses the `DataActor` facade methods for subscriptions, cache access, and clock access. ### Handler methods Override any handler on the `DataActor` trait to receive the corresponding data or event. All handlers have default no-op implementations, so you only override what you need. | Handler | Receives | | ---------------------- | ------------------------- | | `on_start` | Actor started. | | `on_stop` | Actor stopped. | | `on_quote` | `QuoteTick` | | `on_trade` | `TradeTick` | | `on_bar` | `Bar` | | `on_book_deltas` | `OrderBookDeltas` | | `on_book` | `OrderBook` (at interval) | | `on_instrument` | `InstrumentAny` | | `on_mark_price` | `MarkPriceUpdate` | | `on_index_price` | `IndexPriceUpdate` | | `on_funding_rate` | `FundingRateUpdate` | | `on_option_greeks` | `OptionGreeks` | | `on_option_chain` | `OptionChainSlice` | | `on_instrument_status` | `InstrumentStatus` | | `on_time_event` | `TimeEvent` | For a step-by-step walkthrough, see the [Write an Actor (Rust)](../how_to/write_rust_actor.md) how-to guide. For a complete example, see [`BookImbalanceActor`](https://github.com/nautechsystems/nautilus_trader/tree/master/crates/trading/src/examples/actors/imbalance). ## Strategies A strategy extends an actor with order management. Implement `DataActor` for data handling and use `nautilus_strategy!` to wire your `StrategyCore` field into the strategy runtime contract. `StrategyCore` stores the runtime strategy state; normal strategy logic reaches it through facade methods on `self`. Runtime registration requires the native wiring generated by the macro, but normal strategy logic uses `Strategy` methods and the facade methods on `self`. Strategies also override order event handlers on the `Strategy` trait, such as `on_order_filled` (`OrderFilled`) and `on_order_canceled` (`OrderCanceled`). ### Order management The `Strategy` trait provides order methods through the facade: | Method | Action | | --------------------- | ----------------------------------------- | | `submit_order` | Submit a new order to the venue. | | `submit_order_list` | Submit a list of contingent orders. | | `modify_order` | Modify price, quantity, or trigger price. | | `cancel_order` | Cancel a specific order. | | `cancel_orders` | Cancel a filtered set of orders. | | `cancel_all_orders` | Cancel all orders for an instrument. | | `close_position` | Close a position with a market order. | | `close_all_positions` | Close all open positions. | The `OrderApi` (accessed via `self.order()`) builds orders and order lists: - `generate_client_order_id` - `generate_order_list_id` - `market` - `limit` - `stop_market` - `stop_limit` - `market_to_limit` - `market_if_touched` - `limit_if_touched` - `trailing_stop_market` - `trailing_stop_limit` - `bracket` - `create_list` ### Core wiring macros Rust actors, strategies, and execution algorithms keep their runtime core as a struct field. The macros tell the traits where that field lives. | Macro | Core field | Generates | | ---------------------------------------------- | ------------------------ | ------------------------------ | | `nautilus_actor!(Type)` | `DataActorCore` | Runtime wiring. | | `nautilus_strategy!(Type)` | `StrategyCore` | Runtime wiring and `Strategy`. | | `nautilus_execution_algorithm!(Type, { ... })` | `ExecutionAlgorithmCore` | Runtime wiring and algorithm. | The macros expect a field named `core`; pass a field name as the second argument when needed. They do not make the actor, strategy, or `StrategyCore` deref to runtime internals. The execution algorithm macro takes an `on_order()` implementation block because that method defines the algorithm's required order handling. ### Native traits Use facade methods by default: - `actor_id()` - `trader_id()` - `is_registered()` - `config()` - `strategy_id()` - `clock()` - `cache()` - `order()` - `portfolio()` `DataActorNative`, `StrategyNative`, and `ExecutionAlgorithmNative` are for native-only access below that facade. This section documents engine, runtime, and explicit latency-sensitive native Rust code, not the portable authoring path. | Authoring path | Native traits? | Normal API | | ------------------------- | ---------------- | ----------------------------------- | | Native Rust binary | Only when needed | `Strategy` and `DataActor` facades. | | Rust launched from Python | Only when needed | Same as native Rust. | | Python-authored component | No | Facades only. | Native traits expose borrowed core state, `Rc>`, and runtime references. Use them when native Rust code intentionally accepts those borrow rules for an explicit latency-sensitive path. Engine, runtime, registration, PyO3, and testkit code can import `DataActorNative`, `StrategyNative`, or `ExecutionAlgorithmNative` when they need actor-core, strategy-core, or execution-algorithm-core access. Do not use them in ordinary portable actor, strategy, or execution algorithm logic or Python-authored components, because those types do not cross the Python boundary. `ExecutionAlgorithmCore` owns a `DataActorCore`, but it does not deref to one. Normal execution algorithm logic should use `id()`, `actor_id()`, `trader_id()`, `clock()`, and `cache()`. Reach for `ExecutionAlgorithmNative` only when the code needs native execution-algorithm state. Choose the smallest native handle and keep each borrow scoped. Use `order()` for normal strategy order construction. Reach for `order_factory()` only when native code needs the raw mutable factory borrow. #### `DataActorNative` methods | Native method | Return shape | Use when | | ------------- | ------------------------ | ------------------------------- | | `core()` | `&DataActorCore` | Read actor internals. | | `core_mut()` | `&mut DataActorCore` | Mutate actor internals. | | `clock_mut()` | `RefMut<'_, dyn Clock>` | Need a mutable clock borrow. | | `clock_rc()` | `Rc>` | Store or pass the shared clock. | | `cache_ref()` | `Ref<'_, Cache>` | Need short live-cache reads. | | `cache_rc()` | `Rc>` | Mutate, store, or pass cache. | #### `StrategyNative` methods | Native method | Return shape | Use when | | --------------------- | --------------------------- | -------------------------------- | | `strategy_core()` | `&StrategyCore` | Read strategy internals. | | `strategy_core_mut()` | `&mut StrategyCore` | Mutate strategy internals. | | `order_factory()` | `RefMut<'_, OrderFactory>` | Need raw mutable factory borrow. | | `order_factory_rc()` | `Rc>` | Store or pass the factory. | | `portfolio_rc()` | `Rc>` | Store or pass the portfolio. | #### `ExecutionAlgorithmNative` methods | Native method | Return shape | Use when | | --------------------------- | ----------------------------- | ------------------------------------- | | `exec_algorithm_core()` | `&ExecutionAlgorithmCore` | Read execution algorithm internals. | | `exec_algorithm_core_mut()` | `&mut ExecutionAlgorithmCore` | Mutate execution algorithm internals. | For a step-by-step walkthrough, see the [Write a Strategy (Rust)](../how_to/write_rust_strategy.md) how-to guide. For complete examples, see [`EmaCross`](https://github.com/nautechsystems/nautilus_trader/tree/master/crates/trading/src/examples/strategies/ema_cross) and [`GridMarketMaker`](https://github.com/nautechsystems/nautilus_trader/tree/master/crates/trading/src/examples/strategies/grid_mm). ### Running Rust components Rust strategies and actors can run through two paths. The examples below use strategies, but the same pattern applies to bundled actors via `add_actor` (pure Rust) and `add_builtin_actor` (from Python). #### Pure Rust Write your strategy and `main` function in Rust, then build a standalone binary with `cargo build`. This path requires no Python runtime. ```rust let strategy = GridMarketMaker::new(config); node.add_strategy(strategy)?; node.run().await?; ``` See [Run Live Trading (Rust)](../how_to/run_rust_live_trading.md) for a full walkthrough. #### Built-in examples from Python Pass a type name and config to `add_builtin_strategy` to register a built-in example strategy from Python. This path exists to single-source the bundled example strategy code across Rust and Python docs, examples, and tests. It is not a first-class extension path for adding native strategies. For custom native components, use pure Rust. ```python from nautilus_trader.trading import GridMarketMakerConfig config = GridMarketMakerConfig( instrument_id=InstrumentId.from_str("BTC-USDT-SWAP.OKX"), max_position=Quantity.from_str("10.0"), trade_size=Quantity.from_str("0.1"), num_levels=5, grid_step_bps=15, ) node.add_builtin_strategy("GridMarketMaker", config) ``` Built-in strategy configs: | Config | Strategy | | ---------------------------- | ---------------------- | | `CompositeMarketMakerConfig` | `CompositeMarketMaker` | | `DeltaNeutralVolConfig` | `DeltaNeutralVol` | | `EmaCrossConfig` | `EmaCross` | | `ExecTesterConfig` | `ExecTester` | | `GridMarketMakerConfig` | `GridMarketMaker` | | `HurstVpinDirectionalConfig` | `HurstVpinDirectional` | `add_builtin_actor` follows the same bundled-only rule for actors used by examples and tests. Built-in actor configs (via `add_builtin_actor`): | Config | Actor | | -------------------------- | -------------------- | | `BookImbalanceActorConfig` | `BookImbalanceActor` | | `DataTesterConfig` | `DataTester` | ## Backtesting For annotated walkthroughs of both APIs, see the [Run a Backtest (Rust)](../how_to/run_rust_backtest.md) how-to guide. ### `BacktestEngine` (low-level API) Construct the engine, add venues and instruments, load data, register strategies, and run. See the full working example: ```bash cargo run -p nautilus-backtest --features examples --example engine-ema-cross ``` Source: [`crates/backtest/examples/engine_ema_cross.rs`](https://github.com/nautechsystems/nautilus_trader/tree/master/crates/backtest/examples/engine_ema_cross.rs) ### `BacktestNode` (high-level API) Loads data from a `ParquetDataCatalog` and supports streaming in configurable chunk sizes. Requires the `streaming` feature on `nautilus-backtest`. See the full working example: ```bash cargo run -p nautilus-backtest --features examples,streaming --example node-ema-cross ``` Source: [`crates/backtest/examples/node_ema_cross.rs`](https://github.com/nautechsystems/nautilus_trader/tree/master/crates/backtest/examples/node_ema_cross.rs) ## Live trading For an annotated walkthrough, see the [Run Live Trading (Rust)](../how_to/run_rust_live_trading.md) how-to guide. The `LiveNode` connects to real venues and data sources through adapter clients. The builder pattern configures data and execution clients, then `run()` starts the async event loop. Each adapter provides its own factory and config types. | Adapter | Example | | ------------------- | ----------------------------------------------- | | Architect AX | `crates/adapters/architect_ax/examples/` | | Betfair | `crates/adapters/betfair/examples/` | | Binance | `crates/adapters/binance/examples/` | | BitMEX | `crates/adapters/bitmex/examples/` | | Blockchain | `crates/adapters/blockchain/examples/` | | Bybit | `crates/adapters/bybit/examples/` | | Coinbase | `crates/adapters/coinbase/examples/` | | Databento | `crates/adapters/databento/examples/` | | Deribit | `crates/adapters/deribit/examples/` | | Derive | `crates/adapters/derive/examples/` | | dYdX | `crates/adapters/dydx/examples/` | | Hyperliquid | `crates/adapters/hyperliquid/examples/` | | Interactive Brokers | `crates/adapters/interactive_brokers/examples/` | | Kraken | `crates/adapters/kraken/examples/` | | Lighter | `crates/adapters/lighter/examples/` | | OKX | `crates/adapters/okx/examples/` | | Polymarket | `crates/adapters/polymarket/examples/` | | Sandbox | `crates/adapters/sandbox/examples/` | | Tardis | `crates/adapters/tardis/examples/` | Most adapters include `node_data_tester.rs` and `node_exec_tester.rs` examples. These test data requests, streaming, and order execution against live venues. ## Related guides - [Python](python.md) - Python ownership, runtime, and public API boundaries. - [Write an Actor (Rust)](../how_to/write_rust_actor.md) - Step-by-step actor walkthrough. - [Write a Strategy (Rust)](../how_to/write_rust_strategy.md) - Step-by-step strategy walkthrough. - [Run a Backtest (Rust)](../how_to/run_rust_backtest.md) - BacktestEngine and BacktestNode usage. - [Run Live Trading (Rust)](../how_to/run_rust_live_trading.md) - LiveNode setup and venue connection. - [Architecture](architecture.md) - System design and data/execution flow. - [Actors](actors.md) - Actor concepts (applies to both Python and Rust). - [Strategies](strategies.md) - Strategy concepts and handler reference. - [Events](events/) - Event types and handler dispatch. - [Backtesting](backtesting/) - Backtest concepts and matching engine behavior. # Strategies Source: https://nautilustrader.io/docs/latest/concepts/strategies/ A strategy inherits the `Strategy` class and implements the methods its logic requires. `Strategy` builds on `DataActor` and adds order management. **Capabilities**: - Historical data requests. - Live data feed subscriptions. - Setting time alerts or timers. - Cache access. - Portfolio access. - Creating and managing orders and positions. :::tip Review the [Actors](actors.md) guide before developing a strategy. It covers the subscription, request, and callback behavior a strategy inherits. ::: Add strategies to a Nautilus system in any [environment context](architecture.md#environment-contexts). They start sending commands and receiving events based on their logic as soon as the system starts. These building blocks of data ingest, event handling, and order management (discussed below) support any strategy type, including directional, momentum, re-balancing, pairs, and market making. There are two main parts of a Nautilus trading strategy: - The strategy implementation itself, defined by inheriting the `Strategy` class. - The *optional* strategy configuration, defined by inheriting the `StrategyConfig` class. :::note The same strategy source can run in backtest and live environments. Live execution still introduces venue, transport, timing, persistence, external-activity, and reconciliation behavior that a simulation may not reproduce. See [Backtest and live differences](live.md#backtest-and-live-differences). ::: See the [`Strategy` API Reference](/docs/python-api-latest/trading.html) for all available methods. :::info Rust implementation Rust strategy authors implement the `DataActor` callbacks they need and use `nautilus_strategy!` to generate the `Strategy` implementation, then call facade methods such as `clock()`, `cache()`, `order()`, and `portfolio()` on `self`. `DataActorNative` is native-only access to runtime wiring and actor-core state; `StrategyNative` exposes borrowed strategy state such as order factory, order manager, and portfolio access. Import them only for same-binary performance paths or internal runtime wiring. ::: ## Strategy implementation A trading strategy inherits from `Strategy`, so you must define a constructor. At minimum, initialize the base class: ```python from nautilus_trader.trading import Strategy class MyStrategy(Strategy): def __init__(self) -> None: super().__init__() # <-- the superclass must be called to initialize the strategy ``` From here, you can implement handlers as necessary to perform actions based on state transitions and events. :::warning `clock`, `cache`, `portfolio`, and `order_factory` raise a `RuntimeError` until the strategy is registered with a trader, which happens after `__init__` returns. Initialize plain state in the constructor and do system work in `on_start()`. ::: ### Handlers Handlers are methods on the `Strategy` class that perform actions based on events or state changes. These methods use the `on_*` prefix. Implement any or all of them as your strategy requires. Multiple handlers exist for similar event types to give you control over granularity. Respond to a specific event with a dedicated handler, or use a generic handler for a range of related events (using typical switch statement logic). The system calls handlers in sequence from **most specific to most general**. Subscribed data, order, and position handlers dispatch only while the strategy is `RUNNING`. Messages that arrive in any other state are logged but not passed to your handlers. Request responses are not gated this way: an `on_historical_*` handler still runs if its response lands after the strategy stops. #### Stateful actions Lifecycle state changes trigger these handlers. Recommendations: - Use the `on_start` method to initialize your strategy (e.g., fetch instruments, subscribe to data). - Use the `on_stop` method for cleanup tasks (e.g., cancel open orders, close open positions, unsubscribe from data). ```python def on_start(self) -> None: def on_stop(self) -> None: def on_resume(self) -> None: def on_reset(self) -> None: def on_dispose(self) -> None: def on_degrade(self) -> None: def on_fault(self) -> None: def on_save(self) -> dict[str, bytes]: # Returns user-defined dictionary of state to be saved def on_load(self, state: dict[str, bytes]) -> None: ``` #### Data handling These handlers receive data updates, including built-in market data and custom user-defined data. ```python from collections.abc import Sequence from typing import Any from nautilus_trader.common import Signal from nautilus_trader.model import Bar from nautilus_trader.model import CustomData from nautilus_trader.model import FundingRateUpdate from nautilus_trader.model import IndexPriceUpdate from nautilus_trader.model import InstrumentClose from nautilus_trader.model import InstrumentStatus from nautilus_trader.model import MarkPriceUpdate from nautilus_trader.model import OptionChainSlice from nautilus_trader.model import OptionGreeks from nautilus_trader.model import OrderBook from nautilus_trader.model import OrderBookDelta from nautilus_trader.model import OrderBookDeltas from nautilus_trader.model import OrderBookDepth10 from nautilus_trader.model import QuoteTick from nautilus_trader.model import TradeTick def on_book_deltas(self, deltas: OrderBookDeltas) -> None: def on_book_depth(self, depth: OrderBookDepth10) -> None: def on_book(self, order_book: OrderBook) -> None: def on_quote(self, tick: QuoteTick) -> None: def on_trade(self, tick: TradeTick) -> None: def on_bar(self, bar: Bar) -> None: def on_mark_price(self, mark_price: MarkPriceUpdate) -> None: def on_index_price(self, index_price: IndexPriceUpdate) -> None: def on_funding_rate(self, funding_rate: FundingRateUpdate) -> None: def on_instrument(self, instrument: Any) -> None: def on_instrument_status(self, data: InstrumentStatus) -> None: def on_instrument_close(self, data: InstrumentClose) -> None: def on_option_greeks(self, greeks: OptionGreeks) -> None: def on_option_chain(self, chain: OptionChainSlice) -> None: def on_historical_data(self, data: CustomData | Sequence[CustomData]) -> None: def on_historical_book_deltas(self, deltas: Sequence[OrderBookDelta]) -> None: def on_historical_book_depth(self, depths: Sequence[OrderBookDepth10]) -> None: def on_historical_quotes(self, quotes: Sequence[QuoteTick]) -> None: def on_historical_trades(self, trades: Sequence[TradeTick]) -> None: def on_historical_bars(self, bars: Sequence[Bar]) -> None: def on_historical_mark_prices(self, mark_prices: Sequence[MarkPriceUpdate]) -> None: def on_historical_index_prices(self, index_prices: Sequence[IndexPriceUpdate]) -> None: def on_historical_funding_rates(self, rates: Sequence[FundingRateUpdate]) -> None: def on_data(self, data: CustomData) -> None: def on_signal(self, signal: Signal) -> None: ``` Subscribed updates and request responses reach different handlers. See [Actors: callback handlers](actors.md#callback-handlers) for the operation-to-handler mapping. #### Order management These handlers receive events related to orders. `OrderEvent` type messages are passed to handlers in the following sequence: 1. Specific handler (e.g., `on_order_accepted`, `on_order_rejected`, etc.) 2. `on_order_event(...)` ```python from typing import Any from nautilus_trader.model import OrderAccepted from nautilus_trader.model import OrderCanceled from nautilus_trader.model import OrderCancelRejected from nautilus_trader.model import OrderDenied from nautilus_trader.model import OrderEmulated from nautilus_trader.model import OrderExpired from nautilus_trader.model import OrderFilled from nautilus_trader.model import OrderFillVoided from nautilus_trader.model import OrderInitialized from nautilus_trader.model import OrderModifyRejected from nautilus_trader.model import OrderPendingCancel from nautilus_trader.model import OrderPendingUpdate from nautilus_trader.model import OrderRejected from nautilus_trader.model import OrderReleased from nautilus_trader.model import OrderSubmitted from nautilus_trader.model import OrderTriggered from nautilus_trader.model import OrderUpdated def on_order_initialized(self, event: OrderInitialized) -> None: def on_order_denied(self, event: OrderDenied) -> None: def on_order_emulated(self, event: OrderEmulated) -> None: def on_order_released(self, event: OrderReleased) -> None: def on_order_submitted(self, event: OrderSubmitted) -> None: def on_order_rejected(self, event: OrderRejected) -> None: def on_order_accepted(self, event: OrderAccepted) -> None: def on_order_canceled(self, event: OrderCanceled) -> None: def on_order_expired(self, event: OrderExpired) -> None: def on_order_triggered(self, event: OrderTriggered) -> None: def on_order_pending_update(self, event: OrderPendingUpdate) -> None: def on_order_pending_cancel(self, event: OrderPendingCancel) -> None: def on_order_modify_rejected(self, event: OrderModifyRejected) -> None: def on_order_cancel_rejected(self, event: OrderCancelRejected) -> None: def on_order_updated(self, event: OrderUpdated) -> None: def on_order_filled(self, event: OrderFilled) -> None: def on_order_fill_voided(self, event: OrderFillVoided) -> None: def on_order_event(self, event: Any) -> None: # All order event messages are eventually passed to this handler ``` :::note The Python API does not export an `OrderEvent` base type. `on_order_event(...)` receives the same concrete event object the specific handler received, such as an `OrderAccepted`. ::: #### Position management These handlers receive events related to positions. `PositionEvent` type messages are passed to handlers in the following sequence: 1. Specific handler (e.g., `on_position_opened`, `on_position_changed`, etc.) 2. `on_position_event(...)` ```python from typing import Any from nautilus_trader.model import PositionChanged from nautilus_trader.model import PositionClosed from nautilus_trader.model import PositionOpened def on_position_opened(self, event: PositionOpened) -> None: def on_position_changed(self, event: PositionChanged) -> None: def on_position_closed(self, event: PositionClosed) -> None: def on_position_event(self, event: Any) -> None: # All position event messages are eventually passed to this handler ``` As with order events, the Python API does not export a `PositionEvent` base type, so `on_position_event(...)` receives the concrete event object. Use `on_time_event()` for timer events, `on_order_event()` for aggregate order events, and `on_position_event()` for aggregate position events. The Python API does not expose a generic `on_event()` hook. #### Handler example The following example shows a typical `on_start` handler method implementation (taken from the example EMA cross strategy). Here we can see the following: - Indicators being registered to receive bar updates. - Historical data being requested (to hydrate the indicators). - Live data being subscribed to. The cache check matters in live trading. Direct subscriptions assume the instrument was loaded by the instrument provider config or by an earlier instrument request. ```python def on_start(self) -> None: """ Actions to be performed on strategy start. """ self.instrument = self.cache.instrument(self.instrument_id) if self.instrument is None: self.log.error(f"Could not find instrument for {self.instrument_id}") self.stop() # Transitions strategy to STOPPED state return # Register the indicators for updating self.register_indicator_for_bars(self.bar_type, self.fast_ema) self.register_indicator_for_bars(self.bar_type, self.slow_ema) # Get historical data and subscribe to live data self.request_bars(self.bar_type) self.subscribe_bars(self.bar_type) self.subscribe_quotes(self.instrument_id) ``` Registered indicators receive the bars from a request response before `on_historical_bars()` runs, so a request hydrates them. Check `indicators_initialized()` before acting on indicator values. ### Clock and timers Strategies have access to a `Clock` which provides a number of methods for creating different timestamps, as well as setting time alerts or timers to trigger `TimeEvent`s. See the [`Clock` API Reference](/docs/python-api-latest/common.html) for all available methods. #### Current timestamps While there are multiple ways to obtain current timestamps, here are two commonly used methods as examples: To get the current UTC timestamp as a tz-aware `datetime`: ```python from datetime import datetime now: datetime = self.clock.utc_now() ``` To get the current UTC timestamp as nanoseconds since the UNIX epoch: ```python unix_nanos: int = self.clock.timestamp_ns() ``` #### Time alerts Time alerts can be set which will result in a `TimeEvent` being dispatched to the `on_time_event` handler at the specified alert time. In live trading, scheduling and queued work can delay delivery; the alert time is not a latency guarantee. This example sets a time alert to trigger one minute from the current time: ```python from datetime import timedelta # Fire a TimeEvent one minute from now self.clock.set_time_alert( name="MyTimeAlert1", alert_time=self.clock.utc_now() + timedelta(minutes=1), ) ``` #### Timers Continuous timers can be set up which will generate a `TimeEvent` at regular intervals until the timer expires or is canceled. This example sets a timer to fire once per minute. The timer starts immediately and its first event fires one interval later; pass `fire_immediately=True` to fire at the start time instead: ```python from datetime import timedelta # Fire a TimeEvent every minute self.clock.set_timer( name="MyTimer1", interval=timedelta(minutes=1), ) ``` Pass a `callback` to route `TimeEvent` objects to your own method rather than `on_time_event`. Timer names share the clock namespace, so use names unique to the component. See [Actors: timers and alerts](actors.md#timers-and-alerts). ### Cache access The trader's central `Cache` stores data and execution objects (orders, positions, etc). Many methods are available with filtering. Here are some basic use cases. #### Fetching data The following example fetches data from the cache (assuming some instrument ID attribute is assigned). These methods return `None` if the requested data is not available. ```python last_quote = self.cache.quote(self.instrument_id) last_trade = self.cache.trade(self.instrument_id) last_bar = self.cache.bar(bar_type) ``` #### Fetching execution objects The following example shows how individual order and position objects can be fetched from the cache: ```python order = self.cache.order(client_order_id) position = self.cache.position(position_id) ``` See the [`Cache` API Reference](/docs/python-api-latest/cache.html) for all available methods. ### Portfolio access The trader's central `Portfolio` provides account and positional information. The following shows a general outline of available methods. #### Account and positional information ```python import decimal import typing from nautilus_trader.model import AccountId from nautilus_trader.model import Currency from nautilus_trader.model import InstrumentId from nautilus_trader.model import Money from nautilus_trader.model import Price from nautilus_trader.model import Venue def account( self, venue: Venue | None = None, account_id: AccountId | None = None, ) -> typing.Any | None def balances_locked( self, venue: Venue | None = None, account_id: AccountId | None = None, ) -> dict[Currency, Money] | None def instrument_initial_margins( self, venue: Venue | None = None, account_id: AccountId | None = None, ) -> dict[InstrumentId, Money] | None def instrument_maintenance_margins( self, venue: Venue | None = None, account_id: AccountId | None = None, ) -> dict[InstrumentId, Money] | None def unrealized_pnls( self, venue: Venue | None = None, account_id: AccountId | None = None, target_currency: Currency | None = None, ) -> dict[Currency, Money] def realized_pnls( self, venue: Venue | None = None, account_id: AccountId | None = None, target_currency: Currency | None = None, ) -> dict[Currency, Money] def total_pnls( self, venue: Venue | None = None, account_id: AccountId | None = None, target_currency: Currency | None = None, ) -> dict[Currency, Money] def net_exposures( self, venue: Venue | None = None, account_id: AccountId | None = None, target_currency: Currency | None = None, ) -> dict[Currency, Money] | None def unrealized_pnl( self, instrument_id: InstrumentId, price: Price | None = None, account_id: AccountId | None = None, target_currency: Currency | None = None, ) -> Money | None def realized_pnl( self, instrument_id: InstrumentId, account_id: AccountId | None = None, target_currency: Currency | None = None, ) -> Money | None def total_pnl( self, instrument_id: InstrumentId, price: Price | None = None, account_id: AccountId | None = None, target_currency: Currency | None = None, ) -> Money | None def net_exposure( self, instrument_id: InstrumentId, price: Price | None = None, account_id: AccountId | None = None, target_currency: Currency | None = None, ) -> Money | None def net_position( self, instrument_id: InstrumentId, account_id: AccountId | None = None, ) -> decimal.Decimal def is_net_long(self, instrument_id: InstrumentId, account_id: AccountId | None = None) -> bool def is_net_short(self, instrument_id: InstrumentId, account_id: AccountId | None = None) -> bool def is_net_flat(self, instrument_id: InstrumentId, account_id: AccountId | None = None) -> bool def is_completely_net_flat(self, account_id: AccountId | None = None) -> bool ``` If both `venue` and `account_id` are supplied, they must resolve to the same account, otherwise the query raises `ValueError`. The Portfolio exposes queries to strategies; engine commands remain internal to the Rust runtime. See the [`Portfolio` API Reference](/docs/python-api-latest/portfolio.html) for all available methods. #### Reports and analysis Use `Portfolio.statistics()` and `Portfolio.snapshots(account_id)` for performance analysis. See the [Analysis API Reference](/docs/python-api-latest/analysis.html) and [Portfolio statistics](portfolio.md#portfolio-statistics) guide. The [Portfolio](portfolio.md) guide also covers equity, mark-to-market valuation, and multi-account query scope. ### Trading commands The following trading commands are available for order management. See also the [Execution](execution/) guide for the full flow through the system. #### Submitting orders An `OrderFactory` is provided on the base class for every `Strategy` as a convenience, reducing the amount of boilerplate required to create different `Order` objects (although these objects can still be initialized directly with the `Order.__init__(...)` constructor if the trader prefers). The component a `SubmitOrder` or `SubmitOrderList` command will flow to for execution depends on the following: - If an `emulation_trigger` is specified, the command will *firstly* be sent to the `OrderEmulator`. - If an `exec_algorithm_id` is specified (with no `emulation_trigger`), the command will *firstly* be sent to the relevant `ExecutionAlgorithm`. - Otherwise, the command will *firstly* be sent to the `RiskEngine`. This example submits a `LIMIT` BUY order for emulation (see [Emulated Orders](orders/emulated.md)): ```python from nautilus_trader.model import LimitOrder from nautilus_trader.model import OrderSide from nautilus_trader.model import TriggerType def buy(self) -> None: """ Users simple buy method (example). """ order: LimitOrder = self.order_factory.limit( instrument_id=self.instrument_id, order_side=OrderSide.BUY, quantity=self.instrument.make_qty(self.trade_size), price=self.instrument.make_price(5000.00), emulation_trigger=TriggerType.LAST_PRICE, ) self.submit_order(order) ``` :::info You can specify both order emulation and an execution algorithm. In this case, the order is first sent to the `OrderEmulator`, and upon release is then routed to the `ExecutionAlgorithm`. ::: This example submits a `MARKET` BUY order to a TWAP execution algorithm: ```python from nautilus_trader.model import ExecAlgorithmId from nautilus_trader.model import MarketOrder from nautilus_trader.model import OrderSide from nautilus_trader.model import TimeInForce def buy(self) -> None: """ Users simple buy method (example). """ order: MarketOrder = self.order_factory.market( instrument_id=self.instrument_id, order_side=OrderSide.BUY, quantity=self.instrument.make_qty(self.trade_size), time_in_force=TimeInForce.FOK, exec_algorithm_id=ExecAlgorithmId("TWAP"), exec_algorithm_params={"horizon_secs": "20", "interval_secs": "2.5"}, ) self.submit_order(order) ``` See [Execution algorithms](execution/algorithms.md) for TWAP parameter rules and spawned-order behavior. #### Canceling orders Orders can be canceled individually, as a batch, or all orders for an instrument (with an optional side filter). If the order is already *closed* or already pending cancel, then a warning will be logged. If the order is currently *open* then the status will become `PENDING_CANCEL`. Routing depends on the command and on the state of each order: - `cancel_order(...)` goes *firstly* to the `OrderEmulator` when the order is emulated, to the relevant `ExecutionAlgorithm` when the order has an `exec_algorithm_id` and is still active within the local system, and to the `ExecutionEngine` otherwise. - `cancel_all_orders(...)` cancels each matching order associated with the strategy by default. Each order follows the same routing as `cancel_order(...)`. - `cancel_orders(...)` always goes to the `ExecutionEngine` as a single `BatchCancelOrders` command. :::info Any managed GTD timer will also be canceled after the command has left the strategy. ::: The following shows how to cancel an individual order: ```python self.cancel_order(order.client_order_id) ``` The following shows how to cancel a batch of orders. Every order in the batch must be for the same instrument, and the batch must not include emulated or local orders: ```python from nautilus_trader.model import ClientOrderId client_order_ids: list[ClientOrderId] = [ order1.client_order_id, order2.client_order_id, order3.client_order_id, ] self.cancel_orders(client_order_ids) ``` The following shows how to cancel all orders: ```python self.cancel_all_orders(self.instrument_id) ``` :::warning Pass `strategy_only=False` to use the broad cancellation path. The strategy sends one `CancelAllOrders` command even when its cache has no matching order. The execution engine resolves one execution client and account, then routes venue, emulated, and execution-algorithm cancellation within that boundary. Matching orders associated with other strategies may be canceled, but orders assigned to other execution clients remain untouched. Use broad mode only when that cross-strategy scope is intended. See [Cancel-all routing](execution/index.md#cancel-all-routing) for the complete flow and client-selection rules. ::: #### Modifying orders Orders can be modified individually when emulated, or *open* on a venue (if supported). If the order is already *closed* or already pending cancel, then a warning will be logged. If the order is currently *open* then the status will become `PENDING_UPDATE`. :::warning At least one value must differ from the original order for the command to be valid. ::: The component a `ModifyOrder` command will flow to for execution depends on the following: - If the order is currently emulated, the command will *firstly* be sent to the `OrderEmulator`. - Otherwise, if the order has an `exec_algorithm_id` and is still active within the local system, the command will be sent to the relevant `ExecutionAlgorithm`. - Otherwise, the order will *firstly* be sent to the `RiskEngine`. :::info An `ExecutionAlgorithm` refuses a `ModifyOrder` for a primary order that is still active locally, and logs a warning without emitting an event. The algorithm's schedule is keyed to the quantity it computed, so applying the modification in place would break that state; the primary order is left unchanged. ::: The following shows how to modify the size of `LIMIT` BUY order currently *open* on a venue: ```python from nautilus_trader.model import Quantity new_quantity: Quantity = Quantity.from_int(5) self.modify_order(order.client_order_id, quantity=new_quantity) ``` :::info The price and trigger price can also be modified (when emulated or supported by a venue). ::: Use `modify_orders(...)` to send several modifications as a single `BatchModifyOrders` command to the `RiskEngine`. As with a batch cancel, every order must be for the same instrument, and the batch must not include emulated or local orders. #### Market exit The `market_exit()` method provides a graceful way to exit all positions and cancel all orders for a strategy. The strategy remains running after the exit completes, allowing you to re-enter positions later if desired. ```python self.market_exit() ``` The call logs a warning and returns without effect if the strategy is not `RUNNING`, or if an exit is already in progress. The market exit process: 1. Calls `on_market_exit()`. 2. Cancels all open and in-flight orders for the strategy. 3. Closes all open positions with market orders tagged `MARKET_EXIT`. 4. Periodically checks (at `market_exit_interval_ms`) until all orders resolve and positions close, re-submitting a closing order for any position still open once no orders remain working. 5. Calls `post_market_exit()` once flat, or after `market_exit_max_attempts` is reached, logging the orders and positions still outstanding. Two hooks are available for custom logic: - `on_market_exit()`: called when the exit process begins. - `post_market_exit()`: called when the exit process completes. ```python class MyStrategy(Strategy): def on_market_exit(self) -> None: self.log.info("Beginning market exit...") def post_market_exit(self) -> None: self.log.info("Market exit complete") ``` During a market exit, non-reduce-only orders are automatically denied with the reason `MARKET_EXIT_IN_PROGRESS`. The exit's own closing orders pass through because they carry the `MARKET_EXIT` tag. For order lists, if any order in the list is non-reduce-only, the entire list is denied to preserve list semantics (e.g., bracket orders with interdependencies). To check if an exit is in progress (e.g., to skip order submission logic), use `is_exiting()`: ```python def on_quote(self, tick: QuoteTick) -> None: if self.is_exiting(): return # Skip order logic during exit # ... normal order logic ``` To automatically perform a market exit when the strategy is stopped, set `manage_stop=True`: ```python config = StrategyConfig(manage_stop=True) ``` With this option, calling `stop()` first performs a market exit, then stops the strategy once flat or once `market_exit_max_attempts` is reached. Reaching the attempt limit can leave orders or positions outstanding. :::warning In a backtest, a managed stop requested at the end of the run cannot complete. `market_exit()` schedules its first completion check `market_exit_interval_ms` after the current time, which falls beyond the requested end, and the engine has already performed its final timer flush by then. No callback runs to observe that the exit is done, so the strategy ends the run still `RUNNING` and is reported at `ERROR` even when it is already flat. ::: Configuration options in `StrategyConfig`: - `manage_stop` (default: `False`): if `True`, `stop()` performs a market exit before stopping. - `market_exit_interval_ms` (default: `100`): interval between exit completion checks. - `market_exit_max_attempts` (default: `100`): maximum checks before completing the exit. - `market_exit_time_in_force` (default: `GTC`): time in force for closing market orders. - `market_exit_reduce_only` (default: `True`): if closing market orders should be reduce only. #### Closing positions Use `close_position(...)` and `close_all_positions(...)` to flatten without running the full market exit process. Both submit closing market orders and leave the strategy free to submit new orders. See the [Execution](execution/) guide. ## Strategy configuration A separate configuration class gives full flexibility over where and how a strategy is instantiated. Configurations serialize over the wire, enabling distributed backtesting and remote live trading. This is opt-in. You can skip configuration and pass parameters directly to your strategy constructor. If you want distributed backtests or remote live trading, define a configuration. `StrategyConfig` is implemented in Rust. Its `__new__` reads the base fields (`strategy_id`, `order_id_tag`, `oms_type`, and the rest) out of the constructor call and validates them, ignoring any keyword it does not recognize. There is no base `__init__`. A subclass therefore needs only three things: - Declare its own fields as keyword-only arguments, so no positional argument is matched against a base field. - Accept `**_kwargs` so the base fields pass through the subclass signature. - Call `super().__init__()` with no arguments, then assign its own fields. Do not give a custom field the same name as a base field: both constructors would read it, and `__new__` raises a `TypeError` when the value does not match the base field's type. Here is an example configuration: ```python from decimal import Decimal from nautilus_trader.config import StrategyConfig from nautilus_trader.model import Bar from nautilus_trader.model import BarType from nautilus_trader.model import InstrumentId from nautilus_trader.model import StrategyId from nautilus_trader.trading import Strategy # Configuration definition class MyStrategyConfig(StrategyConfig): def __init__( self, *, instrument_id: InstrumentId, bar_type: BarType, trade_size: Decimal, fast_ema_period: int = 10, slow_ema_period: int = 20, **_kwargs, ) -> None: super().__init__() self.instrument_id = instrument_id self.bar_type = bar_type self.trade_size = trade_size self.fast_ema_period = fast_ema_period self.slow_ema_period = slow_ema_period # Strategy definition class MyStrategy(Strategy): def __init__(self, config: MyStrategyConfig) -> None: # Always initialize the parent Strategy class # After this, configuration is stored and available via `self.config` super().__init__(config) # Custom state variables self.time_started = None self.count_of_processed_bars: int = 0 def on_start(self) -> None: self.time_started = self.clock.utc_now() # Remember time, when strategy started self.subscribe_bars( self.config.bar_type ) # See how configuration data are exposed via `self.config` def on_bar(self, bar: Bar): self.count_of_processed_bars += 1 # Update count of processed bars # Instantiate configuration with specific values. By setting: # - InstrumentId - we parameterize the instrument the strategy will trade. # - BarType - we parameterize bar-data, that strategy will trade. # - StrategyId - we name this instance, which also fixes its order ID tag. config = MyStrategyConfig( instrument_id=InstrumentId.from_str("ETHUSDT-PERP.BINANCE"), bar_type=BarType.from_str("ETHUSDT-PERP.BINANCE-15-MINUTE-LAST-EXTERNAL"), trade_size=Decimal("1"), strategy_id=StrategyId("MyStrategy-001"), ) # Pass configuration to our trading strategy. strategy = MyStrategy(config=config) ``` Access configuration values through `self.config`. This provides clear separation between: - Configuration data (accessed via `self.config`): - Contains initial settings, that define how the strategy works. - Example: `self.config.trade_size`, `self.config.instrument_id` - Strategy state variables (as direct attributes): - Track any custom state of the strategy. - Example: `self.time_started`, `self.count_of_processed_bars` This separation makes code easier to understand and maintain. :::note Even though it often makes sense to define a strategy which will trade a single instrument. The number of instruments a single strategy can work with is only limited by machine resources. ::: ### Claiming external orders Live adapters can report venue orders that a strategy did not submit or that NautilusTrader has not yet observed locally. NautilusTrader creates these as external orders and assigns them to the `EXTERNAL` strategy unless an active claim identifies another strategy as their owner. Set `external_order_instrument_ids` to list the instruments a strategy intends to claim for external orders, fills, and materialized reconciliation activity: ```python from nautilus_trader.config import StrategyConfig from nautilus_trader.model import InstrumentId config = StrategyConfig( external_order_instrument_ids=[ InstrumentId.from_str("ETHUSDT-PERP.BINANCE"), ], ) ``` `external_order_instrument_ids` is serializable configuration intent. When a live node registers the strategy, the runtime materializes that intent as active external order claims in the shared cache. Each claim assigns one `InstrumentId` to one `StrategyId`. Registration fails without changing the active claims if the shared cache is already borrowed, the list repeats an instrument, or any listed instrument already has a claim, including a claim for the same strategy. After registration, call `set_external_order_instrument_ids(...)` to replace the strategy's complete active claim set. The method retains listed claims already owned by the strategy, releases omitted claims, and acquires listed instruments that have no owner. Passing an empty list releases every claim owned by the strategy. If an instrument appears more than once or belongs to another strategy, the call fails and leaves the active claim set unchanged. A change affects ownership decisions made after the update; it does not reassign external orders already held in the cache. The strategy must be registered before it can update active claims, and the update fails if the shared cache is already borrowed. For Python strategies, this method changes active routing state; the original `strategy.config` object continues to show the construction intent. Stopping a strategy retains its active claims so routing remains stable if it restarts. Removing the strategy from its trader releases the claims. A cache reset also preserves them. :::warning Transferring an instrument between strategies requires two calls: the existing owner releases the instrument before the new owner claims it. The transfer is not atomic across strategies. An external report processed between those calls has no active claim and is assigned to the `EXTERNAL` strategy. ::: See [External order creation](execution/reconciliation.md#external-order-creation) for how execution and reconciliation use active claims. ### Managed GTD expiry It's possible for the strategy to manage expiry for orders with a time in force of GTD (*Good 'till Date*). This may be desirable if the exchange/broker does not support this time in force option, or for any reason you prefer the strategy to manage this. To use this option, pass `manage_gtd_expiry=True` to your `StrategyConfig`. When an order is submitted with a time in force of GTD, the strategy will automatically start an internal time alert. Once the internal GTD time alert is reached, the order will be canceled (if not already *closed*). On start, the strategy also reinstates alerts for its open GTD orders held in the cache, and cancels any whose expiry has already passed. When a cancel request is rejected, a running strategy restores a missing expiry alert for an open or inflight GTD order. If expiry has already passed, it immediately retries the cancel instead. Existing alerts are preserved. This runs before `on_order_cancel_rejected`, so an immediate retry can return the order to `PENDING_CANCEL` before the callback runs. A stopped strategy does not restore alerts or retry cancels on rejection. Some venues (such as Binance Futures) support the GTD time in force, so to avoid conflicts when using `manage_gtd_expiry` you should set `use_gtd=False` for your execution client config. ### Multiple strategies If you intend running multiple instances of the same strategy, with different configurations (such as trading different instruments), then each instance needs a **unique strategy ID and order ID tag**. The system must be able to identify which strategy various commands and events belong to. The order ID tag also keeps generated client order IDs unique across strategies for the same trader. #### Order ID tag Set `strategy_id` on each config. The runtime takes the order ID tag from the final hyphen-separated part of the strategy ID, so `MyStrategy-001` and `MyStrategy-002` produce the tags `001` and `002`. Supplying `order_id_tag` as well appends the tag to the runtime strategy ID, unless the ID already ends with that tag. For example, `strategy_id=StrategyId("MyStrategy-PRIMARY")` with `order_id_tag="ABC"` registers as `MyStrategy-PRIMARY-ABC`. A strategy registered without `strategy_id` takes its base ID from the strategy type name. An `order_id_tag` becomes the suffix, so `MyStrategy` with `order_id_tag="ABC"` registers as `MyStrategy-ABC`; without a tag, registration assigns the next numeric tag, starting with `000`. Because the runtime reads the tag back from the final hyphen-separated part of the strategy ID, an `order_id_tag` cannot contain a hyphen. `StrategyConfig(order_id_tag="A-B")` raises a `ValueError`, and so does a subclass constructed with that keyword. A config class that does not inherit from `StrategyConfig` carries the tag to registration instead, which raises a `RuntimeError`. The trader ID carries a tag in the same way, and its final hyphen-separated part reaches the same generated IDs. See [Configure a live trading node](../how_to/configure_live_trading.md) for the requirement that it stays unique across nodes. :::note The platform has built-in safety measures. Registering a duplicated strategy ID raises a `RuntimeError` indicating the strategy ID is already registered, and two different strategy IDs that share an order ID tag raise a `RuntimeError` reporting the tag conflict. ::: :::info Rust implementation Rust treats `StrategyConfig` as immutable construction input. The runtime `StrategyId` carries the order ID tag, matching the Python behavior. This keeps actor registration, client order ID generation, order list ID generation, and position ID generation aligned through `strategy_id.get_tag()`. ::: See the [`StrategyId` API Reference](/docs/python-api-latest/model/identifiers.html) for further details. ## Related guides - [Actors](actors.md) - Base class that strategies extend. - [Events](events/) - Event types and handler dispatch. - [Orders](orders/) - Order types and management from strategies. - [Backtesting](backtesting/) - Test strategies with historical data. # Synthetics Source: https://nautilustrader.io/docs/latest/concepts/synthetics/ **Synthetic instruments** are locally defined instruments whose prices derive from other instruments. They can combine components from one venue or many venues and expose the result as a standard Nautilus instrument with the synthetic venue code `SYNTH`. Synthetic instruments are useful for: - Enabling `DataActor` and `Strategy` components to subscribe to quote or trade feeds. - Triggering emulated orders from derived prices. - Constructing bars from synthetic quotes or trades. :::info Synthetic instruments cannot be traded directly. They exist locally within the platform and serve as analytical tools. In the future, Nautilus may support trading component instruments based on synthetic instrument behavior. ::: ## Formula language Each synthetic instrument defines a derivation formula. Nautilus evaluates this formula with its built-in numeric expression engine and converts the final numeric result to the synthetic `Price`. ### Supported syntax Formulas can reference component `InstrumentId` values directly, including IDs that contain `/` and `-`. | Construct | Example | Notes | | ------------------- | ---------------------------------------------- | --------------------------------------------------------------------- | | Component reference | `BTCUSDT.BINANCE` | Use the raw `InstrumentId` text. | | Component reference | `AUD/USD.SIM` | IDs containing `/` are valid. | | Component reference | `ETH-USDT-SWAP.OKX` | IDs containing `-` are valid. | | Numeric literal | `1`, `0.5`, `1.2e-3` | Evaluated with `f64` semantics. | | Boolean literal | `true`, `false` | Used in conditions and logical expressions. | | Parentheses | `(a + b) / 2` | Use parentheses to override precedence. | | Unary operators | `-x`, `!flag` | Unary `-` negates numbers. Unary `!` negates booleans. | | Binary operators | `+ - * / % ^`, `== !=`, `< <= > >=`, `&& \|\|` | Arithmetic is numeric. Logical operators are boolean. | | Local assignment | `spread = a - b; spread / 2` | Statements run from left to right. The formula must end with a value. | | Comments | `// line`, `/* block */` | Comments are ignored. | :::note New formulas should use raw `InstrumentId` values. For backward compatibility, formulas that replace `-` with `_` in component IDs remain accepted. ::: ### Operator precedence The expression engine evaluates operators in the following order, from highest precedence to lowest precedence: | Level | Operators | Notes | | ------- | -------------------- | ------------------------------------------------------------ | | Highest | `^` | Exponentiation. Right associative. | | | Unary `-`, unary `!` | `-2 ^ 2` evaluates as `-(2 ^ 2)`. | | | `*`, `/`, `%` | Multiplication, division, and modulo. | | | `+`, `-` | Addition and subtraction. | | | `<`, `<=`, `>`, `>=` | Numeric comparisons. | | | `==`, `!=` | Equality and inequality. Both sides must have the same type. | | Lowest | `&&`, `\|\|` | Boolean operators. | Assignments are not expression operators. Separate statements with `;`, and make the last statement the value you want the synthetic to produce. ### Built-in functions | Function | Signature | Notes | | -------- | -------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `abs` | `abs(x)` | Absolute value. | | `ceil` | `ceil(x)` | Ceiling. | | `floor` | `floor(x)` | Floor. | | `round` | `round(x)` | Round to the nearest integer using Rust `f64` rules. | | `min` | `min(x1, x2, ...)` | Accepts one or more numeric arguments. | | `max` | `max(x1, x2, ...)` | Accepts one or more numeric arguments. | | `if` | `if(condition, when_true, when_false)` | The condition must be boolean. Both branches must have the same type. Only the selected branch evaluates. | ### Type rules - Component inputs are numeric. - Arithmetic operators require numeric operands and return numeric results. - `<`, `<=`, `>`, `>=` require numeric operands and return boolean results. - `==` and `!=` accept any matching type (both numeric or both boolean) and return boolean results. - `&&`, `||`, and unary `!` require boolean operands. - `&&` and `||` short-circuit. The right-hand side evaluates only when needed. - Local variables must be assigned before use. - Local variable names must start with an ASCII letter or `_` and then use ASCII letters, digits, or `_`. - The final formula result must be numeric. A formula that ends with an assignment or produces a boolean result is invalid for a synthetic instrument. ### Limits The expression engine enforces the following compile-time limits. Formulas that exceed them produce a clear error at construction time. | Limit | Value | Description | | --------------- | ----- | ----------------------------------------------------------------------------------------------- | | Stack depth | 32 | Maximum number of intermediate values on the evaluation stack. | | Local variables | 16 | Maximum number of distinct local variable names. | | Nesting depth | 128 | Maximum syntax nesting and expression tree depth; the top-level expression counts as one level. | A weighted sum of 8 components uses a peak stack depth of 3 and zero locals. A weighted sum of N components builds a tree of depth N + 1, so the nesting depth limit caps a weighted sum at 127 components. ### Examples ```python # Simple spread formula = "BTCUSDT.BINANCE - ETHUSDT.BINANCE" # Average of two FX pairs formula = "(AUD/USD.SIM + NZD/USD.SIM) / 2" # Reuse an intermediate value formula = "spread = BTCUSDT.BINANCE - ETHUSDT.BINANCE; spread / 2" # Conditional output formula = "if(BTCUSDT.BINANCE > ETHUSDT.BINANCE, BTCUSDT.BINANCE, ETHUSDT.BINANCE)" ``` ## Creating a synthetic instrument Make sure all component instruments already exist in the cache, and subscribe to each component's quote or trade feed as well as the synthetic's. Synthetic quotes derive only from component quotes, and synthetic trades only from component trades. When a component tick arrives, the engine combines it with the latest cached prices of the other components to calculate the synthetic price. Until **every component has produced at least one tick**, the synthetic publishes nothing. The following example creates a synthetic instrument with an actor or strategy. This synthetic represents a simple spread between Bitcoin and Ethereum spot prices on Binance. It assumes that `BTCUSDT.BINANCE` and `ETHUSDT.BINANCE` already exist in the cache. ```python from nautilus_trader.model import SyntheticInstrument btcusdt_binance_id = InstrumentId.from_str("BTCUSDT.BINANCE") ethusdt_binance_id = InstrumentId.from_str("ETHUSDT.BINANCE") synthetic = SyntheticInstrument( symbol=Symbol("BTC-ETH:BINANCE"), price_precision=8, components=[ btcusdt_binance_id, ethusdt_binance_id, ], formula=f"{btcusdt_binance_id} - {ethusdt_binance_id}", ts_event=self.clock.timestamp_ns(), ts_init=self.clock.timestamp_ns(), ) self._synthetic_id = synthetic.id self.add_synthetic(synthetic) self.subscribe_quotes(self._synthetic_id) ``` :::note The synthetic `instrument_id` in the example above is `{symbol}.SYNTH`, which produces `BTC-ETH:BINANCE.SYNTH`. ::: ## Updating formulas You can update a synthetic formula at any time. ```python synthetic = self.cache.synthetic(self._synthetic_id) new_formula = "(BTCUSDT.BINANCE + ETHUSDT.BINANCE) / 2" synthetic.change_formula(new_formula) self.update_synthetic(synthetic) ``` ## Trigger instrument IDs You can trigger emulated orders from synthetic prices. In the following example, a synthetic instrument releases an emulated order once the synthetic price reaches the trigger condition. ```python order = self.order_factory.limit( instrument_id=InstrumentId.from_str("ETHUSDT.BINANCE"), order_side=OrderSide.BUY, quantity=Quantity.from_str("1.5"), price=Price.from_str("30000.00000000"), emulation_trigger=TriggerType.DEFAULT, trigger_instrument_id=self._synthetic_id, ) self.submit_order(order) ``` ## Performance Formulas compile once at construction time and evaluate on every incoming component price tick. The expression engine uses a compile-once/eval-many architecture with a zero-allocation f64 stack, so evaluation adds negligible overhead to the tick-processing path. Measured on Apple M4 Pro, rustc 1.94.1, release profile (opt-level 3): ### Evaluation (hot path) | Formula pattern | Time | | --------------------------------------- | ----- | | `(A + B) / 2.0` | 12 ns | | `A * 0.4 + B * 0.3 + C * 0.2 + D * 0.1` | 18 ns | | `if(A > B, A - B, B - A)` | 12 ns | | `spread = A - B; mid = ...; mid + ...` | 19 ns | | `max(min(A, B * 20), abs(A - B))` | 15 ns | ### Evaluation scaling (weighted sum) | Components | Time | | ---------- | ----- | | 2 | 14 ns | | 4 | 18 ns | | 8 | 28 ns | ### Compilation (cold path) | Formula pattern | Time | | ---------------- | ------ | | Simple average | 675 ns | | 4-input weighted | 1.4 us | | Conditional | 1.0 us | | With locals | 1.3 us | | Hyphenated IDs | 755 ns | ## Error handling Nautilus validates synthetic instruments at every boundary. Formula compilation rejects unknown symbols, type errors, and capacity overflows. Evaluation rejects wrong input counts and non-finite prices (NaN, Infinity) before they reach the formula. See the [`SyntheticInstrument` API Reference](/docs/python-api-latest/model/instruments.html#nautilus_trader.model.SyntheticInstrument) for input requirements and exceptions. ## Related guides - [Instruments](instruments/) - Instrument definitions and venue-specific instrument types. - [Data](data/) - Market data types that reference instruments. - [Orders](orders/) - Orders can use synthetic instrument IDs for emulation triggers. # Value Types Source: https://nautilustrader.io/docs/latest/concepts/value_types/ NautilusTrader provides specialized value types for representing core trading concepts: `Price`, `Quantity`, and `Money`. These types use fixed-point arithmetic internally for performant, deterministic calculations across different platforms and environments. ## Overview | Type | Purpose | Signed | Currency | | ---------- | ---------------------------------------- | ------ | -------- | | `Quantity` | Trade sizes, order amounts, positions. | No | - | | `Price` | Market prices, quotes, price levels. | Yes | - | | `Money` | Monetary amounts, P&L, account balances. | Yes | Yes | ## Immutability In Python, all value types are **immutable**. Once a value is constructed, it cannot be changed. Operations do not mutate the original object. ```python from nautilus_trader.model import Quantity qty1 = Quantity(100, precision=0) qty2 = Quantity(50, precision=0) # This creates a NEW Quantity; qty1 and qty2 are unchanged result = qty1 + qty2 print(qty1) # 100 print(qty2) # 50 print(result) # 150 ``` This design provides several benefits: - **Thread safety**: Immutable values can be safely shared across threads without synchronization. - **Predictability**: Values never change unexpectedly, making debugging easier. - **Hashability**: Immutable types can be used as dictionary keys and in sets. ## Arithmetic operations Value types support standard arithmetic operators (`+`, `-`, `*`, `/`, `%`, `//`) and unary operators (`-`, `+`, `abs`). The return type depends on the operator and the operand types. ### Same-type binary operations Addition and subtraction of the same value type return that type, preserving domain meaning (a price plus a price is still a price): | Operation | Result | | --------------------- | ---------- | | `Quantity + Quantity` | `Quantity` | | `Quantity - Quantity` | `Quantity` | | `Price + Price` | `Price` | | `Price - Price` | `Price` | | `Money + Money` | `Money` | | `Money - Money` | `Money` | ```python from nautilus_trader.model import Price price1 = Price(100.50, precision=2) price2 = Price(0.25, precision=2) result = price1 + price2 # Returns Price(100.75, precision=2) print(type(result)) # ``` Multiplication, division, floor division, and modulo between two values of the same type return `Decimal`: | Operation | Result | | ---------------- | --------- | | `Price * Price` | `Decimal` | | `Price / Price` | `Decimal` | | `Price // Price` | `Decimal` | | `Price % Price` | `Decimal` | The same pattern applies to `Quantity` and `Money`. These operations do not return the original type because the result has different dimensional meaning. Multiplying a price by a price produces "price squared", not a price. Dividing a quantity by a quantity produces a dimensionless ratio, not a quantity. Returning `Decimal` makes the unit change explicit and prevents misinterpretation of the result as a value with the original unit. ### Unary operations Unary operators preserve the value type where the result is valid for that type: | Operation | `Price` | `Quantity` | `Money` | | ---------- | --------- | ---------- | --------- | | `-x` (neg) | `Price` | `Decimal` | `Money` | | `+x` (pos) | `Price` | `Quantity` | `Money` | | `abs(x)` | `Price` | `Quantity` | `Money` | | `int(x)` | `int` | `int` | `int` | | `float(x)` | `float` | `float` | `float` | | `round(x)` | `Decimal` | `Decimal` | `Decimal` | `Quantity.__neg__` returns `Decimal` rather than `Quantity` because `Quantity` is unsigned and cannot represent a negative value. ```python from nautilus_trader.model import Currency, Money, Price, Quantity USD = Currency.from_str("USD") price = Price(100.50, precision=2) print(-price) # -100.50 print(type(-price)) # money = Money(-50.00, USD) print(abs(money)) # 50.00 USD print(type(abs(money))) # qty = Quantity(10, precision=0) print(+qty) # 10 print(type(+qty)) # ``` ### Mixed-type operations When operating with other numeric types, the result type follows Python's [numeric tower](https://docs.python.org/3/library/numbers.html) conventions. The general principle is that operations widen to the more general type: `float` operations return `float`, while `int` and `Decimal` operations return `Decimal` for precision preservation. This applies to all six binary operators (`+`, `-`, `*`, `/`, `//`, `%`) and works in both directions (`value op scalar` and `scalar op value`): | Left operand | Right operand | Result type | | ------------ | ------------- | ----------- | | Value type | `int` | `Decimal` | | Value type | `float` | `float` | | Value type | `Decimal` | `Decimal` | | `int` | Value type | `Decimal` | | `float` | Value type | `float` | | `Decimal` | Value type | `Decimal` | ```python from decimal import Decimal from nautilus_trader.model import Quantity qty = Quantity(100, precision=0) # Quantity + int -> Decimal result1 = qty + 50 print(type(result1)) # # Quantity + float -> float result2 = qty + 50.5 print(type(result2)) # # Quantity + Decimal -> Decimal result3 = qty + Decimal("50") print(type(result3)) # ``` ## Precision handling Each value type carries a precision indicating the number of decimal places: `Price` and `Quantity` store an explicit `precision` field, while `Money` uses its currency's precision. Precision is set at construction and is immutable. There is no "unspecified" precision. ### Fixed-point representation Value types are stored internally as integers scaled to a global fixed precision (e.g., 10^16 in high-precision mode), not floating-point numbers. The precision tracks the number of decimal places used at construction, controlling display formatting and serialization, but the underlying raw value always uses the global scale. ```python from nautilus_trader.model import Price p1 = Price(1.23, precision=2) # displays as "1.23" p2 = Price(1.230, precision=3) # displays as "1.230" p1 == p2 # True: same underlying value str(p1) # "1.23" str(p2) # "1.230" ``` **Precision controls display, not identity.** Two prices with the same decimal value but different precisions are equal. The `precision` field determines string formatting and how many decimal places are shown, but equality is based on the underlying numeric value. **Market data serialization uses precision metadata.** When market data types (quotes, trades, order book deltas) are written to Parquet or Arrow format, precision is stored in the file metadata so that values can be correctly decoded. All market data values within a single file must share the same precision. :::warning If a venue changes an instrument's tick size (and thus its precision), data files written before and after the change will have different precision metadata and should not be consolidated into a single file. ::: For how instrument-level precision constrains valid prices and quantities, see the [Precision](instruments/index.md#precision) section of the Instruments guide. ### Arithmetic precision When performing arithmetic between values with different precisions, the result uses the maximum precision of the operands. ```python from nautilus_trader.model import Price price1 = Price(100.5, precision=1) # 1 decimal place price2 = Price(0.125, precision=3) # 3 decimal places result = price1 + price2 print(result) # 100.625 print(result.precision) # 3 (max of 1 and 3) ``` ## Type-specific constraints ### Quantity `Quantity` represents non-negative amounts. Attempting to create a negative quantity or subtract a larger quantity from a smaller one raises an error: ```python from nautilus_trader.model import Quantity # This raises ValueError: Quantity cannot be negative qty = Quantity(-100, precision=0) # This also raises ValueError qty1 = Quantity(50, precision=0) qty2 = Quantity(100, precision=0) result = qty1 - qty2 # Would be -50, which is invalid ``` ### Money `Money` values include a currency. Addition and subtraction between `Money` values require **matching currencies**: ```python from nautilus_trader.model import Currency, Money USD = Currency.from_str("USD") EUR = Currency.from_str("EUR") usd_amount = Money(100.00, USD) eur_amount = Money(50.00, EUR) # This works - same currency result = usd_amount + Money(25.00, USD) # This raises ValueError - currency mismatch result = usd_amount + eur_amount ``` ## Common patterns ### Accumulating values Since value types are immutable, accumulate by reassigning: ```python from nautilus_trader.model import Currency, Money USD = Currency.from_str("USD") total = Money(0.00, USD) amounts = [Money(100.00, USD), Money(50.00, USD), Money(25.00, USD)] for amount in amounts: total = total + amount # Reassign to new Money instance print(total) # 175.00 USD ``` ### Converting to other types Value types provide conversion methods: ```python from nautilus_trader.model import Price price = Price(123.456, precision=3) # Convert to Decimal (preserves precision) decimal_value = price.as_decimal() # Convert to float float_value = price.as_double() # Convert to string string_value = str(price) # "123.456" ``` ### Creating from strings Parse value types from string representations: ```python from nautilus_trader.model import Money, Price, Quantity qty = Quantity.from_str("100.5") price = Price.from_str("99.95") money = Money.from_str("1000.00 USD") ``` # Visualization Source: https://nautilustrader.io/docs/latest/concepts/visualization/ NautilusTrader provides interactive HTML tearsheets for analyzing backtest results through an extensible visualization system built on Plotly. You can generate reports with minimal code and add custom charts and themes. ## Overview The visualization system has three parts: 1. **Chart Registry** - Decoupled chart definitions that can be extended with custom visualizations. 2. **Theme System** - Consistent styling with built-in and custom themes. 3. **Configuration** - Declarative specification of what to render and how to display it. Tearsheets are written as **self-contained HTML files** that can be viewed in any modern browser, shared with stakeholders, or archived for future reference. Passing a static image extension (such as `.png` or `.pdf`) as the output path exports a static image via Kaleido instead. :::note The visualization system requires the `visualization` extra. It installs Pandas for DataFrame handling, Plotly for interactive figures, and Kaleido for static image export: ```bash uv pip install --pre "nautilus_trader[visualization]" ``` ::: ## Tearsheets A tearsheet is a performance report that combines multiple charts and statistics into a single interactive visualization. Tearsheets are generated after completing a backtest run and provide immediate visual feedback on strategy performance. ### Quick start Generate a tearsheet with default settings: ```python from nautilus_trader.analysis import create_tearsheet from nautilus_trader.backtest import BacktestEngine # After running your backtest engine.run() # Generate tearsheet create_tearsheet( engine=engine, output_path="backtest_results.html", ) ``` This produces an HTML file with all default charts, using the light theme and automatic layout. Open `backtest_results.html` in your browser to view the interactive tearsheet. ### Backtest result input Let `result` be a `BacktestResult` returned by a completed backtest. Pass it without its node for a result-only tearsheet: ```python create_tearsheet( engine=result, output_path="backtest_results.html", ) ``` To include starting account balances from node reports, retain the node state. The node is also required when the configured tearsheet includes a cache-backed chart such as `bars_with_fills`. Follow the complete [`BacktestNode` setup](backtesting/apis-and-runs.md#high-level-api), setting `dispose_on_completion=False` on its `BacktestRunConfig`. Then pass the completed result and retained node: ```python create_tearsheet( engine=result, node=node, output_path="backtest_results.html", ) ``` :::warning Passing a node whose matching run configuration enables disposal raises `ValueError` because its cache and reports are no longer available. ::: ### Customization Control which charts appear and how they're styled: ```python from nautilus_trader.config import TearsheetConfig from nautilus_trader.analysis import TearsheetDrawdownChart from nautilus_trader.analysis import TearsheetEquityChart from nautilus_trader.analysis import TearsheetRunInfoChart from nautilus_trader.analysis import TearsheetStatsTableChart config = TearsheetConfig( charts=[ TearsheetRunInfoChart(), TearsheetStatsTableChart(), TearsheetEquityChart(), TearsheetDrawdownChart(), ], theme="nautilus_dark", height=2000, ) create_tearsheet( engine=engine, output_path="custom_tearsheet.html", config=config, ) ``` ### Currency filtering For multi-currency backtests, filter statistics to a specific currency: ```python from nautilus_trader.model import Currency create_tearsheet( engine=engine, output_path="usd_only.html", currency=Currency.from_str("USD"), # Shows only USD statistics ) ``` When `currency` is `None` (default), statistics for all currencies are displayed separately in the tearsheet. For `BacktestEngine` input, return-based charts require a single currency: they are derived from portfolio equity snapshots, falling back to account reports, and cannot be built for mixed-currency accounts without a filter; pass `currency` for multi-currency backtests so return charts use the selected currency. For `BacktestResult` input, `currency` filters PnL statistics and account balances. The result's **stored return series remains unchanged**. ## Available charts The tearsheet can include any combination of the following built-in charts: | Chart Name | Type | Description | | ----------------- | ----------- | ------------------------------------------------------- | | `run_info` | Table | Run metadata and account balances. | | `stats_table` | Table | Performance statistics (PnL, returns, general metrics). | | `equity` | Line | Cumulative returns over time with optional benchmark. | | `drawdown` | Area | Drawdown percentage from peak equity. | | `monthly_returns` | Heatmap | Monthly portfolio return percentages organized by year. | | `distribution` | Histogram | Distribution of individual return values. | | `rolling_sharpe` | Line | 60-day rolling Sharpe ratio. | | `yearly_returns` | Bar | Annual return percentages. | | `bars_with_fills` | Candlestick | Price bars (OHLC) with order fills overlaid as markers. | All charts are registered in the chart registry and are configured via chart objects in `TearsheetConfig.charts` (each chart object maps to a built-in chart name). ### Run information table The `run_info` chart displays key metadata about the backtest run: - Run ID, start time, finish time - Backtest period (start/end dates) - Total iterations processed - Event, order, and position counts - Account starting and ending balances (per currency) This table appears in the top-left position by default. ### Performance statistics table The `stats_table` chart displays performance metrics organized into sections: - **PnL Statistics** (per currency): Total PnL, win rate, profit factor, etc. - **Returns Statistics**: Sharpe ratio, Sortino ratio, max drawdown, etc. - **General Statistics**: Total trades, average trade duration, etc. This table appears in the top-right position by default. ### Equity curve The `equity` chart plots cumulative returns over the backtest period. When `benchmark_returns` is provided to `create_tearsheet()`, the benchmark is overlaid for comparison. ```python import pandas as pd # Load benchmark returns (e.g., from a market index) # Index should be datetime, aligned with strategy returns timeframe benchmark_returns = pd.read_csv("sp500_returns.csv", index_col=0, parse_dates=True)["return"] create_tearsheet( engine=engine, output_path="with_benchmark.html", benchmark_returns=benchmark_returns, benchmark_name="S&P 500", ) ``` The benchmark series is plotted as-is; ensure the index aligns with your strategy's return dates for accurate comparison. ### Monthly and yearly returns The `monthly_returns` and `yearly_returns` charts default to compounded (time-weighted) returns: each cell measures the period's gain against the running start-of-period balance, and the periods compound to the total return. Set `compounding=False` to report simple, non-compounding returns measured against fixed initial capital. Each cell then measures the period's gain as a percentage of the starting capital, so the periods sum to the total return instead of compounding to it. This is the nominal rate of return, the convention used for constant-capital strategies that trade fixed size and withdraw profits. ```python config = TearsheetConfig( charts=[ TearsheetMonthlyReturnsChart(compounding=False), TearsheetYearlyReturnsChart(compounding=False), ], ) create_tearsheet(engine=engine, config=config) ``` The standalone `create_monthly_returns_heatmap()` and `create_yearly_returns()` functions accept the same `compounding` argument. For the non-compounding figures to faithfully represent constant capital, size positions at a fixed quantity rather than as a fraction of current equity; otherwise later periods inflate as the running balance grows. ## Themes Themes control the visual styling of charts including colors, fonts, and backgrounds. NautilusTrader provides four built-in themes: | Theme Name | Description | Use Case | | --------------- | --------------------------------------------- | ------------------------------ | | `plotly_white` | Clean light theme with dark gray headers. | Default, professional reports. | | `plotly_dark` | Dark background with standard Plotly colors. | Low-light environments. | | `nautilus` | Light theme with NautilusTrader brand colors. | Official light mode. | | `nautilus_dark` | Dark theme with teal/cyan signature colors. | Official dark mode. | ### Selecting a theme Specify the theme in `TearsheetConfig`: ```python config = TearsheetConfig(theme="nautilus_dark") create_tearsheet(engine=engine, config=config) ``` ### Custom themes Register a custom theme for consistent branding across all visualizations: ```python from nautilus_trader.analysis import register_theme register_theme( name="corporate", template="plotly_white", # Base Plotly template colors={ "primary": "#003366", # Navy blue "positive": "#2e8b57", # Sea green "negative": "#c41e3a", # Cardinal red "neutral": "#808080", # Gray "background": "#ffffff", # White "grid": "#e5e5e5", # Light gray # Optional table colors (defaults will be provided if omitted) "table_section": "#e5e5e5", "table_row_odd": "#f8f8f8", "table_row_even": "#ffffff", "table_text": "#000000", }, ) # Use the custom theme config = TearsheetConfig(theme="corporate") ``` The theme system automatically provides sensible defaults for `table_*` colors based on the `background` and `grid` colors, ensuring backward compatibility with themes registered before table-specific colors were introduced. ## Configuration The `TearsheetConfig` class provides declarative control over tearsheet generation: ```python from nautilus_trader.analysis import GridLayout from nautilus_trader.config import TearsheetConfig from nautilus_trader.analysis import TearsheetDrawdownChart from nautilus_trader.analysis import TearsheetEquityChart from nautilus_trader.analysis import TearsheetStatsTableChart config = TearsheetConfig( charts=[ TearsheetEquityChart(), TearsheetDrawdownChart(), TearsheetStatsTableChart(), ], theme="nautilus_dark", title="Q4 2024 Strategy Performance", height=1800, include_benchmark=True, benchmark_name="SPY", layout=GridLayout( rows=2, cols=2, heights=[0.60, 0.40], vertical_spacing=0.08, horizontal_spacing=0.12, ), ) ``` ### Configuration parameters | Parameter | Type | Default | Description | | ------------------- | ---------------------- | ---------------- | ----------------------------------- | | `charts` | `list[TearsheetChart]` | Built-ins | Charts to include, in order. | | `theme` | `str` | `"plotly_white"` | Theme name for styling. | | `layout` | `GridLayout` | `None` | Custom subplot grid layout. | | `title` | `str` | Auto-generated | Tearsheet title. | | `include_benchmark` | `bool` | `True` | Show benchmark when provided. | | `benchmark_name` | `str` | `"Benchmark"` | Display name for benchmark. | | `height` | `int` | `1500` | Total height in pixels. | | `show_logo` | `bool` | `True` | Reserved for future logo rendering. | When `layout` is `None`, the grid dimensions and row heights are automatically calculated based on the number of charts. For 8 charts (the default), a 4x2 grid is used with heights `[0.50, 0.22, 0.16, 0.12]` to give more space to the top row tables. ## Custom charts The registry pattern lets you add custom charts. Charts are functions that render traces onto a Plotly figure object. ### Registering a custom chart ```python from nautilus_trader.analysis import register_chart import plotly.graph_objects as go def my_custom_chart(returns, output_path=None, title="Custom Chart", theme="plotly_white"): """ Create a custom visualization. This function signature matches the built-in chart functions for consistency. """ from nautilus_trader.analysis import get_theme theme_config = get_theme(theme) # Create your visualization fig = go.Figure() fig.add_trace( go.Scatter( x=returns.index, y=returns.cumsum(), mode="lines", name="Custom Metric", line={"color": theme_config["colors"]["primary"]}, ) ) fig.update_layout( title=title, template=theme_config["template"], xaxis_title="Date", yaxis_title="Value", ) if output_path: fig.write_html(output_path) return fig # Register the chart for standalone use (via `get_chart()` / `list_charts()`) register_chart("my_custom", my_custom_chart) ``` ### Tearsheet integration For tearsheet integration with proper grid placement, use `register_tearsheet_chart`. Unlike `register_chart` (which registers a standalone function that returns its own figure), a tearsheet renderer draws traces directly onto a shared subplot grid cell, so its signature takes the target `fig` plus the `row` and `col` to render into. ```python from nautilus_trader.config import TearsheetConfig from nautilus_trader.analysis import TearsheetCustomChart from nautilus_trader.analysis import TearsheetEquityChart from nautilus_trader.analysis import TearsheetStatsTableChart from nautilus_trader.analysis import register_tearsheet_chart def _render_my_metric(fig, row, col, returns, theme_config, **kwargs): """ Render custom metric directly onto a subplot. Parameters ---------- fig : go.Figure The figure to add traces to. row : int Subplot row position. col : int Subplot column position. returns : pd.Series Strategy returns series supplied to the renderer. theme_config : dict Theme configuration dictionary. **kwargs : dict Additional parameters (stats_pnls, stats_returns, benchmark_returns, etc.). """ metric_values = returns.rolling(30).std() * 100 # Example metric fig.add_trace( go.Scatter( x=returns.index, y=metric_values, mode="lines", name="30-Day Volatility", line={"color": theme_config["colors"]["neutral"]}, ), row=row, col=col, ) fig.update_xaxes(title_text="Date", row=row, col=col) fig.update_yaxes(title_text="Volatility (%)", row=row, col=col) # Register for tearsheet use register_tearsheet_chart( name="volatility", subplot_type="scatter", title="Rolling Volatility (30-day)", renderer=_render_my_metric, ) # Now "volatility" can be used in TearsheetConfig.charts: config = TearsheetConfig( charts=[ TearsheetStatsTableChart(), TearsheetEquityChart(), TearsheetCustomChart(chart="volatility"), ], ) ``` The renderer function receives all necessary data (returns, statistics, theme configuration) and renders directly onto the specified subplot position. ## Offline analysis For situations where you have precomputed statistics but not a `BacktestEngine` instance, use the lower-level API: ```python import pandas as pd from nautilus_trader.analysis import create_tearsheet_from_stats # Load precomputed data. The structure matches BacktestResult stats fields. stats_pnls = {"USD": {"PnL (total)": 1500.0, "Win Rate": 0.55, ...}} # Per-currency stats_returns = {"Sharpe Ratio (252 days)": 1.2, "Max Drawdown": -0.15, ...} stats_general = {"Avg Winner": 100.0, "Avg Loser": -50.0, ...} returns = pd.Series(...) # Daily returns with datetime index create_tearsheet_from_stats( stats_pnls=stats_pnls, stats_returns=stats_returns, stats_general=stats_general, returns=returns, output_path="offline_analysis.html", ) ``` The dictionary keys should match those returned by `engine.get_result().stats_pnls`, `engine.get_result().stats_returns`, and `engine.get_result().stats_general`. This approach is useful for: - Analyzing results from multiple backtest runs stored separately. - Comparing strategies using precomputed metrics. - Integrating with external analysis pipelines. ## Best practices ### Chart selection - Use default charts for exploratory analysis to see all available metrics. - Customize charts when you know which metrics matter for your strategy. - Remove irrelevant charts to reduce visual clutter and file size. ### Theme usage - Use `plotly_white` for professional reports and presentations. - Use `nautilus_dark` for official materials or low-light viewing. - Create custom themes to match internal guidelines or personal preferences. ### Performance considerations - Tearsheet HTML files contain all data inline and can be several megabytes for long backtests. - Consider generating separate tearsheets for different analysis timeframes. - For very large datasets, use the individual chart functions instead of full tearsheets. ### Custom statistics integration Custom charts work best when paired with statistics supplied through the same `stats_pnls`, `stats_returns`, and `stats_general` dictionaries used by the built-in tearsheet charts. For `BacktestEngine` input these values come from `engine.get_result()`, so a statistic registered with `Portfolio.register_statistic()` reaches the tearsheet without any extra wiring; see [Custom statistics](portfolio.md#custom-statistics). For offline analysis, pass compatible dictionaries directly to `create_tearsheet_from_stats()`: ```python stats_returns = { "Sharpe Ratio (252 days)": 1.2, "Custom Volatility Score": 0.42, } ``` ## API levels The visualization system provides two API levels, plus standalone chart functions: ### High-level API Recommended for most use cases: ```python create_tearsheet(engine=engine, config=config) ``` Automatically extracts data from a `BacktestEngine` or `BacktestResult`, generates all configured charts, and produces a complete HTML tearsheet. ### Low-level API For advanced customization or offline analysis: ```python create_tearsheet_from_stats( stats_pnls=stats_pnls, stats_returns=stats_returns, stats_general=stats_general, returns=returns, run_info=run_info, account_info=account_info, config=config, ) ``` Provides fine-grained control over data inputs and allows analysis of precomputed statistics. ### Standalone chart functions Individual chart functions can be used independently to generate single-purpose HTML visualizations or Plotly figures for custom analysis workflows. #### Price bars with fills The `create_bars_with_fills` function generates a candlestick chart with order fills overlaid, useful for visually analyzing strategy execution within price action. It can be used standalone or included in tearsheets: ```python from nautilus_trader.analysis import create_bars_with_fills from nautilus_trader.analysis import create_tearsheet from nautilus_trader.analysis import TearsheetBarsWithFillsChart from nautilus_trader.config import TearsheetConfig from nautilus_trader.analysis import TearsheetEquityChart from nautilus_trader.analysis import TearsheetStatsTableChart from nautilus_trader.model import BarType # Standalone usage bar_type = BarType.from_str("ESM4.XCME-1-MINUTE-LAST-EXTERNAL") fig = create_bars_with_fills( engine=engine, bar_type=bar_type, title="ES Futures - Entry/Exit Analysis", ) fig.show() # Display in Jupyter fig.write_html("bars_with_fills.html") # Or save to file # Include in tearsheet config = TearsheetConfig( charts=[ TearsheetStatsTableChart(), TearsheetEquityChart(), TearsheetBarsWithFillsChart( bar_type="ESM4.XCME-1-MINUTE-LAST-EXTERNAL", title="Bars with Fills", ), ], ) create_tearsheet(engine=engine, config=config) # Multiple bars-with-fills charts in one tearsheet config = TearsheetConfig( charts=[ TearsheetStatsTableChart(), TearsheetEquityChart(), TearsheetBarsWithFillsChart( bar_type=f"{instrument.id}-5-MINUTE-MID-INTERNAL", title=f"Bars with Order Fills - {instrument.id}", ), TearsheetBarsWithFillsChart( bar_type=f"{other_instrument.id}-5-MINUTE-MID-INTERNAL", title=f"Bars with Order Fills - {other_instrument.id}", ), ], ) create_tearsheet(engine=engine, config=config) ``` The visualization shows candlesticks for OHLC price action with triangle markers representing order fills (up-triangles for buys and down-triangles for sells, colored with the theme's positive and negative colors). Charts that need extra configuration (like `bar_type`) take those parameters directly on the chart object (e.g. `TearsheetBarsWithFillsChart(bar_type=...)`). Other individual chart functions include `create_equity_curve`, `create_drawdown_chart`, `create_monthly_returns_heatmap`, and more. See the API reference for the complete list. ## Related guides - [Backtesting](backtesting/) - Learn how to run backtests that generate tearsheets. - [Reports](reports.md) - Understand the underlying statistics displayed in tearsheets. - [Portfolio](portfolio.md) - Explore portfolio tracking and performance metrics. # Adapters Source: https://nautilustrader.io/docs/latest/developer_guide/adapters/ ## Introduction Use this guide to build or extend a Rust-native adapter for NautilusTrader. Adapters connect the platform to venues and data providers, preserve venue semantics, produce valid Nautilus domain events, and make uncertain outcomes explicit. They implement the platform data and execution client traits in Rust, then expose configs, factories, and selected low-level APIs to Python through PyO3. :::note For out-of-tree adapters implemented in Python or an independent Rust/PyO3 package, use the [Python adapter interface](python_adapters.md). This guide covers in-tree Rust adapters. ::: Use reference adapters selectively. Their layouts reflect different venue protocols, product families, and implementation histories. | Adapter | Useful reference | | ------------------ | ------------------------------------------------------------------------------------------------------ | | [Bybit][bybit] | Multi-product HTTP and WebSocket clients, options data, and execution outcome handling. | | [OKX][okx] | Public, private, and business WebSocket endpoints with broad instrument coverage. | | [Binance][binance] | Spot and futures product splits, trading WebSockets, and SBE market data. | | [Kraken][kraken] | Spot and futures submodules with distinct HTTP, WebSocket, data, and execution paths. | | [Lighter][lighter] | Layer-2 signing, canonical benchmarks, coverage-guided fuzzing, and detailed execution state handling. | | [Derive][derive] | JSON-RPC data and execution, EIP-712 signing, canonical benchmarks, and invariant-based fuzzing. | This guide distinguishes four kinds of guidance: - **Shared rules** come from common traits, network abstractions, hooks, or CI. - **Common patterns** appear in several adapters but allow other designs. - **Examples** show one sound implementation without making it mandatory. - **Exceptions** are valid when venue semantics or protocol boundaries require them. ## Conformance An adapter conforms when it satisfies each rule below that applies to it, or documents an exception. Name the venue behavior that forces the exception, keep it inside the adapter, and cover it with a test that fails if the venue stops requiring it. [Phase 7](#phase-7-prove-conformance) sequences the work that proves conformance. ### Adapter foundations | Rule | Applies to | | ------------------------------------------------------------------------------------------ | ------------------ | | [Repository and Python wiring](#repository-and-python-wiring) | New adapter crates | | [Credentials and secret handling](#credentials-and-secret-handling) | Every adapter | | [Configurations](#configurations-configrs) | Every adapter | | [Symbols and instrument identity](#symbols-and-instrument-identity) | Every adapter | | [Venue payload modeling and precision](#modeling-venue-payloads) | Every adapter | | [Client traits and factories](#client-traits-and-factories-datars-executionrs-factoriesrs) | Every adapter | ### Runtime and client lifecycle | Rule | Applies to | | ----------------------------------------------------- | -------------------------- | | [Connection lifecycle](#connection-lifecycle-connect) | Data and execution clients | | [Data events and request freshness](#data-client) | Data clients | | [Backpressure](#backpressure) | Every adapter | | [Task management](#task-management) | Every adapter | ### Execution and reconciliation | Rule | Applies to | | --------------------------------------------------------------------------------------------- | ----------------- | | [Execution client boundaries](#execution-client) | Execution clients | | [Reconciliation reports](#reconciliation-reports) | Execution clients | | [Commission failure handling](#commission-failure-handling) | Execution clients | | [Bounded mass-status reports](#bounded-mass-status-reports) | Execution clients | | [Instrument resolution during reconciliation](#instrument-resolution-during-reconciliation) | Execution clients | | [Tracked and external execution updates](#tracked-and-external-execution-updates) | Execution clients | | [Event ordering and deduplication](#event-ordering-and-deduplication) | Execution clients | | [Order command outcome policy](#order-command-outcome-policy) | Execution clients | | [Naming the evidence classes](#naming-the-evidence-classes) | Execution clients | | [Diagnostics and strategy-facing reasons](#separate-diagnostics-from-strategy-facing-reasons) | Execution clients | ### Transport and streaming | Rule | Applies to | | ------------------------------------------------------------------------------- | -------------------------------- | | [Request flow](#request-flow) | HTTP clients | | [Request signing and authentication](#request-signing-and-authentication) | HTTP and WebSocket request paths | | [Error handling and retry logic](#error-handling-and-retry-logic) | HTTP and WebSocket request paths | | [Rate limiting](#rate-limiting) | HTTP and WebSocket clients | | [Handler initialization handshake](#handler-initialization-handshake-setclient) | WebSocket clients | | [Authentication](#authentication) | WebSocket clients | | [Subscription management](#subscription-management) | WebSocket clients | | [Message routing](#message-routing) | WebSocket clients | | [Reconnection and shutdown](#reconnection-and-shutdown) | WebSocket clients | The [data testing specification](spec_data_testing.md) and [execution testing specification](spec_exec_testing.md) hold the scenarios that prove these contracts against a venue. ### Shared baseline Use the shared implementation of each piece below, then use any state structure that satisfies the contract it implements. The shared type carries that contract with it and keeps behavior comparable across venues, so a local structure has to prove the same contract on its own terms. Two execution clients implement the same trait without trading through a venue API, so the baseline does not apply to them: [sandbox](../../crates/adapters/sandbox/src/execution.rs) simulates fills locally, and [blockchain](../../crates/adapters/blockchain/src/execution/client.rs) executes on-chain behind the `defi` feature. Deterministic simulation eligibility also sits outside the baseline, as an optional capability proven per adapter rather than a requirement. | Target | Shared piece | Contract | | -------------------------- | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- | | Command outcome evidence | [`CommandFailure`](../../crates/live/src/execution/failure.rs) | [Naming the evidence classes](#naming-the-evidence-classes) | | Order identity and context | [`OrderIdentity` and `OrderContext`](../../crates/live/src/execution/context.rs) | [Tracked and external updates](#tracked-and-external-execution-updates) | | Replay deduplication | [`FifoCache` and `FifoCacheMap`](../../crates/common/src/cache/fifo.rs) | [Event ordering and deduplication](#event-ordering-and-deduplication) | | Order denial reasons | [`OrderDeniedReason`](../../crates/model/src/events/order/denied_reason.rs) | [Diagnostics and reasons](#separate-diagnostics-from-strategy-facing-reasons) | | Task lifecycle | [`TaskGroup`](../../crates/live/src/task.rs) and [`TaskHandles`](../../crates/common/src/live/task.rs) | [Task management](#task-management) | | Ingestion precision | [Domain numeric types](rust.md#domain-numeric-types) | [Venue payload modeling](#modeling-venue-payloads) | | HTTP transport | [`HttpClient`](../../crates/network/src/http/client.rs) | [Request flow](#request-flow) | | Authentication state | [`AuthTracker`](../../crates/network/src/websocket/auth.rs) | [Authentication](#authentication) | | Subscription identity | [`SubscriptionState`](../../crates/network/src/websocket/subscription.rs) | [Subscription management](#subscription-management) | | Reconnect requests | [`request_reconnect`](../../crates/network/src/websocket/client.rs) | [Reconnection and shutdown](#reconnection-and-shutdown) | | Retry machinery | [`RetryManager`](../../crates/network/src/retry.rs) | [Error handling and retry logic](#error-handling-and-retry-logic) | | Inferred fill commission | [`ExecutionClient`](../../crates/common/src/clients/execution.rs) | [Commission failure handling](#commission-failure-handling) | Where a venue transmits a discrete value as an IEEE-754 field rather than a decimal string or JSON number, contain that at the parsing boundary as a documented exception instead of letting `f64` spread inward from it. Retry classification is the exception to this table: it stays adapter-owned because venue status codes and rate-limit semantics differ. The shared machinery around it is not. See [error handling and retry logic](#error-handling-and-retry-logic) for both halves. ## Structure of an adapter The Rust crate is the source of truth for protocol behavior. An adapter commonly separates these concerns: ```text crates/adapters// ├── Cargo.toml ├── src/ │ ├── common/ # Shared credentials, enums, models, parsing, symbols, and URLs │ ├── http/ # Typed requests, responses, signing hooks, and transport client │ │ ├── client.rs │ │ ├── error.rs │ │ ├── models.rs │ │ ├── parse.rs │ │ └── query.rs │ ├── websocket/ # Streaming transport, protocol messages, parsing, and routing │ │ ├── client.rs │ │ ├── handler.rs │ │ ├── messages.rs │ │ ├── parse.rs │ │ ├── subscription.rs # When subscription identity or replay needs a boundary │ │ └── dispatch.rs # When execution routing needs a boundary │ ├── config.rs │ ├── data.rs # Or data/ when product implementations need a split │ ├── execution.rs # Or execution/ when product implementations need a split │ ├── factories.rs │ ├── python/ # PyO3 projection │ ├── signing/ # When authentication or transaction signing is a subsystem │ └── lib.rs ├── tests/ # Public Rust boundary tests ├── test_data/ # Canonical venue payloads and protocol vectors ├── benches/ # When confirmed hot paths warrant benchmarks │ ├── common/ # Shared benchmark fixtures │ ├── data.rs │ ├── exec.rs │ └── micros.rs ├── fuzz/ # When untrusted codecs warrant coverage-guided fuzzing │ ├── fuzz_targets/ │ └── README.md ├── examples/ # Rust tester nodes and focused usage examples ├── bin/ # Optional protocol inspection or capture tools └── README.md ``` Python and documentation surfaces sit outside the crate: ```text python/nautilus_trader/adapters// # Public package and generated stubs examples/live// # Python data and execution testers python/tests/unit/adapters// # Public Python package tests docs/integrations/.md # User-facing integration guide ``` Only `Cargo.toml` and `src/lib.rs` are universal crate boundaries. Add the other modules when the adapter needs them: - Put symbols, credentials, URLs, shared enums, and shared parsing under `common/`. - Put transport models, typed requests, signing, and HTTP clients under `http/`. - Put frames, messages, subscription state, routing, and WebSocket clients under `websocket/`. - Implement live data and execution traits in `data.rs` and `execution.rs`, or in product submodules when the venue exposes materially different protocols. - Keep PyO3 projection code under `python/`. - Organize integration tests by public boundary or product. Do not force all adapters into the same filenames. Product-specific splits are legitimate when product families have different protocols. A shared client can also span distinct endpoints when request and state semantics remain common. Match the venue's real boundaries and keep shared behavior above those splits. An adapter's public Python package lives under `python/nautilus_trader/adapters//` and usually re-exports generated bindings. Change Rust binding metadata or other generator inputs, then run `make py-stubs`; do not edit generated `.pyi` files. ### Repository and Python wiring A new adapter crate must be discoverable by each build surface that owns it: | Surface | Required change | Enforcement or proof | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Root Rust workspace | Add the crate to the members and workspace dependencies in [`Cargo.toml`](../../Cargo.toml). | Workspace metadata and targeted Cargo checks discover the crate. | | Workspace test inventory | Add the crate to `ADAPTER_CRATES` in the [`Makefile`](../../Makefile). | The [workspace coverage check](../../scripts/ci/check-workspace-test-coverage.sh) requires one test inventory. | | PyO3 crate | Add the optional dependency and feature propagation in [`crates/pyo3/Cargo.toml`](../../crates/pyo3/Cargo.toml). | Building the matching PyO3 feature compiles the adapter projection. | | PyO3 root module | Register the adapter module in [`crates/pyo3/src/lib.rs`](../../crates/pyo3/src/lib.rs). | The conventions hook treats this module list as the public API allowlist. | | Adapter PyO3 registry | Register each applicable factory and config extractor with `get_global_pyo3_registry()`. | Factory boundary tests prove Python config objects reach the Rust factories. | | Python package and user guide | Add package projection, tests, examples, and an integration guide only for capabilities the adapter exposes. | Import, generated drift, example build, and documentation checks cover these surfaces. | The [Nautilus conventions hook](../../.pre-commit-hooks/check_nautilus_conventions.sh) treats the PyO3 module list as a public API allowlist. The [PyO3 conventions hook](../../.pre-commit-hooks/check_pyo3_conventions.sh) also enforces: - Stub metadata uses `nautilus_trader.adapters.`. - Runtime extension imports use `nautilus_trader._libnautilus.`. - A Rust function renamed with `#[pyo3(name = ...)]` has a `py_` Rust name. - Python exceptions use the project error conversion functions. ## Adapter implementation sequence Use these phases to organize the work. They describe dependencies, not release gates. A market-data-only adapter omits execution, and an adapter can complete one product before starting another. Keep the capability matrix current throughout the work rather than waiting for the final documentation phase. Omit phases and steps that do not apply to the adapter. ### Phase 0: Define scope | Step | Component | Work | | ---- | ------------------- | --------------------------------------------------------------------------------------------------------- | | 0.1 | Capability matrix | List the products, environments, account modes, data types, order types, and reports in scope. | | 0.2 | Venue constraints | Record venue restrictions, unsupported capabilities, and testnet differences. | | 0.3 | Protocol boundaries | Identify separate product APIs, public and private endpoints, and binary or JSON transports. | | 0.4 | Initial slice | Choose the smallest slice that proves an end-to-end path. | | 0.5 | Repository wiring | Add the crate to the Rust workspace and test inventory, then add only the projection surfaces it exposes. | **Exit:** The integration guide contains an initial capability matrix, known gaps, and a test plan. ### Phase 1: Build the protocol core | Step | Component | Work | | ---- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | 1.1 | HTTP error types | Model transport, HTTP status, venue, parsing, and validation failures; classify retryability when supported. | | 1.2 | HTTP client | Implement endpoint resolution and typed requests, plus credentials, signing, rate limits, retries, and pagination as needed. | | 1.3 | HTTP API models | Define typed requests and responses, commonly under `http/` or its product-specific modules. | | 1.4 | HTTP parsing | Convert venue responses to domain types at deterministic boundaries in `http/parse.rs` or `common/parse.rs`. | | 1.5 | WebSocket error types | Model connection, protocol, and parsing failures, plus authentication and command failures when applicable. | | 1.6 | WebSocket client | Implement lifecycle and shutdown, plus authentication, heartbeat, subscription state, and reconnection when applicable. | | 1.7 | WebSocket messages | Define frames and messages under `websocket/` or product-specific modules; include acknowledgements and venue errors as needed. | | 1.8 | WebSocket parsing | Decode each frame once, convert domain events, and route data or execution messages by typed identity. | | 1.9 | Protocol tests | Prove fixtures, canonical requests, applicable signing vectors, lifecycle, and raw exchanges with mock peers. | **Exit:** The crate compiles, protocol fixtures parse, applicable signing vectors pass, and mock or controlled requests complete any required authentication and exchange raw venue messages. ### Phase 2: Implement instruments | Step | Component | Work | | ---- | ------------------ | -------------------------------------------------------------------------------------------------------- | | 2.1 | Instrument parsing | Parse every supported family with complete identity, precision, currency, and contract fields. | | 2.2 | Instrument loading | Load, filter, cache, and emit definitions at each parsing boundary that needs context. | | 2.3 | Symbol mapping | Define bidirectional venue symbol and `InstrumentId` conversion without collapsing distinct instruments. | | 2.4 | Instrument updates | Implement fresh instrument requests and any supported definition or status updates. | **Exit:** Distinct fixtures cover every supported instrument family, invalid definitions fail clearly, and the data client emits or returns complete Nautilus instruments. ### Phase 3: Implement market data Start with one public stream and one instrument before adding product or endpoint fan-out. | Step | Component | Work | | ---- | ------------------------ | ----------------------------------------------------------------------------------------------------------- | | 3.1 | Public WebSocket streams | Subscribe and unsubscribe each advertised live data type while preserving subscription intent. | | 3.2 | Historical data requests | Request supported bars, trades, quotes, or order book snapshots with exact correlation and freshness rules. | | 3.3 | Data client | Implement `DataClient` requests, subscriptions, lifecycle, and complete `DataEvent` emission. | | 3.4 | Order book handling | Preserve snapshot, incremental update, sequence, clear, and batch boundaries. | | 3.5 | Stream recovery | Handle malformed input, gaps, unsubscribe, disconnect, reconnect, and subscription replay. | **Exit:** Unit and mock transport tests prove complete domain events for the supported request and subscription matrix. ### Phase 4: Implement execution Establish account state and reconciliation before enabling order flow. | Step | Component | Work | | ---- | ---------------------- | ------------------------------------------------------------------------------------------------------------ | | 4.1 | Account bootstrap | Establish account identity, initial account state, private subscriptions, and connected readiness. | | 4.2 | Reconciliation reports | Generate applicable order, fill, position, and mass-status reports at startup and on demand. | | 4.3 | Basic order submission | Implement supported market and limit order submission with deterministic local validation. | | 4.4 | Order modification | Implement supported modify and cancel commands, including cancel-replace venue semantics. | | 4.5 | Execution client | Implement `ExecutionClient` commands, lifecycle, tracked and external routing, and ordered event emission. | | 4.6 | Outcome recovery | Preserve unknown outcomes, deduplicate fills, and resolve state through streams, queries, or reconciliation. | **Exit:** Mock transport tests cover every supported command, definitive rejection, uncertain transmission, duplicate or out-of-order updates, and startup reconciliation. ### Phase 5: Add optional venue capabilities Add these only after the base lifecycle is stable. | Step | Component | Work | | ---- | -------------------------- | ------------------------------------------------------------------------------------------------- | | 5.1 | Advanced order types | Add applicable conditional, stop, take-profit, trailing-stop, or other advanced orders. | | 5.2 | Batch operations | Add batch submission, batch cancellation, and mass cancel with per-order result handling. | | 5.3 | Venue-specific data | Add funding, greeks, liquidations, or venue extensions as separate capability slices. | | 5.4 | Product or endpoint splits | Split ownership only when protocol, authentication, quota, or recovery boundaries require it. | | 5.5 | Capability proof | Add fixtures, functional tests, acceptance cases, and documented limitations for each capability. | **Exit:** Each optional capability is independently testable and does not weaken the established base paths. ### Phase 6: Complete factories and projection | Step | Component | Work | | ---- | --------------------- | ------------------------------------------------------------------------------------------------ | | 6.1 | Configuration structs | Finalize typed data and execution configs, defaults, environment fallback, and secret redaction. | | 6.2 | Client factories | Implement Rust factories with `CacheView` inputs and the data client clock. | | 6.3 | PyO3 registration | Register applicable factories and config extractors with the PyO3 registry. | | 6.4 | Python package | Add the public package and Python boundary tests for the capabilities exposed to Python. | | 6.5 | Generated stubs | Add Rust stub metadata and regenerate the `.pyi` output with `make py-stubs`. | **Exit:** Rust factory tests and PyO3 boundary tests pass, package imports resolve, and generated output matches its Rust inputs. ### Phase 7: Prove conformance | Step | Component | Work | | ---- | ---------------------- | ------------------------------------------------------------------------------------------------------ | | 7.1 | Rust unit tests | Prove parsers, serializers, symbols, signatures, state transitions, and malformed input. | | 7.2 | Rust integration tests | Exercise public HTTP, WebSocket, data, and execution boundaries against deterministic mock transports. | | 7.3 | Python boundary tests | Prove imports, config extraction, factories, type conversion, and representative async calls. | | 7.4 | Acceptance tests | Run every applicable `DataTester` and `ExecTester` case on testnet or a controlled account. | | 7.5 | Recovery tests | Exercise connection failure, reconnect, shutdown, rate limits, and state recovery. | | 7.6 | Specification gaps | Record every skipped specification case with a venue or capability reason. | **Exit:** The applicable data and execution testing specifications pass, and every advertised capability has deterministic and venue evidence. ### Phase 8: Measure performance and robustness | Step | Component | Work | | ---- | -------------------- | --------------------------------------------------------------------------------------------------------- | | 8.1 | Canonical benchmarks | Measure confirmed end-to-end data and execution hot paths with representative fixtures. | | 8.2 | Microbenchmarks | Isolate confirmed signing, hashing, authentication, codec, parsing, or serialization costs. | | 8.3 | Fuzz targets | Fuzz untrusted parsing, decoding, normalization, signing, and encoding boundaries with realistic corpora. | | 8.4 | Invariants | Assert domain and protocol properties stronger than panic freedom. | **Exit:** Canonical benchmark and fuzz suites run with representative fixtures, documented invariants, and no mandatory categories that the adapter does not use. ### Phase 9: Finish documentation and operations | Step | Component | Work | | ---- | ------------------- | ---------------------------------------------------------------------------------------------- | | 9.1 | Capability matrix | Reconcile every support claim and exception with the tested implementation. | | 9.2 | Integration guide | Document credentials, config, limits, reconciliation, environment differences, and known gaps. | | 9.3 | Tester entry points | Provide applicable Rust and Python data and execution testers with safe defaults. | | 9.4 | Operations | Document recovery, troubleshooting, and any venue behavior an operator must understand. | | 9.5 | Final verification | Verify links, generated output, examples, and the focused documentation checks. | **Exit:** A user can configure, test, operate, and diagnose the adapter without reading its source. ## Rust adapter patterns Repository-wide import policy applies to adapter code: import Nautilus types and use their short names instead of fully qualifying them at call sites. The [Nautilus conventions hook](../../.pre-commit-hooks/check_nautilus_conventions.sh) enforces this rule and documents its scoped exception marker. ### Configurations (`config.rs`) Follow the shared [configuration guide](../concepts/configuration.md). In particular, Rust configs use typed fields, strict Serde decoding, one source of truth for defaults, and `bon::Builder`. Adapter configs then add only venue semantics: - Use an enum for a closed set such as environment, product family, account mode, or endpoint. - Use `Option` only when absence has a distinct meaning, including runtime credential fallback. - Keep data and execution config separate when their capabilities or credentials differ. - Store fields that must not appear in `Debug` as `SecretString`. Derive `Debug` when every sensitive field uses a redacting type; write a custom implementation only when a field cannot use one or the type requires more restrictive output. - Keep Python config projection thin. It converts types and delegates to the Rust config. Centralize default HTTP and WebSocket endpoint resolution so one environment selection cannot mix live and test endpoints. Keep explicit URL overrides only where custom gateways, mock servers, or venue deployments require them. Test every supported environment and any precedence between an environment choice and an explicit override. ### Credentials and secret handling When HTTP and WebSocket clients use the same key material, centralize credential handling in a type, commonly under `common/credential.rs`. Keep configs as data transfer objects: resolve credentials when constructing the credential, factory, or client, not in Python wrappers or individual request methods. #### Classify sensitive values Classify a value before choosing its type and diagnostic output. Apply the more restrictive rule when a venue gives one value more than one role. | Value class | Examples | Diagnostic output | | ---------------------------- | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | Secret material | Passwords, private keys, API secrets, passphrases, bearer and session tokens, refresh tokens, and signatures. | Always show ``. | | Credential identity | API keys, client IDs, usernames, and account identifiers used during authentication. | Redact by default. Use a masked API key only when operational correlation requires it. | | Secret-bearing location | Proxy URLs, RPC URLs, request paths, and query parameters that can contain credentials. | Redact the complete location from logs and errors. | | Deliberately public identity | Wallet addresses, vault addresses, and public account names that the venue exposes publicly. | Show only when the type and adapter contract deliberately classify the value as public. | Do not infer that an API key, username, or URL is safe to print because it is not sufficient to authenticate by itself. Configs often cross logging, exception, and Python representation boundaries where partial credential identity remains sensitive. #### Use the common secret types Use `nautilus_core::string::secret` and `zeroize` instead of defining adapter-local redaction or zeroization conventions. | Mechanism | Use | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `SecretString` | Own a string that must zeroize on drop and render as `` with `Debug`. | | `REDACTED` | Replace an unconditional secret field in a custom `Debug` or `Display` implementation. | | `redact_option` | Preserve `Some` versus `None` while redacting an optional field in a custom `Debug`. | | `mask_api_key` | Correlate an API key only through an explicit masked-identity method. Do not use it for secret material. | | `Zeroizing` | Bound the lifetime of an owned plaintext `String`, byte buffer, decoded key, canonical payload, or serialized authentication message. | | `Zeroize` and `ZeroizeOnDrop` | Clear secret-bearing fields in structs that cannot use `SecretString`, including byte arrays and signing types. | | `zeroize_json_value` | Clear owned strings in a mutable JSON value after serializing secret-bearing fields. | #### Use `SecretString` safely - Treat serialization as plaintext. `SecretString` uses the underlying string for wire-format compatibility, so never serialize a config, credential, or authentication model for diagnostics. - Do not use ordinary `SecretString` equality to verify attacker-controlled secrets; it is not constant-time. - Borrow plaintext through `expose_secret()` only at the signing, encoding, or transport boundary that needs it. - Consume with `into_inner()` only to transfer ownership. If the receiving API requires `String`, create that final copy at the call boundary and do not retain it in adapter code. - Take `&SecretString` when a function only reads the value. Take it by value when the function retains or consumes it. - Put secret-bearing fields in the authenticated wire model instead of creating a second model only to change `Debug`. Derive `Serialize` and derive `Debug` when every sensitive field redacts. - Write a custom `Debug` for credentials backed by byte arrays, signing keys, or other types that cannot store their secret fields as `SecretString`. - Avoid `Display` for secret-bearing types unless a caller requires it. Any implementation must redact secret material. #### Resolve and share credentials - Define environment variable names once and select them from typed environment and product values. - Document the established environment variable names in the adapter's integration guide. - Register every adapter environment variable in `scripts/strip-adapter-env.bash`. `make pre-flight` runs through that wrapper with all of them unset, so an unregistered variable can let a test pass locally while depending on ambient credentials. - Resolve all fields as one credential set. Public clients may remain unauthenticated, but an authenticated client rejects an incomplete or invalid set before sending a request. - Convert config and environment strings into zeroizing owners at the credential boundary. Do not retain a non-zeroizing plaintext copy in adapter state after conversion. - Share credential storage across transports only when they use the same key material. Keep HTTP, WebSocket, and transaction signing methods separate when their canonical payloads differ. #### Project credentials into Python - Convert credential strings accepted by a Python constructor to `SecretString` at the Rust boundary. - Apply the Rust `Debug` and `Display` redaction rules to Python `__repr__` and `__str__`. - Expose only a presence check for secret material and secret-bearing locations. - Return credential identity, such as a username, only when an existing public API or another explicit caller needs it. Document the choice and keep the value out of diagnostics. - Never expose passwords, private keys, API secrets, passphrases, tokens, or signatures through plaintext getters. #### Bound plaintext lifetime - Zeroize each owned plaintext allocation after its final use, including normalized and decoded keys, secret-bearing signing payloads, serialized authentication messages, encoded form values, and mutable request models. - Prefer borrowed slices and existing zeroizing owners over intermediate `String` and `Vec` copies. - Limit the guarantee to allocations the adapter owns. Serialization libraries, transports, TLS, and the operating system may make copies the adapter cannot reach. - Keep plaintext lifetimes short; do not promise process-wide or transport-wide erasure. #### Redact diagnostics and transport errors - Never include credentials, signatures, secret material, or secret-bearing URLs in errors or logs at any level. - Log request metadata such as the method, field count, and byte lengths instead of credentials or authentication payloads. Shared transports log metadata rather than payload contents. - Treat TRACE as developer-facing diagnostic output. Raw inbound payloads are allowed when their schema cannot contain credential material. - Treat raw private-stream TRACE output as sensitive because it can disclose orders, balances, positions, and account identity. Redact it before sharing. - Prefer metadata or a sanitized, bounded excerpt when either can diagnose the protocol. - Clear mutable source models after serialization when they own another plaintext copy. - Never log a raw authentication request or response, or any frame whose schema can contain secret material. | Surface | Required handling | Zeroization boundary | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | HTTP secret body | Use `HttpClient::request_with_secret_body`. | The client retains the zeroizing owner; lower layers may copy it. | | HTTP path or `HashMap` query | Use `HttpClient::request_with_url_redacted`. | The URL is removed from logs and transport errors. | | HTTP typed query | Use `HttpClient::request_with_params_url_redacted`. | The URL is removed from logs and transport errors. | | HTTP headers and proxy | Create credential-bearing strings at the client boundary, avoid clones, and do not retain them in adapter state. | The shared client or transport may retain copies. | | WebSocket authentication | Keep fields and serialized frames in `SecretString`; create the final `String` immediately before `send_text`. | The shared client has no secret-owner-preserving send method. | | Unsupported combination | Extend the common client instead of implementing adapter-local URL or error scrubbing. | The common API must define the resulting ownership and redaction rule. | #### Verify credential handling Test the secret-handling contract as well as successful authentication: - Cover explicit values, environment fallback, incomplete credentials, and invalid credentials. - Assert that config, credential, request, response, and client `Debug` output omits the exact input secrets. Test `Display` separately for every secret-bearing type that implements it. - Assert that Python `__repr__` and `__str__` omit credential identity and secret material. Test presence checks and every deliberately exposed identity getter. - Assert that serialization and transport preserve the exact wire value where the venue requires plaintext. - Force transport failures for credential-bearing URLs and assert that both `Display` and `Debug` error output omit the URL, path secret, and query secret. - Use compile-time trait assertions for `Zeroize` or `ZeroizeOnDrop`, and test explicit clearing for mutable request and response models. - Keep deterministic signature vectors so redaction and zeroization changes cannot alter signing bytes, field order, or encoding. ### Symbols and instrument identity Separate venue symbols from Nautilus `InstrumentId` values. A symbol module commonly owns: - Parsing and formatting venue symbols. - Product or contract suffixes required for a unique Nautilus symbol. - Validation of venue and product identity. - Round-trip tests for supported forms and rejection tests for ambiguous forms. Choose the mapping from the venue's identity scheme: | Venue identity | Nautilus representation | Example | | --------------------------------------------------- | ----------------------------------------------- | --------------------------------------------------- | | Native symbol distinguishes the product | Preserve the symbol and add the venue. | `BTC-USDT-SWAP` -> `BTC-USDT-SWAP.OKX`. | | Raw symbol is reused across product families | Add and validate a stable product suffix. | Bybit linear `BTCUSDT` -> `BTCUSDT-LINEAR.BYBIT`. | | Nautilus and the venue use different contract marks | Implement both directions at one boundary. | Binance USD-M `BTCUSDT` -> `BTCUSDT-PERP.BINANCE`. | | Transport casing differs from canonical identity | Convert only when building the transport value. | Binance stream `BTCUSDT-PERP.BINANCE` -> `btcusdt`. | The [`BybitSymbol`](../../crates/adapters/bybit/src/common/symbol.rs) wrapper and [Binance symbol conversions](../../crates/adapters/binance/src/common/symbol.rs) show the suffix and bidirectional conversion patterns. Treat them as examples, not a shared suffix scheme. Do not normalize distinct venue instruments to the same `InstrumentId`. Give test fixtures distinct symbols, precisions, currencies, and contract fields so swaps and omissions fail visibly. For every supported product family, test venue symbol -> `InstrumentId` -> venue symbol. Normalize case once at the identity boundary and preserve venue-significant case elsewhere. When the mapping requires a product marker, reject a missing or ambiguous marker before caching the instrument. Construct instruments from current venue definitions. Validate required identity and precision before caching or emission. Keep parsing functions deterministic and independent of live client state where practical. ### Modeling venue payloads Model the wire format, not an imagined stable subset: - Use typed request and response structs for known fields. - Use Serde aliases or custom deserializers only when supported payloads require them. - Reject unknown values for closed sets whose meaning affects domain behavior. - Preserve or explicitly classify unknown values for open venue sets that may expand without a protocol version change. - Keep raw models separate from Nautilus domain objects. Convert at one auditable boundary. - Pass required parsing context explicitly, including instrument precision, currencies, account identity, and `ts_init`. Keep live client state outside parsers. - Treat missing, null, and empty values according to the venue schema. Do not collapse them into one fallback when they carry different meanings. - Use the venue timestamp for `ts_event` when the payload supplies one. Assign `ts_init` from the adapter clock when it receives or constructs the event. Use receipt time as event time only when the venue has no authoritative timestamp, and cover that fallback with a test. Avoid permissive fallbacks that silently turn a new venue value into an existing semantic value. Stable error handling is part of the parser contract. #### Numeric precision Deserialize prices, quantities, money, fees, and other discrete values as `Decimal`. Construct domain values with `Price::from_decimal`, `Price::from_decimal_dp`, `Quantity::from_decimal`, `Quantity::from_decimal_dp`, `Money::from_decimal`, or `Money::zero`; never route wire values through `f64`. See [domain numeric types](rust.md#domain-numeric-types). Choose domain precision from the field contract, not incidental payload formatting: | Field contract | `"25.000"` result | Conversion | | -------------------------------------------------- | ----------------------- | ------------------------------------------------------------------- | | Venue-declared scale is meaningful | `25.000` at precision 3 | Use `Price::from_decimal` or `Quantity::from_decimal`. | | Documented trailing zeros are non-semantic padding | `25` at precision 0 | Call `Decimal::normalize`, then use the scale-inferred constructor. | | Instrument or currency precision governs the value | `25.00` at precision 2 | Use `Price::from_decimal_dp` or `Quantity::from_decimal_dp`. | Use instrument or currency precision for event and report values when available. A venue may send the same value as `"25"`, `"25.0"`, or `"25.000"`, so do not infer precision per payload unless the adapter defines and tests an explicit compatibility fallback. The declared-precision constructors apply banker's rounding when a value has excess non-zero digits; validate round-trip equality when the field contract requires exact representation. During reconciliation, follow [instrument resolution](#instrument-resolution-during-reconciliation) when precision metadata is missing. #### Venue enum fallbacks Venues extend wire enums without notice: new order states, order types, and category codes appear in production before clients update. Give each extensible venue enum a forward-compatible fallback variant (`Unknown` for venue states, `Other` for open value sets such as types and categories) with `#[serde(other)]`, so one new value cannot fail deserialization of the message carrying it. Closed sets the adapter defines stay strict. The fallback changes where strictness lives, not whether it exists: - Never panic on an unknown wire variant; the fallback keeps the connection and the sibling records in the same payload alive. - Never map an unknown variant onto an existing domain value. Make the domain mapping fallible (`TryFrom`) so the fallback variant is rejected explicitly at the mapping boundary. - Preserve safety-critical payload data even when a sibling classification is unmapped. A fill must still be parsed and emitted when its order state or order type is unknown, because fill fields carry their own prices, quantities, and fees. - Skip only the unmappable classification and log a warning with the venue identifiers (order ID, instrument) needed to investigate. When the message carries no data worth preserving, fail the record explicitly instead of inventing a status. Reconciliation heals the gap once the order reaches a mapped state; an unmapped value fails the same way on the reconciliation path, so treat the warning as the signal to add the mapping. #### Separate authority from projections Use separate response models when one endpoint returns both evidence that establishes permission or authorizes state mutation and data needed for a narrower read. | Boundary | Purpose | Validation | Meaning of success | | -------------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | **Authoritative response** | Establish permission or authorize state mutation. | Requires all authoritative fields; rejects legacy conflicts and semantic duplicates before mapping. | The response can support the authority decision it models. | | **Narrow projection** | Read a balance, health value, or metadata without using authority. | Decodes returned fields only; its type cannot expose, grant, or infer omitted authority. | Only the projected value; omitted permission or account evidence is unknown. | Use the projection when malformed authority fields must not block the narrower read. Keep the authoritative model strict. ### Client traits and factories (`data.rs`, `execution.rs`, `factories.rs`) The shared [`DataClient`](../../crates/common/src/clients/data.rs), [`ExecutionClient`](../../crates/common/src/clients/execution.rs), and [client factory](../../crates/common/src/factories/client.rs) traits define the adapter boundary. Implement the supported methods and leave unsupported capabilities explicit in the integration guide. The client traits use `#[async_trait(?Send)]`. Client objects are not intended to move across threads and may hold non-`Send` Python state. Move owned, `Send` inputs into explicit runtime tasks when asynchronous work must outlive a synchronous trait call. #### Client naming and registration Name each client family symmetrically: `DataClient`, `DataClientConfig`, and `DataClientFactory` for data; `ExecutionClient`, `ExecutionClientConfig`, and `ExecutionClientFactory` for execution. Each factory consumes its corresponding client config directly. Do not add a separate factory config wrapper. The live node passes its `LiveNodeConfig.trader_id` to execution factories, while venue-specific values such as `account_id` belong on the execution client config. Within a Python module, order client-family `add_class` registrations alphabetically by exported type name so the data and execution families remain grouped. Do not prefix the ordinary client family with `Live`: a connected client is the default, while names such as `SandboxExecutionClient` and `DatabentoHistoricalClient` state alternate behavior. Retain `Live` only when it distinguishes explicit runtime or protocol siblings. Runtime types such as `LiveNode`, `LiveClock`, and the `Live*EngineConfig` family retain the qualifier. Do not shorten `Execution` in public, project-owned PascalCase type names. Internal implementation types may retain established `Exec` names. Also keep `Exec` where the [general naming convention](coding_standards.md#naming-conventions) allows it, including venue protocol terms such as `ExecType`. Name protocol-specific wire models after the venue concept, such as `HyperliquidExchangeAction`. Preserve established public names, and apply this convention to new APIs. #### Factory inputs and cache ownership Factories receive a downcast `ClientConfig` and a read-only [`CacheView`](../../crates/common/src/cache/mod.rs). Data factories also receive the shared clock. Use the view to resolve instruments and existing state. Engine cache writes stay in the engines: emit domain events and reports instead of mutating the engine cache from an adapter. A private protocol cache is valid when parsing, subscription replay, or response correlation needs it. ### Adapter-owned state Choose collections from ownership and update behavior: - Use a plain `AHashMap` or `AHashSet` for state owned by one task. - Use `AtomicMap` or `AtomicSet` for read-heavy immutable snapshots with infrequent writes. Use `rcu` when writers can race; a separate load and store can lose another writer's update. - Use `DashMap` or `DashSet` for independent keys that receive concurrent entry updates. Adapters use these patterns in different combinations. Keep the collection behind the component that owns its invariant instead of sharing it merely to avoid passing a message. Use `Ustr` for repeated protocol strings when interning reduces allocation or comparison cost; keep unique request IDs and short-lived payload text in their natural types. ### Connection lifecycle (`connect`) Treat each lifecycle method as a contract: | Method | Responsibility | Successful postcondition | | ------------ | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- | | `start` | Install local event plumbing and start client-owned background work. | Local event paths exist before any task can publish. | | `connect` | Establish transports, authenticate, load required definitions or account state, and start stream processing. | Public commands can use the transport, and required bootstrap state is observable. | | `disconnect` | Stop new network work and close transports. | The client no longer sends or receives venue traffic. | | `stop` | End client-owned work using an idempotent path. | Repeated teardown is safe. | | `reset` | Clear reconnectable caches, counters, cancellation state, and stale in-flight state. | A later start or connection does not inherit invalid session state. | | `dispose` | Release background tasks, threads, and external handles. | No client-owned resource remains active. | Do not report connected until public commands can use the transport and required engine-side state is observable. In particular, an execution client that emits initial account state asynchronously waits until the engine cache contains the account before calling `set_connected`; reconciliation and strategy startup treat connected as a readiness signal. Apply the same rule to required instrument or stream bootstrap state. When transport connection completes before the socket becomes active, use a bounded `wait_until_active` step before subscribing or reporting readiness. On partial connection failure, clean up resources already started and leave state consistent for retry or disposal. When an execution client uses [`ExecutionEventEmitter`](../../crates/live/src/execution/emitter.rs), resolve the execution event sender with `try_get_exec_event_sender` and install it in the factory's `create`, before the client is returned. `LiveNodeBuilder` binds the runner's senders to thread-local storage before it calls any registered factory, so `create` runs with the sender available. `None` from `try_get_exec_event_sender` means the calling thread has no bound senders, which is expected in a factory unit test or in a host that binds later; it is not a construction failure. Install the sender in `start` as well, from `get_exec_event_sender`, and do so unconditionally: `LiveNode` rebinds the runner's senders on the calling thread before it starts clients, so the `start` install is the authoritative one, and the emitter shares one sender slot across its clones, so it reaches every clone taken during construction, including those handed to client-owned tasks. A host that calls a factory outside `LiveNodeBuilder` binds the runner's senders on the client's thread before `start`: the `start` install resolves through `get_exec_event_sender`, which reads only the thread-local slot and panics when nothing has bound it, so a sender passed through the host's own factory or client constructor - which reaches the emitter's shared slot via `set_sender` but not the thread-local slot - does not satisfy that lookup on its own. Constructor injection stands alone only for a client whose `start` accepts an already-installed sender instead of performing the unconditional lookup; the emitter-backed execution clients in this repository all perform it. #### Bootstrap ordering Connection code varies, but its dependencies do not. A data client typically: 1. Resolves the environment and validates any credentials needed during bootstrap. 1. Fetches required instrument definitions and populates parsing context. 1. Publishes definitions that the engine must observe before data arrives. 1. Starts the transport and waits for the command path to become active. 1. Subscribes or replays intent only after handler initialization. 1. Reports connected after the required engine and protocol state is ready. An execution client typically: 1. Validates credentials, account identity, and required instrument context. 1. Establishes and authenticates the private transport. 1. Starts stream processing and subscriptions in an order that cannot lose acknowledgements or account updates. 1. Fetches and emits initial account state, or waits for an authoritative stream snapshot. 1. Waits for required account and instrument state to become observable to the engine. 1. Reports connected only after commands and reconciliation can run. Treat these as dependency constraints, not required function names. A venue can combine or reorder steps when tests prove the same postconditions. If any step fails after resources start, tear down those resources before returning the error. ### Data client Subscriptions express ongoing intent. Requests ask the provider for current or historical data. Keep their freshness semantics distinct: - An explicit instrument request fetches from the provider unless the API contract explicitly permits a cache result. - Private caches can provide parsing context, but must not turn a new request into a stale response. - Emit a response only for data that satisfies the request identity and filters. - Preserve the request correlation ID and original parameters in response events. The shared [`DataEvent`](../../crates/common/src/messages/mod.rs) envelope determines how data enters the engine: | Variant | Use | Contract to preserve | | ----------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | `DataEvent::Instrument` | Instrument definitions from bootstrap, requests, or updates. | Preserve complete identity, precision, and venue timestamps when available. | | `DataEvent::InstrumentStatus` | Trading or availability status changes. | Emit meaningful transitions rather than unchanged polling snapshots. | | `DataEvent::Data` | Trades, quotes, order-book data, bars, and other typed market data. | Complete parsing and event boundary construction before emission. | | `DataEvent::Response` | Results for current or historical data requests. | Preserve request correlation, parameters, filters, and freshness semantics. | | `DataEvent::FundingRate` | Funding rate updates for derivatives. | Preserve the venue's effective or event time and instrument identity. | | `DataEvent::OptionGreeks` | Venue-provided option greeks. | Preserve the source instrument and distinguish venue values from local calculation. | | `DataEvent::DeFi` | Feature-gated decentralized finance data. | Emit only when the adapter and build expose the shared `defi` feature. | Add a regression test that changes the upstream instrument response between two requests. The second response must reflect the new venue state rather than a private cache entry. Publish typed data through the engine's data event path. Parse and validate before emission, and do not hold mutable adapter state across downstream dispatch. A closed event receiver normally means the engine is stopping: log the send failure and let lifecycle teardown own recovery rather than retrying the same event indefinitely. For order-book deltas, follow the [delta flag and event boundary contract](../concepts/data/index.md#delta-flags-and-event-boundaries). Every logical update ends with `F_LAST`; snapshots use `F_SNAPSHOT` and end with `F_SNAPSHOT | F_LAST`, including an empty snapshot represented only by `Clear`. When a venue exposes instrument status only as a polled snapshot, diff it against the prior full snapshot and emit changes rather than repeating every status. Treat an instrument removed from the snapshot according to the venue contract. Map removal to `NotAvailableForTrading` only when disappearance means the instrument is unavailable. Update the full private cache even when emissions are filtered to active subscriptions. ### Execution client Execution clients translate commands, preserve order identity, publish account state, and generate reports for reconciliation. They must support these boundaries consistently: - Validate deterministic local constraints before submission. - Emit `OrderSubmitted` only when the command enters the adapter's submission path. - Correlate venue responses and stream updates to the correct client and venue order IDs. - Emit balances and margins with the account type and base currency used by the factory. - Generate order, fill, position, and mass-status reports from venue state for reconciliation. - Release shared clock, cache, or account borrows before publishing account state because subscribers may access the same state synchronously. Keep deterministic adapter-specific checks in one `validate_order` function that returns `OrderDeniedReason`. Call it before emitting `OrderSubmitted` from single-order and order-list submission paths. Do not infer support from a venue API alone. Implement and test the Nautilus command and event semantics, then advertise the capability. #### Reconciliation reports Reconciliation reads venue state through five [`ExecutionClient`](../../crates/common/src/clients/execution.rs) report methods. They return reports rather than emitting order events, leaving the execution engine to decide what a difference between cached state and venue state means. | Method | Produces | Driven by | | ---------------------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------- | | `generate_order_status_report` | One optional [`OrderStatusReport`](../../crates/model/src/reports/order.rs). | A targeted probe for one order the open-order check left unresolved. | | `generate_order_status_reports` | [`OrderStatusReport`](../../crates/model/src/reports/order.rs) values. | Mass status and the periodic open-order check. | | `generate_fill_reports` | [`FillReport`](../../crates/model/src/reports/fill.rs) values. | Mass status. | | `generate_position_status_reports` | [`PositionStatusReport`](../../crates/model/src/reports/position.rs) values. | Mass status and the periodic position check. | | `generate_mass_status` | One optional [`ExecutionMassStatus`](../../crates/model/src/reports/mass_status.rs). | Startup reconciliation, once per execution client. | [Execution reconciliation](../concepts/execution/reconciliation.md) documents what the engine does with these reports, including the startup procedure, the runtime checks that drive the periodic and targeted requests, and their retry and throttling rules. Cases TC-E84 to TC-E87 and TC-E101 in the [execution testing specification](spec_exec_testing.md) exercise startup reconciliation against a venue. Cases TC-E88 and TC-E89 use deterministic fixtures to exercise REST and private-stream commission failure. ##### Startup mass status `generate_mass_status` runs once per execution client before trading starts. Its default implementation composes the three bulk methods concurrently from one `ts_init`, derives each command's `start` from `lookback_mins`, and requests full order history with `open_only=false`. Implementing the bulk methods is therefore enough for startup. Override the composition when the client declares a history bound, as described in [bounded mass-status reports](#bounded-mass-status-reports), or when it does not use the realtime clock. Returning `Ok(None)` logs a warning and leaves that client unreconciled, while an error fails startup. ##### Bulk report filters The bulk methods take a filter command carrying `instrument_id`, `start`, and `end`, plus `open_only` for order reports and `venue_order_id` for fill reports. Apply every filter the venue endpoint supports and complete the rest locally: - `open_only` separates the currently open orders a periodic check needs from the history a mass status needs. Retain a report for `open_only` when its status is open **or** in-flight, not open alone: a venue holding an order it has not yet acknowledged reports it as `SUBMITTED`, which is in-flight rather than open. - Apply `start` and `end` only to closed reports, since an order working at the venue is authoritative however long it has rested without an update. - Test a report for a terminal status with `is_closed()`, never `!is_open()`, which classifies `SUBMITTED` as terminal. Log report counts at the command's `log_receipt_level` so periodic checks stay at debug while mass status logs at info. When a periodic check request fails, the engine marks that client failed for the cycle and stops inferring absence for the orders and positions it covers. Returning an error is therefore safer than returning an empty set. ##### Single-order probes `generate_order_status_report` resolves a single order. The engine issues it after the open-order check retries without confirming a cached order, which requires that check to run in full-history mode (`open_check_open_only=false`). The command carries the queried `instrument_id` and `client_order_id`, plus `venue_order_id` when the order has one, so support a lookup that has no venue identifier yet. The engine discards a report whose identity does not match the query. Distinguish absence from failure in that probe, because the engine acts on the difference: - `Ok(None)` states that the venue answered and has no such order. The engine treats that as proof and resolves an accepted, submitted, or partially filled order to a terminal state, while pending cancel and update states stay unresolved. - An error states that the lookup did not answer, so the engine defers the missing-order resolution to a later cycle. A failed lookup returned as `Ok(None)` can therefore reject or cancel an order that is live at the venue. The trait default returns `Ok(None)` after logging that the handler is not implemented, so implement this method before an open-order check runs in full-history mode. #### Commission failure handling Commission is part of a fill's economic record. Calculate it with exact decimal arithmetic, then construct the venue currency's [`Money`](../../crates/model/src/types/money.rs) value. The shared `ExecutionClient::calculate_commission` hook distinguishes these outcomes: | Result | Meaning | | ---------------------- | ---------------------------------------------------------------------------------------- | | `Ok(Some(commission))` | The venue formula applies and produces a representable commission. Use that exact value. | | `Ok(None)` | The adapter has no venue override. The caller may use the generic commission formula. | | `Err(error)` | The venue formula applies but cannot produce a representable value. Fail closed. | Never replace `Err(error)` with zero commission or the generic formula. That substitution records a confirmed trade with economics the venue did not report. ##### REST report construction Commission construction belongs to the REST report request. If it fails for any required fill, return an error from the direct fill report request, targeted recovery, or complete mass status. Never drop the fill, return a partial mass status, or mark a bounded report set incomplete for this failure. Otherwise, an order or position report can cause the engine to infer the same quantity without its venue commission. During startup, the error prevents the node from starting and leaves that client's mass status unapplied. Periodic and targeted reconciliation defer the affected work until a later cycle. ##### Inferred fills Call the hook for every adapter-backed inferred fill: external and cached orders, continuous reconciliation, and targeted order recovery. If commission calculation fails, the engine may apply valid explicit fills, but it leaves the residual inferred quantity and dependent terminal transition pending. For an external order, calculate commission before a cache or event transition could prevent a retry. If the responsible execution client is unavailable, defer the inferred fill instead of treating the missing client as an `Ok(None)` response. Pass the same quantity, price, and liquidity side as the inferred-fill event. For a cached order with prior fills, calculate commission from the back-solved price of the unbooked incremental quantity, not the venue report's cumulative average price. A position-only synthetic correction has no underlying trade evidence and may leave commission unspecified. Do not present an aggregate or generic value as the exact commission for that unknown fill; this case is distinct from a failed venue calculation. ##### WebSocket trade processing Process each WebSocket trade atomically. Construct every owned maker and taker fill report before emitting any report, mutating fill trackers, or consuming the trade's deduplication key. Consume the key only after all reports route successfully. On failure, log the error and leave the trade unprocessed. Do not confirm or terminalize the affected orders or mark them permanently unreconcilable. A duplicate or reconnect replay can retry the trade. Scheduled REST reconciliation remains the authoritative recovery path; the WebSocket handler does not start an immediate REST request. #### Bounded mass-status reports When an execution client applies a lower time bound to historical reconciliation reports, record the contract with `ExecutionMassStatus::set_report_window(Some(lookback_start), reports_complete)`. Capture one cutoff for the mass-status request and use it for every historical order and fill query. A moving cutoff can omit records at different boundaries and produce a report set that never existed at the venue. Set `reports_complete=true` only when every source needed to interpret the bounded history completed and all required records were parsed, mapped, and linked to their orders. A failed required source, required row that cannot be parsed or mapped, or historical fill without its required order report makes the set incomplete. Preserve successful legs and authoritative active orders, but do not represent a failed historical query as a successful empty result. Commission construction is an exception to partial bounded history. Follow [commission failure handling](#commission-failure-handling) and fail the report request instead of returning a set that omits the affected fill. When positions come from a cached stream, absence proves flat only when a complete snapshot from the current connection epoch positively covers that instrument. Invalidate snapshot coverage on reconnect, and keep a row uncovered when it could not be parsed or mapped. Emit an explicit flat report for an absent touched instrument only after that coverage is established. Preserve stable venue order and trade identities across live dispatch and mass status. Include client order linkage and `venue_position_id` where the venue supplies them so the execution engine can distinguish a coherent lifecycle from ambiguous history. See [Bounded history safety](../concepts/execution/reconciliation.md#bounded-history-safety) for the engine's economic application rules. #### Instrument resolution during reconciliation Report generation resolves each record's instrument to parse venue payloads at the correct price and size precision. Resolve it from the instruments the adapter loaded during connect, and classify a miss by whether the record was in scope. Do not request an instrument from the venue while generating reports: - Per-record requests multiply the bulk queries that startup reconciliation already issues against the venue's rate limits. - Hidden requests make reconciliation timing and results irreproducible. - A failed request cannot be distinguished from an instrument the venue does not have. Load what the adapter needs during connect instead. An in-scope record whose instrument is missing is never dropped silently. A discarded open order report is indistinguishable from an order the venue never had, which leads the engine to resolve a live order as missing at the venue. Scope decides whether a miss is expected, so evaluate it before classifying the record: | Record | Outcome | Report set | | ---------------------------------------------- | --------------------------------------- | ---------------------------------- | | Out of scope for `load_ids` | Log at debug and drop | Unaffected | | In scope, open order or position status report | Return an error from the report request | Not returned | | In scope, closed or historical record | Log a warning naming the instrument | Incomplete when history is bounded | `InstrumentProviderConfig.load_ids` defines that scope. When it names an explicit set, records for instruments outside it are expected absences rather than errors, so a node scoped to one instrument neither fails nor warns because the venue returned records for the rest. Historical queries reach past the loaded instrument set routinely, because expiries retire instruments that earlier fills still reference. Failing a bounded-history query for one expired instrument would withhold every other record it returned, so record the incompleteness through `set_report_window` and let the engine apply its bounded-history rules. The engine acts on that incompleteness only for a mass status that declares `lookback_start`; an adapter that declares no bound follows the compatibility fill-adjustment path instead. `reconciliation_instrument_ids` filters reports after the execution engine receives them, so it cannot prevent a resolution failure inside an adapter. Keep the adapter's scope in its instrument provider configuration. #### Tracked and external execution updates Route execution updates according to order ownership, independent of the dispatch module layout: - For an order submitted and tracked by this client, emit typed order events through the normal order state machine. - For an untracked or external order, emit `OrderStatusReport` and `FillReport` values so the execution engine can reconcile or create the external order. Do not invent strategy or client identity for an untracked order. Preserve available venue identity in the report and let the engine apply [external order ownership](../concepts/execution/reconciliation.md#external-order-creation). The adapter may use any state structure that proves this routing decision. Model tracked ownership with two conceptual layers: - **Order identity** contains the stable fields that associate an update with the submitted order: client order ID, strategy, instrument, side, and order type. - **Order context** combines that identity with the submitted order shape needed to construct later events without accessing the engine cache, such as quantity, price and trigger details, time in force, and execution flags. Keep venue order bindings, request correlation, cumulative fills, and replace state in adapter-owned context around that common surface. [`OrderIdentity` and `OrderContext`](../../crates/live/src/execution/context.rs) provide that surface. Start from them, and keep an adapter-local structure only where it proves the same routing decision. Register the order context before sending or spawning work that can produce an inbound update. Restore context for active local orders before processing their live updates, and retain it while the order can still produce owned updates. Do not evict active context merely to bound replay state. Make every execution update take one explicit route: | Route | Evidence | Result | | ---------- | ------------------------------------------------------ | -------------------------------------------------------------- | | Tracked | Active or pending context owns the order. | Emit, deduplicate, or safely defer typed order events. | | External | No tracked, pending, or terminal ownership exists. | Forward reports for reconciliation or external order creation. | | Suppressed | The update is proven duplicated, stale, or superseded. | Emit neither an event nor a report. | Missing tracked metadata, a parse failure, or an unresolved venue binding does not prove that an update is external. A tracked status with no corresponding Nautilus lifecycle event is a tracked no-op or deferred update unless the adapter documents and tests a report exception. ##### Triggered parent and child orders Some venues replace a tracked parent venue order with a child after a trigger. Treat both venue IDs as one order context: - Keep the client identity stable. - Bind the child atomically with the trigger transition. The child ID becomes authoritative for later events and commands. - If the child arrives during the binding race, use venue linkage to complete the tracked transition or defer the update. Do not route it as external. - Once the child is authoritative, suppress stale parent acceptance and any superseded parent update that would regress child authority. ##### Incomplete and late updates Keep these cases distinct from normal tracked events: - An aggregate parent status without the trade identity or other fields required for a typed event remains a tracked no-op or deferred update. The authoritative child produces the live event, while report-based recovery remains in the reconciliation path. - A fill with a new trade identity that arrives after the tracked lifecycle reached a terminal state remains a `FillReport`. The active order context is no longer available for a typed event, so reconciliation applies the late venue evidence. Suppress terminal status replays and fills whose trade identity was already processed. #### Event ordering and deduplication A venue can report the same transition through an order response, private stream, query, and reconciliation result. Deduplicate by stable venue identity, not by the transport that delivered the update: - Use the venue trade or match ID for fills. Include account, instrument, or product identity when the venue does not guarantee global uniqueness. - Share fill identity across live dispatch and reconciliation when those paths can overlap. - Do not consume a deduplication key before parsing and routing succeeds. If the implementation reserves first, release the key after a failure so a replay can recover the event. - Bound long-lived deduplication state, but retain enough history across reconnects to cover venue replay. Reset it only when the protocol proves old identifiers cannot return. - Reuse the shared [`FifoCache` and `FifoCacheMap`](../../crates/common/src/cache/fifo.rs) when first-in, first-out eviction matches the replay contract. Keep adapter-specific locking where several state changes must remain atomic. - Make repeated acknowledgements and order snapshots idempotent. They must not regress state or emit a second lifecycle event. Keep active order context, pending correlation, replay deduplication, and terminal tombstones as separate lifecycle concepts even when one state object owns them. Bound replay and tombstone state without letting eviction reclassify an update for an active order as external. For a tracked order, a definitive fill can arrive before an acknowledgement or open-order update. Emit any required preceding lifecycle event only when the adapter has complete order identity and the venue evidence proves that state. Record the synthesized transition so a later acknowledgement does not duplicate it. Untracked orders continue through reports rather than synthesized strategy events. When parent and child updates can arrive on different streams: - Serialize the venue order ID binding with the lifecycle events that publish it. - Emit any required `OrderAccepted`, `OrderUpdated`, and `OrderTriggered` events before routing a child fill or cancellation. - Route commands from the same authoritative binding. Do not infer it from an engine cache that may still be processing those events. When a venue implements modify as cancel-replace, update the venue order ID mapping before routing the replacement leg. Distinguish a stale cancel for the old leg from cancellation of the active replacement, and calculate replacement quantity from current cumulative fills. This behavior is venue-specific and needs focused race tests; it does not imply a shared dispatch state layout. Focused tests distinguish tracked and external updates, fills that precede acknowledgement, duplicates from overlapping sources, submission or venue-binding races, and stale post-terminal updates. Test active-context retention separately from bounded replay eviction. #### Order command outcome policy Use [Execution policies](../concepts/execution/policies.md) as the cross-adapter contract for command delivery, event application, persistence, and recovery. Separate three evidence classes: - **Definitive local failure:** local evidence proves that the command was never transmitted. Deterministic validation before a submit is one example. Emit `OrderDenied` before `OrderSubmitted`. For cancel or modify preparation, emit the matching rejection only when the failure is attributable to that command and proves it was not sent. Otherwise, log the failure without inventing a rejection. - **Definitive venue result:** a structured venue response or status explicitly accepts, updates, or rejects one command. Emit the matching domain event. - **Unknown outcome:** the request may have reached the venue, but no definitive result is available. Keep the command in flight for stream updates, polling, queries, or reconciliation. ```mermaid flowchart TD command[Submit order] --> valid{Deterministic local validation passes?} valid -->|No| denied[OrderDenied] valid -->|Yes| submitted[OrderSubmitted] submitted --> unsent{Local evidence proves no transmission?} unsent -->|Yes| rejected[Emit OrderRejected] unsent -->|No| evidence{Definitive venue evidence?} evidence -->|Accepted or updated| event[Apply the venue event] evidence -->|Explicit rejection| rejected evidence -->|No| unknown[Keep the outcome unknown] unknown --> recovery[Resolve from stream, query, polling, or reconciliation] recovery --> event recovery --> rejected ``` If a submit failure occurs after `OrderSubmitted`, emit `OrderRejected` when local evidence proves that the command was never transmitted. Otherwise, leave the order in flight unless definitive venue evidence resolves it. Transport errors, timeouts, disconnects, task cancellation, retry exhaustion, HTTP 5xx responses, rate limits, missing acknowledgements, and parse failures after transmission usually leave an unknown outcome. Do not convert them into a venue rejection. For batch commands, apply evidence per order. A whole-request failure does not prove that every child command failed. Treat venue messages such as "not found" or "already closed" according to documented venue semantics; they may describe a race with a fill or cancellation rather than an unambiguous command rejection. Keep this policy independent of the HTTP or WebSocket path used to send a command. #### Naming the evidence classes Use consistent names so command failure classifications can be compared across adapters and the same wire condition is classified consistently. Classify every state-changing order command failure as one [`CommandFailure`](../../crates/live/src/execution/failure.rs) variant: | Evidence class | `CommandFailure` variant | Terminal event from this evidence | | -------------------------- | ------------------------ | --------------------------------- | | Definitive local failure | `NotSent` | Valid | | Definitive venue rejection | `VenueRejected` | Valid | | Unknown outcome | `Ambiguous` | Never | An `Ambiguous` classification never emits a terminal event by itself. Later definitive evidence from a stream update, query, poll, or reconciliation still resolves the command either way. A definitive venue acceptance or update is not a failure and carries no variant. Apply the venue event directly. Classify once at the execution boundary, using the evidence preserved by lower layers, rather than re-branching on the error enum at each emit site. Apply this to every state-changing order command, submit, modify, and cancel alike, including their batch and list forms. A classifier scoped to one command type leaves the others to drift. Queries produce no terminal command event and need no classification. Keep this axis separate from `is_retryable`. Retryability answers whether to send the request again; ambiguity answers whether the venue may already have acted on the first attempt. An error can be both, either, or neither, and collapsing them is what makes an unknown outcome look like a rejection. Two conditions are easy to misfile: - An HTTP 5xx without definitive command evidence is ambiguous. It proves only that the command was not confirmed, never that it was not applied, and a gateway 5xx does not prove the request failed to reach the venue. - A response parse failure is ambiguous, while a request encoding failure is `NotSent`. Both may surface as one serialization error variant, so classify by which side of the write boundary the failure occurred on. #### Separate diagnostics from strategy-facing reasons Preserve a structured diagnostic error through classification and logging. Derive a strategy-facing reason only at the execution event boundary, after the outcome and retry decisions. | Representation | Consumers | Required content | | ---------------------- | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | Diagnostic error | Classification, retry control, logs, and operators | Typed source plus available status, venue code, endpoint, backoff, transport, and decode context. | | Strategy-facing reason | Rejection events consumed by strategies | Bounded venue meaning without HTTP prefixes, response envelopes, markup, control characters, or secrets. | Format standardized local denial messages from [`OrderDeniedReason`](../../crates/model/src/events/order/denied_reason.rs) with the minimum suffix needed to identify the diagnostic context: - Emit `CODE` when the denial needs no diagnostic suffix. - Use `CODE: value` for one typed value or a free-text diagnostic. The code already identifies a single value, so do not repeat its name. - Use `CODE: key=value, key=value` only when multiple typed values need disambiguation. - Use `CODE: value; free text` when one typed value precedes a free-text diagnostic. Only the leading code is canonical. Do not parse the diagnostic suffix to recover classification, retryability, or command outcome. Apply these rules at the boundary: - Classify from typed or structured evidence. Never recover status, retryability, or command outcome from formatted display text. - Extracting a clean reason must not erase the diagnostic error or the evidence used to classify it. - Prefer documented venue error fields and codes. Sanitize and bound raw fallback text before logging, interning, or emitting it, and use a stable fallback when no useful text remains. - Map equivalent venue evidence through the same adapter-owned classification and reason functions whether it arrives through HTTP, WebSocket, polling, or reconciliation. Set `OrderRejected.due_post_only` from a structured venue code or flag when the protocol provides one. Otherwise, use one narrow adapter-owned, source-backed message classifier across every venue path. Test exact positive cases and close non-matching messages. Do not introduce a cross-adapter venue classifier. ## HTTP client patterns ### Client structure A common design separates three responsibilities: | Layer | Accepts | Produces | Owns | | ---------------- | --------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------ | | Raw client | Venue request and query types. | Venue response models and transport errors. | Transport, authentication, rate limits, and exact wire encoding. | | Domain client | Nautilus identifiers and domain values. | Domain objects, reports, or acknowledgements. | Operation semantics, parsing context, caching, and domain mapping. | | Execution client | Nautilus execution commands. | Lifecycle events and execution reports. | Command lifecycle, failure evidence classification, and terminal event policy. | The execution client is the shared command outcome boundary. Raw and domain clients preserve enough adapter-specific evidence to distinguish a failure before transmission from one after the venue may have received the request. They keep their natural venue and domain return types; do not make them return `CommandFailure` only to standardize command handling. Share the `CommandFailure` evidence classes and terminal event policy across adapters. Keep venue error codes, response statuses, protocol semantics, and their mapping to evidence classes inside the adapter. Do not introduce a cross-adapter venue classifier or shared classification trait. Use one HTTP client layer when the protocol is small and a raw/domain split would only add forwarding methods. Split by product when endpoints, signatures, or response models change for different product families. The execution boundary remains the same in either structure. Name low-level methods after the venue operation when practical, such as `get_instruments` or `place_order`. Name domain methods after Nautilus semantics, such as `request_instruments`, `submit_order`, or `cancel_order`. #### Request flow Whether one client or two own these responsibilities, keep their boundaries explicit: 1. At a domain boundary, validate Nautilus inputs and build typed venue parameters. 1. At the transport boundary, select the HTTP method and path, then serialize the exact query or body. 1. Allocate any required request identity, timestamp, or nonce. Sign the exact wire representation when needed, then send it through the shared `nautilus_network::http::HttpClient` with the applicable rate-limit keys. 1. Decode the response envelope and preserve transport, HTTP status, venue, and parse failures. 1. At a domain boundary, convert successful payloads to domain types with explicit instrument, account, and time context. Keep typed request construction separate from sending. This makes signatures and canonical query encoding testable without a server. Put response conversion in pure parser functions when it does not need live state. Typed request and query builders preserve the difference between an omitted field, an explicit zero, and an empty value. Keep required venue parameters required, omit absent optional parameters from the wire representation, and test the exact serialized query or body. Pagination code also tests cursor direction, inclusive boundaries, duplicate boundary records, and a repeated cursor or empty page so it cannot loop forever. ### Request signing and authentication For each attempt, build the exact canonical bytes required by the venue, then sign that representation once. Test: - Field order and delimiters. - Timestamp and receive-window units. - Decimal and enum encoding. - Body or query hashing. - Environment and account identifiers. - Known venue vectors when available. Keep nonce or sequence ownership explicit. If commands can run concurrently, define how the adapter serializes, allocates, or rejects conflicting nonces. Never retry a signed state-changing request with a new identity unless venue semantics make that safe. Treat request identity, timestamp, and nonce as separate protocol fields even when the venue packs them into one signed payload. The component that allocates a nonce also owns its ordering rule. Build and sign from the same reserved value, then handle pre-send failure, uncertain transmission, and venue nonce rejection according to documented venue consumption semantics. On a sequence mismatch, resynchronize from an authoritative source before issuing further state-changing commands. Test deterministic vectors, concurrent allocation, monotonicity or uniqueness, and recovery after a rejected sequence. ### Error handling and retry logic Map transport, HTTP status, venue error, parse, and validation failures without erasing their source. Retry only when the failure is classified as transient and the operation is safe to repeat. Compose both decisions at the client boundary; a predicate on the error alone is not a complete retry policy. This contract applies to HTTP and WebSocket request paths. #### Classify transient failures Keep transient failure classification adapter-owned because venue status codes, error codes, and rate-limit semantics differ. Give each adapter transport one production classifier entry point. Define rules shared by HTTP and WebSocket once within the adapter, then call them from those entry points. Remove superseded classifier paths. Do not introduce a cross-adapter venue classifier or shared trait. #### Gate retries by operation safety At each call site, bypass retry for an unsafe operation or pass a `should_retry` predicate that combines transient failure classification with operation safety. Reads and other idempotent operations may retry classified transient failures. Retry a state-changing operation only when repeating the same request cannot apply the command twice or cause another state change. The protocol may guarantee this through duplicate detection for a stable request identity or idempotent semantics for the same target. Otherwise, send the command once and resolve an unknown outcome through stream updates, queries, polling, or reconciliation. #### Preserve identity and ambiguity Allocate the semantic request identity outside the retry closure and keep it stable across attempts. Regenerate authentication timestamps, signatures, or other transport fields only when changing them does not alter the venue's request identity or duplicate detection. Ambiguity is monotonic across attempts for one semantic command. Once any attempt may have reached the venue, a later failure remains ambiguous. A later venue rejection resolves it only when documented semantics make the response authoritative for the same request identity and prove that no attempt was applied. An acceptance resolves ambiguity only when it correlates to the same semantic command. Validate a returned venue identifier for syntax and expected scope before constructing a domain identifier or binding it to a local order; a non-empty string alone is not proof. A malformed or mismatched identifier leaves the outcome ambiguous unless separate authoritative evidence proves rejection. Treat a venue duplicate-identity response as evidence that the venue saw an earlier request with that wire identity, not as a rejection of the original command by default. Use it to resolve the current command only when the adapter proves the same semantic identity. If the identity may collide or its scope is uncertain, keep the outcome ambiguous and reconcile it against venue state. #### Handle backoff and termination Respect venue backoff and rate-limit signals, and stop retries on cancellation. Use the shared [`RetryManager`](../../crates/network/src/retry.rs) when its cancellation and backoff model fits. `RetryManager` passes a typed [`RetryError`](../../crates/network/src/retry.rs) to the caller's error callback: `Canceled`, `OperationTimeout`, `ElapsedBudgetExceeded`, or `InvalidConfiguration`. Match a variant when the adapter must distinguish its control reason; never branch on display text. The error returned for `OperationTimeout` is evaluated by `should_retry`, so map it to the adapter's transient timeout variant when timeouts should retry. Other synthesized reasons terminate without reclassification. `InvalidConfiguration` is created before the operation can run, so it is a definitive local failure. Preserve that evidence instead of mapping it to a transport or ambiguous outcome. `RetryManager` control errors do not record whether the operation ran. Track possible transmission at the adapter boundary for state-changing commands, treating entry into the send operation as possible transmission unless more precise evidence exists. Classify cancellation, per-attempt timeout, and retry exhaustion as `CommandFailure::NotSent` only when local evidence proves that no attempt was transmitted; otherwise, classify them as `CommandFailure::Ambiguous`. Map every elapsed-budget termination path by transmission evidence rather than the returned error shape. When an error provides a minimum delay and the effective retry delay cannot fit within the remaining budget, `RetryManager` returns the original operation error instead of a synthesized budget error. #### Test retry behavior Focused tests distinguish: - Transient failures from permanent failures. - HTTP 429 responses with and without a venue backoff hint when the protocol exposes one. - An idempotent operation that retries and a state-changing operation that must not retry. - A final failure after a possibly transmitted earlier attempt, including a later venue rejection. - A duplicate-identity response for the same semantic command and a wire-identity collision with a different command. - Stable semantic request identity across attempts, including retries with refreshed authentication fields when the protocol permits them. - Cancellation, per-attempt timeout, and every elapsed-budget termination path before and after possible transmission. ### Rate limiting The shared [`HttpClient`](../../crates/network/src/http/client.rs) supports one or more rate limiters. Scope limiter state to the venue quota, not to a convenient Rust object: - Share a bucket across clients and operations that consume the same allowance. - Separate buckets only when the venue publishes independent quotas. - Acquire all required quota before sending a request. - Keep pagination and retry loops inside the same policy. Match the venue's actual meter: window shape, burst behavior, endpoint weights, and shared external traffic. A token bucket at the headline rate can still exceed a strict rolling window after an idle burst. Do not assume wire latency creates headroom. When the venue separately caps concurrent unacknowledged commands, add a closed-loop in-flight gate beside the send-rate limiter. Release its slot on every terminal acknowledgement, rejection, or send failure, and reset the gate on reconnect. A rate limiter alone cannot observe acknowledgement latency. Do not copy one adapter's bucket names or quotas into another. Document user-visible limits and configuration in the integration guide. ## WebSocket client patterns WebSocket dispatch follows the shared ownership and routing contract while module layout and state containers remain adapter-specific. Keep new code aligned with the shared network abstractions, bounded cache primitives, and nearest protocol peers. Do not treat one adapter's dispatch modules or a union of venue state as the target architecture. ### Client structure A common pattern separates an outer client from a handler task: - The outer client owns lifecycle, authentication coordination, subscription intent, and the stream exposed to data or execution clients. - The handler owns the `WebSocketClient`, serializes commands, decodes frames, and emits typed messages. - Channels transfer owned commands and messages across the boundary. Some adapters use stream mode and perform reconnection in the adapter. Others use the network client's handler mode and automatic reconnection. Both are legitimate. Split market data and trading handlers only when endpoints, authentication, throughput, or protocol semantics justify the extra lifecycle. Choose client boundaries from protocol facts: | Protocol shape | Structure to consider | Obligation | | --------------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------- | | One endpoint and one multiplexed protocol | One client and handler | Route by typed channel identity without duplicating lifecycle state. | | One protocol across separate product endpoints | One orchestrator with a client collection | Connect, close, and replay intent for every active product client. | | Separate public, private, or trading endpoints | Separate transports with shared models where useful | Authenticate and recover each endpoint according to its own contract. | | Different wire formats, signing, or reconnect rules | Separate protocol modules behind shared data or exec logic | Keep shared instrument and order identity above the protocol-specific code. | This table is a decision aid, not a target dispatch architecture. Do not split a client only to match another adapter's filenames, and do not combine endpoints when doing so hides independent authentication, quotas, or recovery. ```mermaid flowchart LR subgraph client["Client (orchestrator)"] cmd_tx["cmd_tx
├ Subscribe { args }
├ PlaceOrder { params }
└ MassCancel { id }"] out_rx["out_rx
<- {Venue}WsMessage
<- Authenticated
<- ChannelData"] end subgraph handler["Handler (I/O boundary)"] cmd_rx[cmd_rx] out_tx[out_tx] ws[WebSocket] end cmd_tx --> cmd_rx cmd_rx -->|"serialize"| ws ws -->|"parse -> transform"| out_tx out_tx --> out_rx ``` The diagram shows the common ownership boundary, not required type or field names. #### Handler initialization handshake (`SetClient`) Several adapters move a connected `WebSocketClient` into an already running handler through a `SetClient` command. Others construct or publish the handler differently. Preserve this invariant in either design: > No public subscribe, order, or control command can overtake handler initialization. Queue initialization before publishing a command sender or connected state, or use another mechanism that proves the same ordering. Test a command issued at the connection boundary so a race cannot silently drop it. ### Authentication Use [`AuthTracker`](../../crates/network/src/websocket/auth.rs) when authentication state must be shared across the client, handler, and reconnect path. The adapter still owns protocol details: - Begin one authentication attempt and correlate the response. - Mark positive success or failure from a definitive venue frame. - Invalidate authentication on reconnectable connection loss. - Fail waiters on terminal shutdown. - Gate private replay and commands until authentication succeeds. Refreshable tokens, multiple account sessions, and mixed public/private endpoints need adapter-specific state. Keep that state close to the credential and subscription paths and cover rotation or expiry with focused tests. ### Subscription management Use [`SubscriptionState`](../../crates/network/src/websocket/subscription.rs) when the venue has acknowledged subscriptions or reconnect replay. It separates intent from confirmation and includes reference counts for duplicate subscribers. | State | Meaning | | ------------------- | ---------------------------------------------------------------------------- | | Pending subscribe | The client intends to subscribe and awaits venue confirmation or first data. | | Confirmed | The venue acknowledged the subscription or sent authoritative data for it. | | Pending unsubscribe | The client intends to remove the subscription and awaits confirmation. | | Trigger | Shared operation | Result | | ------------------------------------------------------ | ---------------------------------------- | --------------------------------------------------------------- | | First local subscriber | `try_mark_subscribe` or `mark_subscribe` | Record pending subscribe intent and send when required. | | Subscribe acknowledgement or authoritative first frame | `confirm_subscribe` | Move pending intent to confirmed. | | Subscribe failure | `mark_failure` | Keep subscribe intent pending for recovery. | | Last local subscriber | `mark_unsubscribe` | Remove active intent and record pending unsubscribe. | | Unsubscribe acknowledgement | `confirm_unsubscribe` | Remove pending unsubscribe without erasing a later resubscribe. | Confirm from an explicit venue acknowledgement when the protocol provides one. If acknowledgements are absent or unreliable, authoritative first data can confirm the topic. Both paths can coexist because confirmation is idempotent. Never confirm from local send success alone. On a negative subscribe result, call `mark_failure` so reconnect retains the intent. Correlate unsubscribe results separately so a late subscribe acknowledgement cannot revive removed intent and a stale unsubscribe acknowledgement cannot erase a later resubscription. Derive a stable topic key from the venue subscription arguments, but keep the original arguments when replay would otherwise require lossy parsing. On reconnect: 1. Invalidate connection and authentication state. 1. Re-establish the transport. 1. Authenticate when required. 1. Replay active and pending subscribe intent. 1. Confirm subscriptions from explicit acknowledgements or authoritative data. 1. Notify downstream consumers when they must reset protocol state. Do not replay pending unsubscriptions. Handle late or stale acknowledgements without reviving removed subscriptions. `SubscriptionState` provides these state transitions; the adapter provides the wire correlation. ### Message routing Keep the routing boundary auditable: - Decode a raw frame once. - Handle transport control, authentication, and subscription acknowledgements before domain data. - Validate the channel and product identity before choosing a parser. - Convert a venue payload to one typed adapter message or a bounded sequence of messages. - Dispatch domain events outside mutable protocol state where practical. - Preserve enough identity to correlate execution responses and deduplicate overlapping sources. The handler owns transport control, authentication, subscription acknowledgements, frame decoding, and protocol correlation. The consuming data or execution client owns domain routing and emission. Parsing may remain in the handler when it depends on handler-owned protocol state, but tracked versus external execution ownership remains a client decision. When reporting malformed frames, log the parse error separately from a sanitized, bounded payload excerpt. Never log a raw authentication frame or a frame whose schema can contain secret material. Log a peer close code and reason at the transport layer that receives it; the adapter should not duplicate the shared transport log. Dispatch module layout, intermediate enum names, context registries, venue bindings, and state containers remain adapter-specific. Prefer the smallest design that makes protocol ownership and state transitions testable; extract another component only when multiple adapters share its semantics and atomicity. ### Reconnection and shutdown Reconnection must restore protocol state, not only the socket: - Recreate or replace command paths before reporting the client active. - Reauthenticate private sessions. - Restore subscription intent and required instrument context. - Reset sequence, snapshot, or gap state when the venue requires a fresh bootstrap. - Preserve in-flight execution state needed to correlate late responses or reconciliation. Support both WebSocket control frames and venue text heartbeats when applicable. Let the shared client handle protocol control frames; keep application heartbeat messages in the venue handler. #### Reconnect ownership A handler-mode client requests a reconnect through the shared client rather than a private reconnect loop. Its `request_reconnect` returns `true` only when the call moves an active client into reconnecting. Take the reconnect handle's `request_reconnect` when the adapter must distinguish the `ReconnectRequestOutcome` variants, since an already reconnecting, disconnecting, closed, or unsupported transport each warrant a different response. Stream-mode clients own their reconnect loop, and their handles report `Unsupported`. #### Shutdown Shutdown signals tasks, asks the transport to close, and then joins or aborts owned work according to a bounded policy. Make repeated shutdown safe. Do not assume a handler `JoinHandle` has one owner when client objects can be cloned. ### Backpressure Shared WebSocket transport and adapter event paths use **unbounded** Tokio channels so receive loops do not wait for queue capacity. Preserve that convention for live event paths. Introducing a bounded channel, coalescing, dropping, or disconnect-on-full policy changes platform semantics and needs an explicit shared design, not an adapter-local change. An unbounded queue trades backpressure for memory growth. Keep receive-loop work focused, expose handler failure, and test recovery from a disconnected consumer. Never drop execution events. Market data can use snapshot and resynchronization only when its protocol contract defines that recovery. ## Task management Classify every production task by its owner before choosing its storage and shutdown path. | Ownership | Use | Required behavior | | ------------------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | Session-scoped | Stream consumers, keepalives, health polls, refresh loops, and reconnect drivers. | One session group owns the task from successful admission through disconnect or failed startup. | | Command-scoped | Work spawned by synchronous data requests or execution commands. | A separate command group owns the task without tying its outcome to the transport session. | | Explicitly singular | One task whose handle or typed result must remain in owning state for direct joining. | Store one named handle and apply the same bounded join, forced abort, and failure reporting rules. | | Handler-local | Retry futures, send workers, and child work created and joined inside one handler. | The handler drains the work before it exits and exposes failure to its owner. | | Protocol exception | Typed fan-out results, keyed timeouts, or work whose local join preserves protocol meaning. | Keep the exception local, state why a shared group would lose meaning, and test its shutdown path. | Use separate session and command groups even when both groups have the same timeout policy. A disconnect ends the session, while an accepted command can still need reconciliation or an explicit ambiguous outcome. Do not let transport shutdown silently reclassify that command result. ### Task storage and observation [`TaskHandles`](../../crates/common/src/live/task.rs) stores unit task handles without setting spawn, cancellation, generation, or join policy. Use it inside a component that defines those rules. [`TaskGroup`](../../crates/live/src/task.rs) supplies the shared live-client policy for unit-output session and command tasks. Use `TaskGroup::spawn_named` when client state must observe a grouped task's logical name, instance identity, or terminal state. Its `TaskRef` is read-only: the group remains the sole owner of cancellation and joining. Read-only observation does not make the task explicitly singular. `TaskRef::is_active` and `TaskRef::is_finished` expose the same one-way lifecycle state. Active means the task was admitted and has not reached a terminal state. A task may finish before `spawn_named` returns; neither state proves that the user future received its first poll. The same task module's `finish_task` function applies the bounded policy to an explicitly singular handle without erasing a typed result. For another explicit ownership pattern, spawn through `nautilus_common::live::get_runtime().spawn()` as described in [Async code](rust.md#async-code), then retain or locally await the returned handle. ### Spawn through a task group Synchronous client trait methods must not block an active Tokio runtime. Clone owned inputs, spawn the asynchronous operation, and return the local validation result. Register client-owned work through its `TaskGroup`. The group stores the handle before opening the task's start gate, so a concurrent shutdown either owns the task or rejects its admission. The [Tokio usage hook](../../.pre-commit-hooks/check_tokio_usage.sh) rejects `tokio::spawn` in adapter production code and requires fully qualified Tokio spawn, time, and sync paths. Keep the synchronous boundary small: ```rust fn spawn_request(&self, description: &'static str, future: F) where F: Future> + Send + 'static, { let future = async move { if let Err(e) = future.await { log::warn!("{description} failed: {e:?}"); } }; if let Err(e) = self.command_tasks.spawn(future) { log::warn!("Skipping {description} after shutdown began: {e}"); } } ``` Validate the command and clone every input before constructing the future. Do not capture a `RefCell` borrow, cache guard, clock borrow, or reference to the command in work that outlives the trait call. When a long-lived task creates children, capture a `TaskSpawner` from the owning group. Use `TaskSpawner::spawn_named` when those child tasks also need identity in shutdown failures. A spawner from an older generation cannot register work in the replacement generation. Give each task: - One owner responsible for joining or aborting it. - A stable description for failure logs. - A cancellation path. - Owned inputs that do not retain `RefCell` or engine borrows. Keep typed task results local to the component that awaits them. Do not erase a `JoinHandle` to fit a unit-output group when `T` carries order, transport, or startup evidence. ### Shut down and reopen task generations Task shutdown has separate synchronous and asynchronous phases: 1. `stop`, `reset`, and `dispose` call `begin_shutdown`. This closes admission and cancels the current generation without blocking the runtime. Use `abort` instead only when the existing synchronous contract requires immediate task cancellation. 1. `disconnect`, or the next asynchronous `connect`, closes the generation's transports in their required protocol order and calls `finish_shutdown` with bounded graceful and forced intervals. 1. `finish_shutdown` repeatedly drains tasks registered by an allowed race, reports unexpected cancellations and join failures, aborts unfinished work after the graceful interval, and awaits the forced abort within its second bound. 1. `start_generation` reopens the group only after the prior generation drains. A timeout retains the remaining handles and keeps admission closed. When `connect` fails after creating a transport or admitting a task, apply the same sequence before returning the startup error. Close every transport created by that attempt, drain both session and command work that cannot survive the failure, and include teardown failures in the returned or logged evidence. Make repeated `begin_shutdown` and `finish_shutdown` calls safe. A synchronous lifecycle method may begin teardown more than once before an asynchronous boundary finishes it. ### Never use `block_on` in trait methods Live runners call synchronous data and execution methods from within Tokio. Calling `block_on` there can panic because a runtime is already active. | Boundary | Adapter action | Reason | | ---------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------------- | | Synchronous `DataClient` or `ExecutionClient` | Clone owned inputs, spawn the operation, and return. | The live runner may already be executing the method inside Tokio. | | Async client, handler, or task method | Await the operation or select it with cancellation. | The async boundary already participates in the active runtime. | | Top-level binary or dedicated non-Tokio thread | Block only when that boundary owns the runtime lifecycle. | No ambient runtime exists when the boundary is constructed correctly. | | Test | Use `#[tokio::test]` or a test-owned runtime. | The harness owns runtime setup and avoids nested `block_on` calls. | Do not use the top-level and test exceptions to justify blocking inside a live client trait method. Redesign an ambiguous boundary as async. ### Graceful shutdown with `CancellationToken` Use `CancellationToken` when several tasks share a lifecycle. Select cancellation alongside streams, timers, or response channels. For grouped tasks, obtain child tokens from the generation group or spawner and let `begin_shutdown` cancel their parent. Obtain the replacement token only after `finish_shutdown` drains every handle and `start_generation` reopens the group. Reusing a canceled token makes replacement tasks exit immediately, while replacing it before the drain lets old work cross the reconnect boundary. ## Testing Tests prove adapter semantics at progressively wider boundaries. Store canonical valid fixtures under `test_data/` and keep network access out of ordinary unit and integration tests. Source valid payloads from official venue documentation or captured venue responses; do not hand-fabricate them. Synthetic malformed or mutated inputs remain useful for negative, property, and fuzz tests when the test marks them as such. | Boundary | Typical location | Required proof | | --------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | Pure protocol logic | `src/**` test modules | Symbols, enums, timestamps, decimals, signatures, codecs, parsers, and malformed input. | | Public Rust client boundary | `tests/` | Typed HTTP and WebSocket behavior through mock servers, event dispatch, lifecycle, and retries. | | Rust PyO3 boundary | `tests/integration/python.rs` or another feature-gated crate test | Module registration, conversion, constructors, and representative async calls. | | Public Python package | `python/tests/unit/adapters/` | Package imports, config, factories, and user-visible behavior not proved by Rust tests. | | Live venue acceptance | Adapter examples or test nodes | Authentication, subscriptions, execution, reports, recovery, and advertised limitations. | ### Rust testing Shared repository test policy uses `#[rstest]` for Rust test functions, permits `#[tokio::test]` for async tests, and rejects arrange/act/assert comments. The [testing conventions hook](../../.pre-commit-hooks/check_testing_conventions.sh) enforces these repository-wide rules. #### Fixtures and parser assertions Use exact fixture values and assert every stable output field. Distinct inputs should expose field swaps, omitted values, wrong precision, and accidental defaults. Parser and serializer tests should cover: - One realistic fixture for every supported message or instrument family. - Boundary values for decimal precision, quantities, timestamps, IDs, and enum codes. - Unknown, missing, null, and malformed fields according to the venue contract. - Round trips or canonical bytes where the protocol defines them. - Stable errors for rejected input. Keep the complete venue envelope when status fields, pagination cursors, timestamps, or nested result wrappers affect behavior. Record fixture provenance in the fixture, a nearby README, or a source manifest. Use separate real payloads for structurally distinct states such as long, short, flat, empty, and partially filled; do not mutate one happy-path fixture into every valid case. When HTTP and WebSocket tests share fixture loaders or model builders, place test-only code in a `common::testing` module rather than copying it into production modules. This pattern is optional when no test code is shared. #### Client synchronization Client tests should drive public methods through mock HTTP or WebSocket servers. Assert emitted events, requests, connection state, subscription state, retry count, and shutdown behavior. Prefer a notification owned by the test or mock when the operation exposes one. Subscribe before reading the authoritative state, then recheck it after every notification so a transition between the read and the await cannot be missed. When no suitable signal exists, use [`wait_until_async`](../../crates/common/src/testing.rs). A short sleep is valid when the time window itself is under test, but it should not mask a missing synchronization point. ### Functional and integration testing Exercise each public boundary with both successful and adverse protocol evidence: | Surface | Successful evidence | Failure and recovery evidence | | ---------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | HTTP client | Exact method, path, query or body, authentication, and typed response. | Missing credentials, venue errors, malformed bodies, retry classification, and pagination termination. | | WebSocket client | Connection, authentication, heartbeat, subscription acknowledgement, and typed routing. | Authentication failure, malformed frames, stale acknowledgements, disconnect, replay, and shutdown. | | Data client | Requests and subscriptions produce complete domain events with correct identity and time. | Freshness, filtering, malformed input, stream gaps, unsubscribe, and reconnect behavior. | | Execution client | Commands produce ordered events, account state, and reconciliation reports. | Local denial, definitive rejection, unknown outcome, duplicates, partial batches, and startup recovery. | Mock transports should expose enough state to assert requests, connection count, authentication, subscriptions, and emitted events. Wait for those observable conditions instead of sleeping. Assert both sides of the boundary: the exact venue request and the resulting Nautilus event or report. Data tests cover each advertised request and subscription, plus: - Instrument identity, precision, and freshness. - Snapshot and incremental order-book boundaries. - Multiple symbols or product families sharing a connection. - Acknowledgement, rejection, unsubscribe, reconnect, and resubscribe behavior. - Malformed or unknown messages without loss of subsequent valid data. Execution tests cover each advertised command and report, plus: - Local denial before submission. - Definitive venue rejection. - Diagnostic context and the exact clean strategy-facing reason for each changed rejection path. - Unknown transport outcomes that remain reconcilable. - An ambiguous attempt followed by a definitive-looking response. - Missing, malformed, and mismatched returned venue identifiers. - Structured error fields and bounded raw-body fallbacks: empty, plain text, malformed structured data, markup, invalid UTF-8, and oversized input. - Equivalent HTTP, WebSocket, polling, and reconciliation evidence producing the same reason and classification. - Structured post-only evidence or an exact text classifier, including close non-matching messages. - Partial and per-order batch results. - Duplicate or out-of-order stream updates. - Account state, open orders, fills, positions, and startup reconciliation. - One fixed cutoff across bounded order and fill queries, including records on the boundary. - Complete and incomplete mass statuses for each independently failing report source. - Position snapshot coverage after reconnect, skipped rows, explicit flats, and absent instruments. - Exact order recovery without position or portfolio effects when bounded history is incomplete or ambiguous. - Idempotent stop, reset, and disposal. Keep adapter tests focused on adapter behavior. The [data testing specification](spec_data_testing.md) and [execution testing specification](spec_exec_testing.md) define the shared scenario catalogs and skip rules; link to them instead of copying partial lists into an adapter README. ### Acceptance testing Run acceptance tests only after deterministic tests pass. Use testnet or a controlled account and record: - Venue environment and product. - Supported and skipped specification cases. - Order types and flags exercised. - Reconnect or recovery cases exercised. - Venue restrictions, rate limits, and known gaps. Acceptance tests must verify events and venue state, not only the absence of errors. Clean up open orders and positions according to the test plan, and never infer production support from one happy path. Provide the applicable tester entry points: - Rust: `crates/adapters//examples/node_data_tester.rs` and `node_exec_tester.rs`, with product subdirectories when protocols split by product. - Python: `examples/live//data_tester.py` and `exec_tester.py`, using `LiveNode` and the Rust config and factory classes. Python tester scripts run out of the box: settings live in module-level constants at the top of the file, and running the script connects and starts immediately without CLI flags. Execution testers place real orders by default, so state this plainly in a warning at the top of the module and set `dry_run=False` explicitly in the `ExecTesterConfig` to advertise the dry-run option. Rust tester controls vary; inspect them before running. ### Python boundary testing For Python-exposed adapters, test the Rust module before testing broad Python workflows. Verify: - The module imports at the runtime path. - Stub metadata points to the public adapter package. - Config conversion preserves optional values and rejects unknown fields. - Factories downcast config and construct the correct Rust client. - Representative async client methods convert success and error results. Use `instrument_any_to_pyobject` and `pyobject_to_instrument_any` at Python instrument boundaries to preserve the concrete instrument variant in both directions. Regenerate stubs with `make py-stubs` after changing exported Rust types or signatures. The [generated drift check](../../scripts/ci/check-generated-drift.bash) verifies that generator inputs and committed `.pyi` output agree. ## Performance and robustness Add these suites late, after functional, integration, and acceptance work establishes correct behavior. They deepen assurance for confirmed hot paths and untrusted venue input; they do not replace conformance tests. ### Canonical benchmarks Use Criterion for a deep performance pass on production boundaries that measurements identify as important. The Lighter and Derive suites provide the reference structure: | Suite | Canonical boundary | Reference | | ------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `benches/data.rs` | Raw venue frame or payload through decoding, parsing, cache lookup where required, and Nautilus domain construction. | [Lighter data](../../crates/adapters/lighter/benches/data.rs), [Derive data](../../crates/adapters/derive/benches/data.rs) | | `benches/exec.rs` | Order command through serialization and signing; where applicable, inbound execution payload through event dispatch. | [Lighter execution](../../crates/adapters/lighter/benches/exec.rs), [Derive execution](../../crates/adapters/derive/benches/exec.rs) | | `benches/micros.rs` | Decode-only, parse-only, and focused component costs that localize a regression found at a pipeline boundary. | [Lighter micros](../../crates/adapters/lighter/benches/micros.rs), [Derive micros](../../crates/adapters/derive/benches/micros.rs) | Put shared realistic instruments, payloads, signer state, and other fixtures in `benches/common/`. Construct stable setup, allocation, and state outside the timed region when production does not pay that cost per operation. Include setup when it is part of the real hot path. Measure representative end-to-end pipelines first. Add diagnostic components to explain a regression, not to inflate the suite. Set throughput when bytes, messages, orders, or another unit clarifies operational capacity. Add venue-specific suites for confirmed hot paths such as signing, hashing, binary codecs, or authentication. Lighter has focused cryptographic suites, and Derive has a signing suite. Do not require a category that the adapter does not use. Recorded Lighter signing numbers and the official Go comparison live in the [Lighter adapter benchmarks](../../crates/adapters/lighter/benches/BENCHMARKS.md). Follow the repository [benchmarking guide](../../BENCHMARKING.md) for tool choice, baselines, noise control, and result reporting. Use the [Criterion practitioner guide](benchmarking.md#writing-criterion-benchmarks) for benchmark structure and local commands. ### Fuzz testing Coverage-guided fuzzing adds assurance where arbitrary venue bytes or values cross a trust boundary. Prioritize: - Raw WebSocket or binary frame decoding. - Decimal, timestamp, symbol, and enum normalization. - Signing payload and canonical encoding. - Hashes and binary codecs. - Nonce or sequence allocation. - Other venue-specific parsers and encoders that accept untrusted input. Seed parser and decoder corpora with representative payloads from `test_data/` when they improve coverage. Keep harnesses below live network and runtime layers unless the target specifically needs one of those boundaries. Panic freedom is only the baseline. Assert deterministic properties such as: - Encode/decode round trips. - Canonical encoding and idempotence. - Length, precision, range, and allocation bounds. - Deterministic hashes and signatures. - Monotonic nonce or sequence models. - Agreement with an independent implementation. - Stable rejection for invalid input. Use differential fuzzing when a sufficiently independent reference exists. Lighter's scalar multiplication target and Derive's nonce model show how to compare implementations without putting network state in the harness. Canonical adapter wiring is: | Surface | Required wiring | Enforcement or use | | ---------------------------------------------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | Adapter `[features]` | `fuzz = ["nautilus-live/fuzz"]` | Enables the shared fuzz support without changing normal builds. | | Adapter `[package.metadata]` | `cargo-fuzz = true` | Lets `cargo fuzz` treat the adapter manifest as a fuzz package. | | Adapter `[[bin]]` | One entry per target with the `fuzz` feature and `test`, `doc`, and `bench` false. | Registers discoverable binaries without adding them to ordinary test runs. | | `fuzz/fuzz_targets/` | Focused targets below live network and runtime layers. | Keeps arbitrary input at the parser, codec, normalization, or model boundary. | | [`scripts/fuzz-adapter.sh`](../../scripts/fuzz-adapter.sh) | Adapter target discovery and repeated time-sliced runs. | Uses the registered binaries and preserves corpus and artifact locations. | Adapter crates must not depend directly on `libfuzzer-sys`; the [Cargo conventions hook](../../.pre-commit-hooks/check_cargo_conventions.sh) enforces the shared feature path. Use the focused [Lighter fuzz README][lighter-fuzz] and [Derive fuzz README][derive-fuzz] for setup, corpus, artifact, and target commands instead of copying every invocation here. ## Documentation Create or update `docs/integrations/.md` with: - Supported products, environments, data types, order types, and reports. - Authentication and environment variables. - Config examples and factory registration. - Venue limits, reconciliation behavior, and known gaps. - Testnet or sandbox differences. - Links to venue protocol documentation used by the implementation. Keep capability claims testable and name legitimate exceptions. Link to shared [configuration](../concepts/configuration.md), [benchmarking](../../BENCHMARKING.md), and testing guides instead of copying their policy. Follow the repository [documentation guide](docs.md) and [Markdown style guide](markdown_style.md). Change generator inputs and regenerate generated output. ## Testing spec references Use these shared specifications to plan and report adapter conformance: - [Data client testing specification](spec_data_testing.md). - [Execution client testing specification](spec_exec_testing.md). [binance]: ../../crates/adapters/binance/src/lib.rs [bybit]: ../../crates/adapters/bybit/src/lib.rs [derive]: ../../crates/adapters/derive/src/lib.rs [derive-fuzz]: ../../crates/adapters/derive/fuzz/README.md [kraken]: ../../crates/adapters/kraken/src/lib.rs [lighter]: ../../crates/adapters/lighter/src/lib.rs [lighter-fuzz]: ../../crates/adapters/lighter/fuzz/README.md [okx]: ../../crates/adapters/okx/src/lib.rs # Benchmarking Source: https://nautilustrader.io/docs/latest/developer_guide/benchmarking/ Use this guide to write, run, and profile NautilusTrader benchmarks. It contains benchmark layout, examples, local commands, and the measurement procedure for published results. For benchmark scope, evidence requirements, and CI policy, see [`/BENCHMARKING.md`](../../BENCHMARKING.md) at the repository root. --- ## Tooling overview Select a tool based on the work and result: | Tool | What it measures | Use it for | | --------------------------------------------------------- | ----------------------------------------- | --------------------------------------------------------- | | [Criterion](https://docs.rs/criterion/latest/criterion/) | Wall-clock time with confidence intervals | Operations above roughly 100 ns and elapsed time | | [iai](https://docs.rs/iai/latest/iai/) | Retired CPU instructions under Cachegrind | Small, deterministic operations and change detection | | [CodSpeed](https://codspeed.io/docs/instruments/cpu) | Simulated CPU cost and cache behavior | Stable pull request comparisons of deterministic CPU work | | [flamegraph](https://github.com/flamegraph-rs/flamegraph) | Sampled call-stack profile | Locating work inside a representative slow path | Criterion reports user-visible elapsed time. iai produces stable counts for the same binary, toolchain, and inputs without requiring host noise controls. Compare iai results only under the same code generation assumptions, and use Criterion when elapsed time is the required result. --- ## Directory layout Each crate keeps its benchmarks in a local `benches/` folder: ```text crates// └── benches/ ├── foo_criterion.rs └── foo_iai.rs ``` Register each benchmark explicitly in the crate's `Cargo.toml` so `cargo bench` discovers it: ```toml [[bench]] name = "foo_criterion" path = "benches/foo_criterion.rs" harness = false [[bench]] name = "foo_iai" path = "benches/foo_iai.rs" harness = false ``` To opt into the nightly CI performance workflow, register the benchmark and add its crate to `CI_BENCH_CRATES` in the workspace `Makefile` when the list does not already include it. Add a deterministic Criterion target to `CODSPEED_BENCH_TARGETS` when CPU simulation preserves what the benchmark intends to measure. Do not add iai, Criterion's `iter_custom` or `with_filter` APIs, OS-dependent work, or concurrent wall-clock benchmarks to the CodSpeed subset. --- ## Writing Criterion benchmarks 1. **Set up outside the timing loop.** All work that doesn't change between iterations belongs in the surrounding code or in `iter_batched_ref`'s setup closure, not in the body passed to `iter`. 2. **Wrap inputs in `black_box`** so the optimizer doesn't fold them away. 3. **Use `iter_batched_ref` for mutating benches.** It excludes input `Drop` from the timed region, which otherwise dominates the measurement on benches that own large structures. 4. **Add `Throughput::Elements(n)`** to size-parameterized groups so Criterion reports per-element throughput. 5. **Comment intent.** State what the benchmark is measuring (the hot path, the worst case, the cache-cold case) so a future reader understands what regressing it would mean. ```rust use std::hint::black_box; use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; const SIZES: &[usize] = &[10, 100, 1_000]; fn bench_my_op(c: &mut Criterion) { let mut group = c.benchmark_group("module/my_op"); for &n in SIZES { group.throughput(Throughput::Elements(n as u64)); group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| { b.iter_batched_ref( || populate(n), |state| state.run(black_box(n)), BatchSize::SmallInput, ); }); } group.finish(); } criterion_group!(benches, bench_my_op); criterion_main!(benches); ``` --- ## Writing iai benchmarks `iai` requires functions that take no parameters. Use it for small, pure operations so the measured instruction count stays focused on the intended work. ```rust use std::hint::black_box; fn bench_add() -> i64 { let a = black_box(123); let b = black_box(456); a + b } iai::main!(bench_add); ``` Allocations, randomness, and system calls add their own instructions to the result. Keep variable setup outside the measured function and compare counts produced by the same toolchain and target. --- ## Running benches locally | Goal | Command | | ---------------------------------- | --------------------------------------------------------------------- | | All benches in one crate | `cargo bench -p nautilus-execution` | | One core bench module | `cargo bench -p nautilus-execution --bench matching_core` | | One engine bench module | `cargo bench -p nautilus-execution --bench matching_engine` | | One core benchmark name pattern | `cargo bench -p nautilus-execution --bench matching_core -- iterate` | | One engine benchmark name pattern | `cargo bench -p nautilus-execution --bench matching_engine -- submit` | | Quick smoke run (low sample count) | `cargo bench ... -- --quick` | | All nightly registered benches | `make cargo-ci-benches` | | Build the CodSpeed subset | `make cargo-codspeed-build` | | Check the built CodSpeed subset | `make cargo-codspeed-run` | Criterion writes HTML reports to `target/criterion/`. Open `target/criterion/report/index.html`. The report includes per-bench violin plots, confidence intervals, and comparisons against the previous run's saved baseline. `make install-tools` installs the pinned `cargo-codspeed` version. A local CodSpeed run checks that the selected benchmark targets build and register, but it does not upload measurements. The `codspeed-benchmarks` job in `.github/workflows/performance.yml` measures and uploads the results. ### Canonical backtest workloads The canonical backtest cases use the first 10,000 rows of the checked-in `test_data/btc-perp-20211231-20220201_1m.csv` file. A shared fixture drives replay-only, scheduled market-order, passive limit-order, and bar-EMA scenarios. The correctness test and both timed paths use the same fixture loader and exact result fingerprints. Run the semantic check first: ```bash CARGO_BUILD_JOBS=16 cargo test --locked -p nautilus-backtest \ --test integration canonical_backtest_workloads:: ``` Then run Criterion in test mode to confirm that every affected benchmark case executes without collecting samples: ```bash CARGO_BUILD_JOBS=16 cargo bench --locked -p nautilus-backtest \ --bench engine -- canonical --test ``` The `run_preloaded` cases load the CSV and build the engine outside the returned `iter_custom` duration. The `load_build_run` cases include CSV loading, engine setup, data registration, and `BacktestEngine::run`. Both exclude result projection and fingerprint verification from the reported duration, while still checking the result after every measured iteration. See [`crates/backtest/benches/BENCHMARKS.md`](../../crates/backtest/benches/BENCHMARKS.md) for the published baseline, measurement record, and current profile target. The [v2 migration guide](../../MIGRATION_V2.md#compare-backtest-performance) contains the cross-version backtest comparison procedure. --- ## Measure Criterion for publication Use the `bench-lto` profile for Criterion results that will be reported or published. The profile inherits from `release`, preserves full debug symbols, enables fat LTO, and uses one code generation unit. The default `bench` profile keeps full debug symbols without LTO and is better suited to local iteration. 1. Quiesce the machine. On Linux, set the CPU governor to `performance` when you administer the host and can restore its prior state: ```bash sudo cpupower frequency-set -g performance ``` 1. On Linux, disable ASLR for the benchmark process and run the selected benchmark with `bench-lto`: ```bash setarch "$(uname -m)" -R cargo bench --profile bench-lto -p --bench ``` 1. Run multiple full sessions and report whether each case uses its best or median result. 1. Record the CPU model, kernel or operating system, Rust toolchain, and build profile with the results: ```text Hardware: , Toolchain: Profile: bench-lto (release + lto = "fat" + codegen-units = 1, debug = full) ``` For deeper analysis, control hyper-threading and dynamic frequency scaling in firmware. Published results must record those controls when they differ from the normal host state. iai runs under Cachegrind's virtual CPU model, so host quiescence, frequency scaling, and ASLR do not affect its instruction counts. Run iai without the Criterion noise controls. --- ## Generating a flamegraph `cargo-flamegraph` produces a sampled call-stack profile for one bench. Use it when a benchmark regresses and the responsible inner call is unclear. 1. Install once per machine: ```bash cargo install flamegraph ``` 2. Run a specific bench with the `bench` profile: ```bash cargo flamegraph --bench matching -p nautilus-common --profile bench ``` 3. Open `flamegraph.svg` in a browser and zoom into hot paths. ### Linux `perf` must be available. On Debian/Ubuntu: ```bash sudo apt install linux-tools-common linux-tools-$(uname -r) ``` If `perf_event_paranoid` blocks the run: ```bash sudo sh -c 'echo 1 > /proc/sys/kernel/perf_event_paranoid' ``` A value of `1` is usually enough. Set it back to `2` (default) afterwards or persist via `/etc/sysctl.conf`. ### macOS `DTrace` requires root, so `cargo flamegraph` must be run with `sudo`. :::warning Running with `sudo` creates files in `target/` owned by root, causing permission errors with subsequent `cargo` commands. You may need to remove root-owned files manually or run `sudo cargo clean`. ::: ```bash sudo cargo flamegraph --bench matching -p nautilus-common --profile bench ``` The `bench` profile keeps full debug symbols, so flamegraphs render with readable function names without bloating production binaries (which still use `panic = "abort"` and are built via `[profile.release]`). > **Note** Benchmark binaries are compiled with the custom `[profile.bench]` > defined in the workspace `Cargo.toml`. That profile inherits from > `release` and sets `debug = "full"`, preserving full optimization *and* > debug symbols so tools like `cargo flamegraph` or `perf` produce > human-readable stack traces. --- ## Templates Starter files live in [`docs/dev_templates/`](../dev_templates/): - **Criterion**: [`criterion_template.rs`](../dev_templates/criterion_template.rs) - **iai**: [`iai_template.rs`](../dev_templates/iai_template.rs) Copy the template into the target crate's `benches/`, adjust imports and group names, register in `Cargo.toml`, and start measuring. # Callback Dispatch Contract Source: https://nautilustrader.io/docs/latest/developer_guide/callback_dispatch/ This page defines the ownership, ordering, and progress requirements for queued actor and strategy callbacks. The [design principles](design_principles.md#queued-callback-dispatch-requirements) explain the policy. :::info These requirements are design constraints for queued actor and strategy callback delivery, not guarantees of the existing synchronous dispatch paths. Support for synchronous message-bus reentry does not activate queued actor or strategy callbacks. ::: ## Implementation limits | Area | Implemented behavior | Limit | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Actor delivery | Private primitives support ordered, owned callback delivery. | **Queued actor and strategy delivery is inactive.** Existing synchronous paths do not gain these guarantees. | | [Root propagation](#synchronous-commands) | Retained work and synchronous data and trading commands preserve causal roots. | Live channels and locally emitted data and execution events do not have complete root propagation. | | [Drain safety](#draining-and-progress) | Explicit drains respect slot budgets and checked access; a busy head blocks later delivery. | Callers must end enclosing mutable borrows. Automatic safe drains and detection of a head that cannot progress require runtime integration. | | [Progress budgets](#callback-roots-and-budgets) | Completed callbacks consume a per-root delivery budget. | Commands do not consume that budget. Command-only loops and the duration of an individual callback are not bounded. | | [Memory accounting](#storage-limits) | Private limits cover retained units and known callback storage. | Command payloads, command-queue capacity, and the listed opaque storage are excluded. This is not a total-process memory cap; limits have no user configuration. | | [Failure handling](#failure-cleanup) | Contexts restore on unwind; retained roots block premature teardown. | Fatal callback errors halt the dispatcher across roots. Command-handler panics propagate and discard pending children and the unprocessed collected batch; completed effects are not rolled back. | | [Access and backends](#backend-compatibility) | Private allocation guards reject overlapping checked access. | Unchecked access and enclosing engine/cache borrows remain outside those guards. Native, Python, and dynamic-backend parity is not established. | | [Observable state](#maintenance-and-observable-state) | Event payloads describe their event; cache mutations and facade effects stay synchronous. | Callbacks observe current cache state, not an event-time snapshot. Queued delivery does not defer or undo facade effects. | ## Ordering and reentrancy :::tip What is reentrancy? Reentrancy occurs when code is entered again before an earlier call to that code returns. For example, a message-bus subscriber publishes another message, causing nested delivery before the original publication finishes. This can happen on one thread; it does not require parallel execution. ::: Within one runtime thread, canonical actor and strategy callbacks must preserve **publication order** across components and topics. The rule applies equally to idle and active components. A nested publication must not overtake an earlier publication's pending deliveries, including all recipients of the earlier publication. Independent nodes have no shared global ordering guarantee. Callbacks require **exclusive access** to their component and a delivery boundary at which enclosing mutable runtime borrows have ended. Native and Python components must follow the same ordering contract. Raw Python [topic messaging](../concepts/message_bus.md) delivers the original object synchronously to subscribed callables, including during nested publication. It shares the runtime bus and topic space with canonical custom-data subscriptions. The queued ordering requirement applies to canonical callbacks; raw callables remain synchronous. A nested publication from a raw callable must still preserve the publication order of pending canonical deliveries. ## Maintenance and observable state Component maintenance runs with each queued event, before that event's author callbacks. This includes indicator updates, timer cleanup, and contingent-order handling. Maintenance retains its applicable lifecycle rules when author callbacks are suppressed. Events emitted by maintenance enter the same ordered dispatch mechanism. Engine [cache](../concepts/architecture.md#cache) mutations and direct facade effects remain synchronous. Callbacks observe **current cache state**; ordered delivery does not provide an event-time cache snapshot. The immutable event payload records the event, while the cache may already reflect later changes. Keeping indicator updates with event delivery preserves their ordering relative to the corresponding callbacks. Author callbacks require eligibility at **both event arrival and delivery**. Stop, reset, or retirement must not carry old callbacks into a new registration or [lifecycle](../concepts/actors.md#lifecycle) generation. ## Bounded progress A **drain** processes queued callbacks in order, attempting delivery when permitted. Pending callbacks participate in the runtime's drain condition. Live execution uses bounded drain batches and yields between them. Backtests finish pending work before advancing simulated time. Runaway callback chains produce an explicit fault. Queue overflow records a fatal error and halts execution at a safe boundary. It must neither silently discard callbacks nor interrupt an operation midway through its synchronous effects. Already completed effects are not rolled back by callback dispatch. ## Private dispatch primitives The actor module contains private access, admission, publication, invocation, and storage primitives. The synchronous data and trading command queues preserve [callback roots](#callback-roots-and-budgets). Production actor lookups, component access, and message-bus routes do not use queued callback delivery. Activating that delivery requires explicit native and Python runtime boundaries; the primitives alone do not establish runtime ownership safety or native, direct, and dynamic callback parity. ### Publication and admission A publication scope reserves an **ordinal** (a sequence number) before synchronous subscribers run. Callback reservations sort by publication ordinal and then admission ordinal, so nested publications follow every pending recipient of their enclosing publication. Scopes nest and drop in stack order. Dropping a scope restores publication state without delivering callbacks. A publication unwind latches a fatal error. The message bus does not install these scopes automatically. Admission reserves count, known storage, and a queue slot **before the caller constructs owned captures**. A successful reservation accepts its capture even if another operation subsequently latches a failure. An unfinished reservation blocks later deliveries and teardown; cancellation leaves a slot that an explicit drain or teardown releases. Actor reservations retain registration identity, so replacement or re-registration of the same allocation cancels stale delivery. ### Draining and progress A drain processes at most its supplied slot budget, including cancelled slots. The queue has a **busy head** when its first callback cannot acquire exclusive access to its component. This blocks later delivery. Drains do no work during publication, recursive draining, teardown, or checked allocation access. :::warning Guard destruction only releases access; it does not drain callbacks. The caller must also end enclosing engine, cache, and other untracked borrows before draining. ::: ### Callback roots and budgets A **root** owns the delivery budget for an incoming publication or reservation and all callback work it causes. Publication order does not depend on which root owns the work. - Nested publications and reservations inherit the active root. - A publication with no active root starts one on its first callback admission or command send. An empty scope allocates none. - Outside a publication scope, each admission starts a separate root unless a root is already active. Each completed delivery counts once against its root's budget. Busy attempts and cancelled slots do not count, though cancelled slots still consume the drain's slot budget. At the chain limit, dispatch latches a fatal runaway error before running another ready callback. The blocked callback stays queued for explicit teardown. **A fatal error halts the whole dispatcher**, even though roots have separate budgets. ### Retained work and cleanup Queued slots, active scopes, and retained invocation work keep their root alive. **An empty queue does not reset a surviving root's budget.** Retained storage captures the active root, or starts one when none is active. Growing that storage preserves its root. Resuming retained work through `with_chain` makes its root active for the call. Return or unwind restores the enclosing root. Invocation batches use this mechanism for each retained value; resuming work does not itself count as a delivery. A delivered capture's destructor runs with the delivery's root active, including during unwind. Uninvoked batch values also restore their root when destroyed. The root and its storage charge release when the last owner drops. Cancellation, failed batches, and explicit teardown use the same ownership cleanup as other retained storage. Retained work prevents teardown until released. ### Synchronous commands Commands capture any root active when they are sent, including during callback delivery and retained-work resumption. A send inside a publication or command-processing scope starts that scope's root if needed. A send with no active root outside those scopes carries no root; processing then creates an independent root when it first admits callback work or sends a nested command. Commands in the same drain batch do not share a root merely because they run together. Processing and destruction restore the enclosing root on return or unwind. A command drain processes the batch collected at entry, in order. Commands enqueued through a synchronous sender by its handlers remain queued for a subsequent drain. Trading handlers can also capture deferred children: these run depth-first, in capture order, before the next command in the collected batch. Each child captures the root active at capture time, including a temporary nested context, rather than using the parent's context after its handler returns. Direct endpoint routing stays unchanged. Queue and dispatch-frame borrows end before handlers run or abandoned commands are destroyed. Command processing does not count as callback delivery and does not automatically drain callbacks. If a handler panics, the panic propagates. Unprocessed commands in the collected batch and pending deferred children are destroyed under their own contexts; newly enqueued commands remain queued. Captured roots keep callback accounting alive and prevent explicit dispatcher teardown until those commands release them. Abandoned commands also release their captured ownership at thread teardown. Callback storage limits do not reject command sends. If a send needs a root and its allocation exceeds those limits, it latches callback overflow while the command is still queued. Command entries do not consume the callback retained-unit limit, and command payloads and queue capacity are excluded from known callback storage. Each root allocation is charged once. **This does not bound command-queue memory or loops that generate only commands.** ### Runtime integration The [root propagation tests](../../crates/common/src/actor/dispatch.rs) cover retained continuations and synchronous data and trading commands, including deferred trading children. Before queued callback activation, runtime integration must: - Extend root propagation to live channels and locally emitted data and execution events. - Preserve independent ingress boundaries when reusing long-lived storage, so unrelated events do not accumulate against one root's budget. - Provide safe drain boundaries. - Detect a busy head that cannot make progress. Public trading messages and their direct `dispatch()` path carry no callback context across threads. The synchronous queue owns its contexts privately; live-channel propagation remains outside this scope. These primitives do not establish runtime integration or backend parity. ### Storage limits The private limits are: - **Retained units**: At most 65,536. Invocation captures and batch-capacity reservations share the count with queued callbacks, so the count can conservatively exceed the number of callbacks. - **Known storage**: At most 64 MiB. - **Callback chain**: At most 1,048,576 completed deliveries per root. These limits are internal and expose no user configuration. Known storage includes callback values, queue slots and visible queue capacity, chain allocations, invocation vector capacity, and the caller-supplied heap charge. Payload measurements include accessible string and vector capacities, recursive JSON contents, metadata entries, order-event collections, visible book entries, option-chain entries, and, with the `defi` feature, directly owned blockchain strings. Shared payload storage may be charged more than once. Data-type names, topics, and identifiers expose string slices, so their charges cover lengths and exclude inaccessible spare capacity. Charges remain held while delivery is in flight and while batches retain captures. :::warning This accounting is not a bound on total process memory. It excludes allocator bookkeeping, collection internals whose capacity is not exposed by these measurements, intern-pool storage, registered actor allocations, opaque custom-data payloads, Python object graphs, and dynamic-module storage. ::: Callers must supply the known heap charge for their capture type; the generic reservation cannot inspect arbitrary owned fields. ### Failure cleanup Overflow latches the first fatal failure and rejects further reservations without constructing captures. Synchronous facade effects can finish before the runtime reports the failure at a safe boundary. Queued slots remain owned for explicit teardown. Failed invocation batches release their captures after preparation exits and its guards release. Teardown must follow reporting of the fatal error and must run after active publication, reservation, invocation, chain, drain, and access scopes end. It detaches storage before releasing captures and rejects destructor-driven admission while clearing. Invocation preparation keeps the batch owner outside the call that acquires guards. On rejection or unwind, previously transferred captures survive until those guards release. The caller reserves before constructing each capture and acquires preparation guards inside that call. The primitives cannot control arbitrary locals that author code constructs or explicitly destroys while holding a borrow. Destructors must not panic during an existing unwind. ### Backend compatibility Direct and dynamic backends must preserve the same callback ordering, exclusive component access, and lifecycle eligibility requirements. Facade effects remain synchronous. The [plug-in boundary rules](plugins.md#boundary-rules) apply to all values crossing a dynamic-library boundary; callback dispatch does not implicitly transfer allocation ownership across that boundary. Admission tickets and cleanup scopes remain framework machinery; author fields, callback signatures, and canonical facade APIs do not expose them. # Coding Standards Source: https://nautilustrader.io/docs/latest/developer_guide/coding_standards/ ## Code Style The current codebase can be used as a guide for formatting conventions. Additional guidelines are provided below. ### Universal formatting rules The following applies to **all** source files (Rust, Python, shell, etc.): - Use **spaces only**, never hard tab characters. - Lines should generally stay below **100 characters**; wrap thoughtfully when necessary. - Use American English spelling (`color`, `serialize`, `behavior`), enforced by `.typos.toml`. Preserve external API spellings through exact identifier exceptions; exclude verbatim data and generated files. Run `prek run typos --all-files` to check the repository. ### Shell scripts Bash is the default shell for repository scripts. Use POSIX `sh` only when a caller cannot rely on Bash being installed. See [Shell](shell.md) for script selection, extensions, portability, structure, testing, formatting, and linting requirements. ### Comment conventions 1. Generally leave **one blank line above** every comment block or docstring so it is visually separated from code. 2. Use *sentence case* - capitalize the first letter, keep the rest lowercase unless proper nouns or acronyms. 3. Do not use double spaces after periods. 4. **Single-line comments** *must not* end with a period *unless* the line ends with a URL or inline Markdown link - in those cases leave the punctuation exactly as the link requires. 5. **Multi-line comments** should separate sentences with commas (not period-per-line). The final line *should* end with a period. 6. Keep comments concise; favor clarity and only explain the non-obvious - *less is more*. 7. Avoid emoji symbols in text. ### Doc comment mood **Rust** doc comments should be written in the **indicative mood** - e.g. *"Returns a cached client."* This convention aligns with the prevailing style of the Rust ecosystem and makes generated documentation feel natural to end-users. ### Terminology and phrasing 1. **Error messages**: Avoid using ", got" in error messages. Use more descriptive alternatives like ", was", ", received", or ", found" depending on context. - Bad: `"Expected string, got {type(value)}"` - Good: `"Expected string, was {type(value)}"` 2. **Spelling**: Use "hardcoded" (single word) rather than "hard-coded" or "hard coded" - this is the more modern and accepted spelling. 3. **Error variable naming**: Use single-letter `e` for caught errors/exceptions: - Rust: `Err(e)` not `Err(err)` or `Err(error)`, and `|e|` not `|err|` in closures - Python: `except SomeError as e:` not `as err:` or `as error:` ### Naming conventions 1. **Internal fields**: Abbreviations are acceptable for private/internal fields (e.g., `_price_prec`, `_size_prec`) to keep hot-path code concise. 2. **User-facing API**: Use full, descriptive names for public properties, function parameters, return types, and metric names/labels (e.g., `price_precision`, `size_precision`). This prevents abbreviated terminology from leaking into dashboards or alerts. 3. **Error messages and logs**: Use full words for clarity (e.g., "price precision" not "price prec"). The user should never see abbreviated terminology. 4. **Execution terminology**: Use `Execution` in public, project-owned PascalCase type names, such as `BinanceExecutionClientConfig`. Internal implementation types may retain established `Exec` names. Also reserve `Exec` for the `ExecAlgorithmId` and `ExecTester` families, established `exec_*` names, and venue or protocol terms such as `BitmexExecType`. Name protocol-specific wire models after the venue concept, such as `HyperliquidExchangeAction`. Preserve established public names, historical release entries, and source names in migration tables. 5. **Runtime qualifiers**: Use `Live` when a type selects or configures real-time runtime semantics, such as `LiveNode` versus `BacktestNode`, `LiveClock` versus `TestClock`, and the `LiveDataEngineConfig`, `LiveRiskEngineConfig`, and `LiveExecutionEngineConfig` family versus reusable core engine configs. Omit `Live` from the ordinary adapter client family because a connected client is the default. Qualify alternate implementations by their behavior, such as `SandboxExecutionClient` or `DatabentoHistoricalClient`. An explicit live/historical protocol pair may retain `Live` to distinguish the two implementations. 6. **Adapter factory configs**: Name the data and execution inputs `DataClientConfig` and `ExecutionClientConfig`. Factories consume these client configs directly rather than a separate factory config wrapper. `LiveNodeConfig` owns `trader_id`; venue-specific `account_id` values belong on execution client configs. #### Data loading APIs Use free functions for stateless data ingestion. Use a class only when instances retain reusable configuration, caches, workers, iteration state, or an open resource across calls. Do not use a zero-state class solely to group static or class methods. Follow the semantic distinction in the [Polars I/O API](https://docs.pola.rs/api/python/stable/reference/io.html), adapted to the established Nautilus `load_*` vocabulary: - `load__` eagerly reads, normalizes, and materializes a complete result. - `scan__` creates a lazy query or deferred execution plan. - `stream__` incrementally yields records or batches. - `write_` eagerly writes an in-memory result. - `sink_` writes through a lazy or streaming execution path. For ingestion, order names from general to specific: verb, source, logical data, then an optional representation. Include the representation only when real sibling formats exist. For example, `load_binance_order_book_deltas` is preferable to a stateless `BinanceOrderBookDeltaDataLoader.load` class or a generic `load_binance_data` function. #### Adapter package facades Each package under `python/nautilus_trader/adapters/` is a thin facade over the private `_libnautilus` extension. Every adapter `__init__.py` declares a deterministic `__all__` that is the single source of truth for its public API; `python/generate_stubs.py` copies that list into the matching `.pyi` so runtime and stub exports agree exactly. A venue adapter exposes its canonical identity constants plus the supported public surface: - ``, `_CLIENT_ID`, `_VENUE`, registered from Rust via `m.add` in the adapter's `python/mod.rs` - data types, `*Config`, `*Factory`, user-facing enums (such as `*Environment` and `*ProductType`) - stateless loaders (`load_*`, `stream_*`, `convert_*`) and intentional utilities (`decode_*`, `get_*_arrow_schema_map`) Keep the facade thin. Never add raw HTTP or WebSocket clients, wire models, endpoint URL resolvers (`get_*_url`, `*_HTTP_URL`), caches, or other internals to `__all__` merely for structural parity. Data providers (such as `databento` and `tardis`), the `blockchain` data client, the `sandbox` execution client, and the multi-venue `interactive_brokers` broker omit venue constants because the constants would be meaningless for them. Order `__all__` entries so the `RUF022` pre-commit gate owns sort order; do not hand-order the list. ### Formatting 1. For longer lines of code, and when passing more than a couple of arguments, you should take a new line which aligns at the next logical indent (rather than attempting a hanging 'vanity' alignment off an opening parenthesis). This practice conserves space to the right, keeps important code more central in view, and survives function/method name changes. 2. The closing parenthesis should be located on a new line, aligned at the logical indent. 3. Multiple hanging parameters or arguments should end with a trailing comma: ```python long_method_with_many_params( some_arg1, some_arg2, some_arg3, # <-- trailing comma ) ``` ## Commit messages Commit messages use a capitalized, imperative subject naming the affected surface, optionally followed by a body explaining the change. ### Subject line - Open with a capitalized imperative verb, so the subject describes what the commit does when applied. `Add`, `Fix`, `Improve`, `Refine`, `Update`, `Remove`, `Refactor`, and `Standardize` cover most of the history. - Name the affected surface (crate, adapter, subsystem, or type) so the log stays scannable. - Keep the subject at 10 characters or more so it can name the affected surface clearly. - Aim for 60 characters or fewer for clear GitHub rendering and concise text. The commit-message hook warns without failing when the subject exceeds this target. The project plans to enforce this limit in the future. - Do not end the subject with a period. - Do not put an issue or pull request number in the subject. GitHub appends the pull request number on squash merge, and any other reference belongs in the body. The commit-message hook rejects a subject containing `#` in any position. ```text Add Decimal constructors to Instrument trait Fix non-atomic order event application Refine cross-platform wheel validation Remove stale security audit exceptions ``` Avoid these shapes: ```text feat(bybit): add due_post_only flag # Conventional Commits type and scope fix: bug # lowercase, unspecific, too short Fixed the Bybit post-only rejection flag. # past tense, trailing period Update stuff # says nothing about the surface Fix the post-only flag (#4544) # pull request number added by hand Fix PR #4544 review feedback # issue or pull request number in the subject ``` ### Conventional Commits Do NOT use [Conventional Commits](https://www.conventionalcommits.org/) syntax for commit messages or pull request titles. Many editors and AI assistants emit that format by default, but no commit in this repository's history uses it, and the type and scope ceremony duplicates what the subject already carries. Pull request titles matter here too, because a squash merge turns the PR title into the commit subject. ### Body The body is optional, but anything beyond a trivial change should say why the change was made rather than restate the diff. - Separate the body from the subject with a blank line. - Keep body lines to 79 characters or fewer to align with PEP 8 and traditional Git tooling. - Use prose paragraphs or bullet points, whichever suits the change. Bullets may keep the same imperative voice as the subject, and do not need terminating periods. - Include informative hyperlinks where they help a future reader. ### Issue references - Reference issues from the body, typically on a final line: `Resolves #4534` when the commit closes the issue, or `Related to #4547` when it is partial work. - GitHub appends the pull request number to the subject on squash merge, producing subjects such as `Fix TWAP child-order sizing and interval validation (#4544)`. Do not add that suffix by hand, and do not reference a pull request or issue anywhere else in the subject either. The subject has no room for detail the body carries better, and a hand-written number duplicates or contradicts the appended one. - Aim to keep the pull request title short enough for the appended suffix to leave the squash-merged subject at 60 characters or fewer. # Design Principles Source: https://nautilustrader.io/docs/latest/developer_guide/design_principles/ This page defines the principles, policies, and trade-offs that guide NautilusTrader design. [Architecture](../concepts/architecture.md) describes the components and runtime structure. These policies guide implementation and review; they do not establish that every existing path already conforms. Specific guides describe current behavior and limits. [Runtime conformance contract](runtime_conformance.md) maps selected requirements to source, representative checks, and implementation limits. ## Design priorities Design decisions weigh these quality attributes in roughly this order: - Reliability - Performance - Modularity - Testability - Maintainability - Deployability Performance improvements must preserve critical invariants and their required verification. Testability and maintainability sustain reliability; deployability affects safe rollout, configuration, and recovery. The priority order does not make these qualities optional. ## Data integrity and failure NautilusTrader prioritizes data integrity over availability for trading operations. Arithmetic and data-handling boundaries return errors or panic rather than silently accepting invalid values that could affect trading decisions. ### Invalid operations The system fails fast, either by returning an error or panicking according to the API contract, for: - Arithmetic overflow or underflow in operations on timestamps, prices, or quantities that exceed valid ranges. - Invalid data during deserialization, including NaN, infinity, or out-of-range values in market data or configuration. - Type conversion failures such as negative values where only positive values are valid (timestamps, quantities). - Malformed input parsing for prices, timestamps, or precision values. In a trading system, one incorrect price, timestamp, or quantity can propagate into: - Incorrect position sizing or risk calculations. - Orders placed at incorrect prices. - Backtests producing misleading results. - Silent financial losses. Failing at the invalid operation provides: - **No silent corruption**: Checked inputs fail before the invalid value propagates. - **Immediate feedback**: The caller receives an error, or the process terminates, at the point of the violated contract. - **Diagnostic context**: Errors and panic messages identify the rejected operation or value. - **Deterministic behavior**: With deterministic ordering and configuration, the same invalid input produces the same failure; nondeterministic inputs can still vary the outcome. Expected network, storage, business-validation, and user-input failures have explicit error surfaces. Unrecoverable invariant violations stop the operation or process before invalid state propagates. The [Rust error contracts](rust.md#failure-contract-examples) define panic and fallible API behavior. ### Failure containment Stop the smallest scope whose integrity can no longer be established, provided the remaining system can continue safely. Rejected input or a failed operation need not stop unrelated components when isolation is established. Untrustworthy shared state may require stopping the node. Containment must follow the API's failure contract and proven isolation boundaries; it must not assume a component can recover from an arbitrary panic. Stopping a process does not cancel working venue orders or remove exposure. Recovery must establish venue state before deciding which further actions are safe. ## Executable invariants NautilusTrader incrementally applies high-assurance practices to critical paths. Executable invariants verify that behavior matches the business requirements: - Identify high-impact components, including core domain types and risk and execution flows, and state their invariants in plain language. - Codify those invariants as executable checks (unit tests, property tests, fuzzers, and static assertions) that run in CI. - Enforce ownership and state invariants through types and explicit failure contracts. Add formal tools where their assurance benefit justifies their cost. - Require integrations to preserve existing critical-path invariants, and add executable coverage for invariants they introduce or alter. This approach gives high-stakes flows additional scrutiny without applying the same assurance cost to every path. Further reading: [High Assurance Rust](https://highassurance.rs/). ## Message immutability Messages (requests, responses, events, and commands) are immutable after creation. Their fields remain unchanged for the rest of the message lifetime. See [Message Bus: message integrity](../concepts/message_bus.md#message-integrity) for the ownership rules that follow from this invariant. The invariant protects several properties the system depends on: - **Stable inputs**: Every consumer sees the same message payload. Replaying a sequence preserves the original logical inputs for backtesting, incident reconstruction, and regression testing. - **Temporal integrity**: A message preserves what its producer reported, observed, requested, or inferred at creation time. Preserve the available provenance; immutability does not establish that the producer's information is true. Corrections require an explicit new record. - **Safer concurrency**: Readers do not need coordination to protect message payloads from later rewrites. This removes a common source of races around shared state. - **Debugging and auditability**: Logs, traces, replay tools, and dead-letter inspection retain the original payload, allowing investigation of what the system received or created, when, and what it did with that information. - **Clear ownership boundaries**: Components treat incoming messages as input. If a component needs a different representation, it derives new local state or a new message explicitly. - **More robust distribution**: Serialized messages already cross process and service boundaries as copies. The same ownership rule keeps the in-memory model aligned with that reality. ## Evidence and authority Distinguish external reports, local observations, inferences, and policy decisions. Preserve their source and uncertainty when deriving state. A valid numeric value or state transition does not by itself establish a venue fact. Missing evidence must not become a confirmed outcome merely because a timeout or retry limit expires. The [execution policies](../concepts/execution/policies.md) apply this rule to command outcomes, reconciliation, and the limits of retained history. ## Controlled nondeterminism Make time, randomness, input ordering, and external effects explicit at the boundaries that consume them. Reproducible tests must control the sources that affect their assertions and retain the inputs, seeds, and configuration needed to investigate a failure. State the binary, platform, and input conditions of any determinism guarantee. The [DST contract](../concepts/dst.md) defines the supported scope of seed-controlled execution. Live venue behavior and independent external inputs remain outside that guarantee. ## Bounded resource use Design queues, retries, retained history, and callback chains with explicit resource budgets and exhaustion behavior. Account for payload size as well as item count where memory use varies, and bound work as well as storage so a replenishing queue cannot monopolize execution. Choose backpressure, rejection, or safe termination according to the affected contract. Overload must not silently lose required state transitions or leave partially applied operations presented as complete. These are design requirements; existing paths can still be unbounded, as documented in [live dispatch and overload behavior](../concepts/live.md#dispatch-priority-and-overload-behavior). ## Recovery after failure Unrecoverable faults must not leave the system operating on potentially invalid state. Normal startup includes configured state recovery, so restart uses the same initialization path as ordinary startup. Recovery depends on retained state and backing-store durability. An external supervisor owns restart after process failure. Recovery aims to minimize downtime; its duration depends on the state to restore and the backing store. Execution recovery must reconcile venue state rather than blindly retry venue commands. Normal operation retains graceful shutdown; an unrecoverable fault may make cleanup unsafe. [Runtime failure and recovery](../concepts/architecture.md#crash-only-design) describes the process and persistence boundaries. ## Domain and integration boundaries Domain types and contracts define trading behavior. Components communicate through explicit interfaces and immutable messages. Ports and adapters keep venue transport and backing-store implementations outside the shared trading core, so custom integrations preserve the same domain contracts. ## Backtest and live behavior Backtest, sandbox, and live environments share core trading components and behavioral contracts. The same strategy and execution-algorithm code can run across these environments. Live execution also introduces venue, transport, timing, persistence, external-activity, and reconciliation behavior that a simulation may not reproduce. Shared code does not imply identical outcomes from different inputs or operating conditions. The [common core](../concepts/architecture.md#common-core) supplies the engines and interfaces; [behavioral models](../concepts/behavioral_models.md) define how model implementations enter the runtime. ## Queued callback dispatch requirements Canonical actor and strategy delivery must preserve publication order, exclusive component access, and lifecycle eligibility, with bounded progress. Reentrancy must not change the ordering rule. Ordered delivery does not imply an event-time cache snapshot. These are requirements for queued dispatch, not guarantees of existing synchronous paths. The [callback dispatch contract](callback_dispatch.md) specifies maintenance timing, ownership boundaries, lifecycle invalidation, and draining behavior. # Docs Style Source: https://nautilustrader.io/docs/latest/developer_guide/docs/ This guide outlines the style conventions and best practices for writing documentation for NautilusTrader. The [Markdown Style](markdown_style.md) guide is the shared baseline for Markdown syntax and formatting, and `.markdownlint.jsonc` enforces its mechanical subset. This guide covers what is specific to NautilusTrader documentation rather than repeating that baseline. ## General principles - We favor simplicity over complexity, less is more. - We favor concise yet readable prose and documentation. - We value standardization in conventions, style, patterns, etc. - Documentation should be accessible to users of varying technical backgrounds. ## Documentation types Most pages should fit one of four types ([Divio documentation system](https://docs.divio.com/documentation-system/)). Mixing types in a single page makes it harder to read and harder to maintain. | Type | Purpose | Section | | ---------------- | ------------------------------- | ---------------- | | **Tutorial** | Teach by walking through a task | `tutorials/` | | **How-to guide** | Solve a specific problem | `how_to/` | | **Explanation** | Clarify design and architecture | `concepts/` | | **Reference** | Describe the machinery | `api_reference/` | Two sections are exceptions: `getting_started/` is an onboarding path that combines tutorial-style walkthroughs with setup instructions, and `integrations/` pages mix reference (capabilities, symbology) with how-to content (setup, configuration) so each venue page is self-contained. Standalone how-to content that is not venue-specific belongs in `how_to/`. ### Choosing the right type - **Does your page walk a newcomer through a learning experience?** Tutorial. - **Does it answer "How do I...?" for someone who already knows the system?** How-to guide. - **Does it explain why something works the way it does?** Explanation. - **Does it list classes, config fields, enums, or capabilities?** Reference. A tutorial says "do this, then this, then this." The author picks the path. A how-to guide says "here is how to achieve X." The reader already knows they want X. Keep these distinct: - Tutorials should not assume prior knowledge. - How-to guides should not teach background concepts. When one type needs to reference another, link to it instead of inlining. For example, a how-to guide that configures `LiveNodeConfig` should link to the API reference for field definitions rather than listing them again. ## Language and tone - Use active voice when possible ("Configure the adapter" vs "The adapter should be configured"). - Write in present tense for describing current functionality. - Use future tense only for planned features. - Avoid unnecessary jargon; define technical terms on first use. - Be direct and concise; avoid filler words like "basically", "simply", "just". - Use parallel structure in lists; keep grammatical patterns consistent across items. ## Markdown tables Table syntax, pipe alignment, and delimiter padding follow the [Markdown Style](markdown_style.md#tables) guide. ### Notes and descriptions - All notes and descriptions should have terminating periods. - Keep notes concise but informative. - Use sentence case (capitalize only the first letter and proper nouns). ### Example ```markdown | Order Type | Spot | Margin | USDT Futures | Coin Futures | Notes | | ---------------------- | ---- | ------ | ------------ | ------------ | ----------------------- | | `MARKET` | ✓ | ✓ | ✓ | ✓ | | | `STOP_MARKET` | - | ✓ | ✓ | ✓ | Not supported for Spot. | | `MARKET_IF_TOUCHED` | - | - | ✓ | ✓ | Futures only. | ``` ### Support indicators - Use `✓` for supported features. - Use `-` for unsupported features (not `✗` or other symbols). - When adding notes for unsupported features, emphasize with italics: `*Not supported*`. - Make unsupported notes specific when the reason matters: use `*Not supported by *` for venue gaps, or `*Not currently implemented*` for adapter gaps. - Leave cells empty when no content is needed. ## Code references Inline code and fenced code blocks follow the [Markdown Style](markdown_style.md#code) guide. When referencing code locations, use `file_path::function_name` or `file_path::ClassName` rather than line numbers, which become stale as code changes. ## Headings Heading style, case, and hierarchy follow the [Markdown Style](markdown_style.md#headings) guide: title case for the page heading, sentence case below it. NautilusTrader makes one exception for `index.md` pages: headings and navigation links that name individual documents use the target document's exact H1 title, including its capitalization. General section headings retain sentence case. Always capitalize proper nouns regardless of heading level (product names, technologies, companies, acronyms). ## Lists List markers, ordering, and indentation follow the [Markdown Style](markdown_style.md#lists) guide. End list items with periods when they are complete sentences. ## Links and references Link text, link style, and images follow the [Markdown Style](markdown_style.md#links-and-images) guide. Reference external documentation when appropriate. ## Technical terminology - Base capability matrices on the Nautilus domain model, not exchange-specific terminology. - Mention exchange-specific terms in parentheses or notes when necessary for clarity. - Use consistent terminology throughout the documentation. ## Examples and code samples - Provide practical, working examples. - Include necessary imports and context. - Use realistic variable names and values. - Add comments to explain non-obvious parts of examples. ## Admonitions Use admonition blocks to highlight important information: | Admonition | Purpose | | ------------ | ------------------------------------------------------------- | | `:::note` | Supplementary context that clarifies but isn't essential. | | `:::info` | Important information the reader should be aware of. | | `:::tip` | Helpful suggestions or best practices. | | `:::warning` | Potential pitfalls or important caveats. | | `:::danger` | Critical issues that could cause data loss or system failure. | Avoid overusing admonitions; too many diminish their impact. ## MDX components The docs site (fumadocs) provides built-in MDX components available in all `.md` files. No imports are needed. ### Tabs Use tabs for language-specific or variant content. List Rust before Python so Rust is the default (left-most) tab. For code examples, add `tab="..."` to consecutive fenced code blocks: ```markdown \`\`\`rust tab="Rust" let params = Params::from([("close_position", true.into())]); \`\`\` \`\`\`python tab="Python" strategy.submit_order(order, params={"close_position": True}) \`\`\` ``` For tables or other content, wrap each variant in `` and ``. The instrument Fields tables use this so each language shows a single type column instead of side-by-side Rust and Python columns. Leave a blank line above and below the inner content so the Markdown renders. ```markdown | Field | Type | Required/default | Notes | | --------------- | -------------- | ---------------- | ----------------------- | | `instrument_id` | `InstrumentId` | Required | Stored as `id` in Rust. | | Field | Type | Required/default | Notes | | --------------- | -------------- | ---------------- | ----- | | `instrument_id` | `InstrumentId` | Required | | ``` ### Steps Use `Steps` and `Step` for sequential procedures. ```markdown Configure the adapter. Start the trading node. ``` ### Accordions Use `Accordions` and `Accordion` for collapsible content. ```markdown Content here. ``` ### Files Use `Files`, `Folder`, and `File` for directory tree visualizations. ```markdown ``` ### Cards Use `Cards` and `Card` for linked content grids. ```markdown ``` ### TypeTable Use `TypeTable` for parameter or type documentation tables. ## API documentation - Document parameters and return types clearly. - Include usage examples for complex APIs. - Explain any side effects or important behavior. - Keep parameter descriptions concise but complete. # Environment Setup Source: https://nautilustrader.io/docs/latest/developer_guide/environment_setup/ Use an editor with current Rust and Python language support, such as PyCharm or Visual Studio Code. [uv](https://docs.astral.sh/uv) is the preferred tool for handling all Python virtual environments and dependencies. [prek](https://github.com/j178/prek) is used to automatically run various pre-commit checks, auto-formatters, and linting tools at commit. Source builds and the Rust crates require [Rust](https://www.rust-lang.org) ([installation guide](https://www.rust-lang.org/tools/install)). [Cap'n Proto](https://capnproto.org) is required for serialization schema compilation. The required version is specified in `.nautilus-engineering/tools.toml`. Ubuntu's default package is typically too old, so you may need to install from source (see below). :::info NautilusTrader *must* compile and run on **Linux, macOS, and Windows**. Please keep portability in mind: use `std::path::Path` in code and follow the [shell portability policy](shell.md#define-the-portability-target) for scripts. ::: ## Setup The following steps are for UNIX-like systems, and only need to be completed once. ### Quick setup Use this as a compact setup path for a new Linux or macOS development machine. The detailed sections below explain each step and cover alternatives. Install platform tools first: ```bash tab="Ubuntu" sudo apt-get update sudo apt-get install -y build-essential clang lld curl git make pkg-config ``` ```bash tab="macOS" xcode-select --install ``` Then clone the repository and install the pinned project tools: ```bash git clone --branch develop https://github.com/nautechsystems/nautilus_trader cd nautilus_trader curl https://sh.rustup.rs -sSf | sh source "$HOME/.cargo/env" curl -LsSf https://astral.sh/uv/install.sh | sh export PATH="$HOME/.local/bin:$PATH" cargo install cargo-binstall --locked make install-tools ./scripts/install-capnp.sh make sync source python/.venv/bin/activate export PYO3_PYTHON="$PWD/python/.venv/bin/python" if [ "$(uname -s)" = "Linux" ]; then PYTHON_LIB_DIR="$("$PYO3_PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')" export LD_LIBRARY_PATH="$PYTHON_LIB_DIR${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" fi export PYTHONHOME="$("$PYO3_PYTHON" -c 'import sys; print(sys.base_prefix)')" prek install make build-debug ``` Windows users should follow the source installation steps in the [installation guide](../getting_started/installation.md#from-source), then use the relevant commands from this guide. ### 1. Install dependencies Follow the [installation guide](../getting_started/installation.md), then sync the development and test dependencies from the repository root: ```bash make sync ``` For frequent development, install a debug build of the package into `python/.venv`: ```bash make install-debug ``` ### 2. Install development tools NautilusTrader pins every development tool so that all contributors and CI run identical versions. A single Makefile target installs the full set: ```bash make install-tools ``` This installs: - **Shared Cargo CLIs** pinned in `.nautilus-engineering/tools.toml`: `cargo-audit`, `cargo-deny`, `cargo-edit`, `cargo-llvm-cov`, `cargo-nextest`, and `cargo-vet`. - **NautilusTrader Cargo CLIs** pinned in `Cargo.toml` under `[workspace.metadata.tools]`: `cargo-codspeed`, `cargo-fuzz`, `cargo-hawk`, `cargo-machete`, `cbindgen`, `flamegraph`, and `lychee`. - **Prebuilt binaries** pinned in `.nautilus-engineering/tools.toml`: `prek` (pre-commit runner) and `osv-scanner` (vulnerability scanner). - **uv**, installed at the shared pinned version. The supported local uv minor series is defined in `python/pyproject.toml`. Cap'n Proto is also pinned in `.nautilus-engineering/tools.toml` but installs separately; see the [Cap'n Proto](#capn-proto) section below. Fuzz targets also require a Rust nightly toolchain at runtime because `cargo-fuzz` uses `libfuzzer-sys` and unstable compiler flags: ```bash rustup toolchain install nightly ``` The docs.rs compatibility check uses the dated nightly pinned in `tools.toml`: ```bash rustup toolchain install "$(bash scripts/tool-version.sh nightly)" --profile minimal ``` #### One-off prerequisite: cargo-binstall `make install-tools` uses [`cargo-binstall`](https://github.com/cargo-bins/cargo-binstall) to fetch `prek` as a prebuilt binary instead of compiling it from source. Install `cargo-binstall` once per machine: ```bash cargo install cargo-binstall --locked ``` This is a one-time step. Subsequent runs of `make install-tools` reuse the installed `cargo-binstall`. #### Single source of truth for versions The repository manifests are the canonical source for dependency and tool versions. Do not copy current version numbers into docs, runner images, or scripts unless there is no manifest-backed way to read them. | Source file or section | Defines | | ----------------------------------------- | -------------------------------------------------------- | | `rust-toolchain.toml` | Rust toolchain. | | `Cargo.toml` and `Cargo.lock` | Rust workspace dependencies and exact resolution. | | `Cargo.toml` `[workspace.metadata.tools]` | NautilusTrader-specific Cargo tools. | | `python/pyproject.toml` | Python dependencies and supported Python and uv ranges. | | `python/uv.lock` | Exact Python dependency resolution. | | `.nautilus-engineering/tools.toml` | Shared engineering tools. | | `tools.toml` | NautilusTrader-specific tools without a native manifest. | The shared catalog includes uv, `prek`, `pip-audit`, `osv-scanner`, Cap'n Proto, and common Cargo tools. The local catalog retains the docs.rs nightly, Miri toolchain, and `pypi-attestations` pins. The Makefile reads these via `scripts/cargo-tool-version.sh`, `scripts/tool-version.sh`, and `scripts/uv-version.sh`, so bumping a version in the source file is the only required version change. To check the pinned cargo tool versions against crates.io, run: ```bash make outdated ``` ### 3. Set up Git hooks Set up the file and commit-message hooks, which run automatically when committing: ```bash prek install ``` Rerun `prek install` after pulling a change to the configured hook types. Before opening a pull-request run the formatting and lint suite locally so that CI passes on the first attempt: ```bash make format make pre-commit ``` Make sure the Rust compiler reports **zero errors** -- broken builds slow everyone down. ### 4. Configure environment variables NautilusTrader keeps its uv-managed environment at `python/.venv`, beside `python/pyproject.toml`. This follows [uv's default project environment layout](https://docs.astral.sh/uv/concepts/projects/layout/#the-project-environment), which keeps the environment where uv and Python editors expect to discover it. Run direct uv project commands from `python/` or pass `--project python` from the repository root. Make targets and CI select the project themselves. :::warning If this checkout previously used the root `.venv`, remove any `UV_PROJECT_ENVIRONMENT` export from your shell startup files and the current shell before running Make or uv. This override takes precedence over uv's project discovery. Also replace any `PYO3_PYTHON` export that points to the root `.venv/bin/python` with this checkout's `python/.venv/bin/python`. Editing a startup file does not update existing shells or running applications: repeat the exports in each shell and restart applications that inherited the old environment. ::: **Required for Rust/PyO3 (Linux and macOS)**: When using Python installed via `uv` on Linux or macOS, set the following environment variables from the repository root after `make sync`: Use the commands for your shell. Bash and Zsh use `export` and `activate`; Fish uses `set -gx` and `activate.fish`. Source Fish scripts only from Fish. ```bash tab="Bash / Zsh" # Set the Python executable path for PyO3 export PYO3_PYTHON="$PWD/python/.venv/bin/python" # Linux only: Set the library path for the uv-managed Python runtime PYTHON_LIB_DIR="$("$PYO3_PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')" export LD_LIBRARY_PATH="$PYTHON_LIB_DIR${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" # Set the Python home path (required for Rust tests) export PYTHONHOME="$("$PYO3_PYTHON" -c 'import sys; print(sys.base_prefix)')" ``` ```fish tab="Fish" set -gx PYO3_PYTHON "$PWD/python/.venv/bin/python" if test (uname -s) = Linux set -l python_lib_dir ("$PYO3_PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))') set -gx LD_LIBRARY_PATH "$python_lib_dir" (string match -v "" -- $LD_LIBRARY_PATH) end set -gx PYTHONHOME ("$PYO3_PYTHON" -c 'import sys; print(sys.base_prefix)') ``` :::note The `LD_LIBRARY_PATH` export is Linux-specific and not needed on macOS or Windows. - `PYO3_PYTHON` tells PyO3 which Python interpreter to use, reducing unnecessary recompilation. - `PYTHONHOME` is required when running `make cargo-test` with a `uv`-installed Python. Without it, tests that depend on PyO3 may fail to locate the Python runtime. ::: To verify your environment is configured correctly: ```bash python -c "import sys; print('Python:', sys.executable, sys.version)" echo "PYO3_PYTHON: $PYO3_PYTHON" echo "PYTHONHOME: $PYTHONHOME" ``` ## Dependency management Python dependencies are managed by [uv](https://docs.astral.sh/uv). The `[tool.uv]` section in `python/pyproject.toml` enforces three supply chain safety settings: - **`required-version`**: local uv commands accept any patch release in the supported minor series. If your local uv is outside that range, `uv lock` and `uv sync` fail with a version mismatch. `.nautilus-engineering/tools.toml` separately pins the exact version used by CI, Docker, pre-commit, and `make update-uv`. The stub targets run through `make sync`, so they enforce the same supported range; see [Generated Python artifacts](rust.md#generated-python-artifacts). - **`exclude-newer = "7 days"`**: `uv lock` ignores package versions published within the last 7 days. This gives the community time to detect and quarantine compromised releases before they enter the lockfile. The value accepts an RFC 3339 timestamp (`"2026-03-30T00:00:00Z"`), a friendly duration (`"7 days"`, `"1 week"`, `"24 hours"`), or an ISO 8601 duration (`"P7D"`, `"P1W"`, `"PT24H"`). uv 0.11.8+ stores the friendly/ISO form as `exclude-newer-span` inside `python/uv.lock` and emits a sentinel `exclude-newer` timestamp alongside it for backwards compatibility. `python/uv.lock` uses that format. - **`no-build-package`**: explicit list of every third-party package locked in `python/uv.lock`. `uv` refuses to build any of them from source. In normal operation uv prefers wheels, so the setting is a no-op; it triggers only if a listed package stops publishing wheels for the target platform, in which case `uv lock` fails rather than silently building from an sdist. The local workspace package is intentionally not in the list because it must be built by the workspace's own build backend. The list is kept in sync with `python/uv.lock` by `scripts/check-no-build-packages.sh`, which also runs as a pre-commit hook on changes to the lockfile or manifest. ### Bypassing the cooldown When a security patch or critical bug fix must be pulled in immediately, review the release and override `exclude-newer` for that lock operation. Prefer a package-scoped override so unrelated packages remain subject to the 7-day default. Do not add persistent package overrides to `python/pyproject.toml`. All forms accept a timestamp, friendly duration, or ISO duration; package overrides additionally accept `false` to exempt a package from the cooldown entirely. ```bash # Shorten the cooldown for a single package (friendly duration) uv lock --project python --exclude-newer-package "somepackage=1 day" # Pin a single package to an absolute cutoff uv lock --project python --exclude-newer-package "somepackage=2026-03-30T00:00:00Z" # Exempt a single package from the cooldown entirely uv lock --project python --exclude-newer-package "somepackage=false" # Disable the cooldown for the whole resolution after reviewing every newly eligible package uv lock --project python --exclude-newer "0 seconds" ``` The CLI flag overrides the `python/pyproject.toml` value for that invocation only. The config remains unchanged for subsequent runs. ### Updating uv To support a new uv minor series, change `required-version` in `python/pyproject.toml`. To update the exact project version within that range, update Nautilus Engineering's `[uv].version`, sync the shared catalog, then update the `rev` in `.pre-commit-config.yaml` and each digest-pinned uv Docker image. Run `make update-uv` to install the project version locally. ### Rust dependency cooldown before compilation Repository builds must check every resolved registry dependency before Cargo can execute dependency build scripts or procedural macros. `make check-cargo-cooldown` checks all tracked `Cargo.lock` files against `[workspace.metadata.cooldown]` in `Cargo.toml`, including versions already committed or pulled from another branch. It does not need a Git comparison base or full checkout history. The Rust build, stub, check, Clippy, test, coverage, documentation, benchmark, and local CLI install targets require this check. Stub generation counts as compilation because it runs the Rust `python-stub-gen` binary through Cargo. Each compilation target waits for the gate, including under parallel Make. Compilation uses the checked lockfile without resolving replacements. Pre-flight also checks early, and CI common setup checks before repository compilation begins. A version inside the cooldown window requires both an exact entry in `[workspace.metadata.cooldown.allow]` and a matching cargo-vet audit. Unsupported registries fail the check. Publication dates come from the committed database at `.supply-chain/crate-dates.json`. Recorded dates are trusted offline; versions missing from the database are looked up on crates.io and fail closed when the registry is unreachable. The pre-commit hook checks all resolved versions using these recorded dates, including entries added since the comparison base. A clean Git diff does not establish that dependencies are old enough. `make cargo-update` records dates for every change it accepts. After a manual lockfile edit, run `bash scripts/check-cargo-cooldown.sh --update-db` to reconcile the database, which also prunes entries no tracked lock resolves. The dependency-update command separately re-verifies newly added dates against crates.io; routine pre-commit and full checks use the committed database offline. Successful full checks are cached as `.cargo-cooldown.json` in `CARGO_TARGET_DIR`, or the Make `TARGET_DIR` when no Cargo target directory is set. CI uses its configured Cargo target directory so persistent runners retain the cache between jobs. Changes to any checked lockfile, the policy, audits, database, or the check script invalidate the cache. Failed checks are not cached. Treat this file as local verification state; do not restore it from an untrusted source. This gate reduces exposure to newly published malicious registry releases. It does not establish that older releases are safe, sandbox build scripts, or vet Git and local path dependencies. Development-tool bootstrap commands such as `make install-tools` install external packages with separate dependency resolutions and are outside this repository-lockfile gate. Direct Cargo and Maturin invocations also bypass Make: run the full check first and pass `--locked` when building repository code. Keep manifests and lockfiles unchanged between the check and compilation. ## Builds The Python package and the standalone Nautilus CLI are separate build artifacts. `make build-debug` and `make build` install the Python package into `python/.venv`; neither command updates the `nautilus` binary in Cargo's binary directory. See the [Nautilus CLI developer guide](#nautilus-cli-developer-guide) when changing or using the CLI. After changing Rust bindings or Python package code, use a debug build for normal development. It skips release optimization and LTO, which reduces build time and peak memory use: ```bash make build-debug ``` Use `make build` when you need an optimized build. The release profile uses fat LTO and one code generation unit, which increases peak memory use. Fat LTO can complete on a 16 GB machine when the build has access to the full memory allocation and sufficient swap. Check VM or container memory limits when applicable. If the linker runs out of memory, use ThinLTO for an optimized local build: ```bash CARGO_PROFILE_RELEASE_LTO=thin make build ``` This override applies only to that command. Use the default fat LTO profile for performance measurements. ### Refresh after pulling changes Use the command that updates the affected artifact. The build targets call their prerequisites, so `make build-debug` also syncs Python dependencies and regenerates Python type stubs. | Changed input | Command | Updated artifact | | --------------------------------------------------- | ---------------------------- | ------------------------------------------------- | | `python/pyproject.toml` or `python/uv.lock` | `make sync` | Dependencies in `python/.venv`. | | Rust bindings, Python package code, or stub sources | `make build-debug` | Debug Python package and generated type stubs. | | CLI code, SQL initialization code, or `schema/sql` | `make install-cli` | Standalone `nautilus` binary in Cargo's bin path. | | Cargo, uv, `prek`, or OSV Scanner tool pins | `make install-tools` | Pinned development tools. | | Cap'n Proto version in the shared catalog | `./scripts/install-capnp.sh` | Cap'n Proto compiler. | The environment variables in [Configure environment variables](#4-configure-environment-variables) contain checkout-specific paths. After switching checkouts, changing the selected Python version, or recreating `python/.venv`, activate that checkout's environment and export the variables again. Verify that the shell resolves Python from the expected checkout: ```bash source python/.venv/bin/activate command -v python python --version ``` In Fish, use `source python/.venv/bin/activate.fish` for activation. Activation alone does not refresh `PYO3_PYTHON`; repeat the environment variable commands above for the selected checkout. ## Cap'n Proto [Cap'n Proto](https://capnproto.org) is required for serialization schema compilation. The required version is defined in `.nautilus-engineering/tools.toml`. Install the correct version for your platform: ```bash tab="Script (Linux/macOS)" ./scripts/install-capnp.sh ``` ```bash tab="macOS (Homebrew)" brew install capnp ``` ```bash tab="Linux (source)" CAPNP_VERSION=$(bash scripts/tool-version.sh capnp) cd ~ wget https://capnproto.org/capnproto-c++-${CAPNP_VERSION}.tar.gz tar xzf capnproto-c++-${CAPNP_VERSION}.tar.gz cd capnproto-c++-${CAPNP_VERSION} ./configure make -j$(nproc) sudo make install sudo ldconfig ``` ```bash tab="Windows (Chocolatey)" choco install capnproto ``` Verify the installed version matches the shared catalog: ```bash capnp --version ``` The install script ensures the pinned version is installed. If Homebrew or Chocolatey provides an older version, install from source or see the [Cap'n Proto installation guide](https://capnproto.org/install.html). ## Faster builds The Cranelift code generation backend can reduce local build time for development, tests, and IDE checks. It requires the nightly Rust toolchain and local changes to `Cargo.toml`: ```bash rustup toolchain install nightly --component rust-analyzer ``` Save the patch below, then apply it with `git apply `. Remove it with `git apply -R ` before pushing changes. :::warning Do not commit these changes. The cranelift patch is for local development only and will break CI if pushed. ::: ```diff diff --git a/Cargo.toml b/Cargo.toml --- a/Cargo.toml +++ b/Cargo.toml @@ -1,3 +1,5 @@ +cargo-features = ["codegen-backend"] + [workspace] resolver = "2" members = [ @@ -424,6 +426,7 @@ lto = false panic = "unwind" incremental = true +codegen-backend = "cranelift" # Compile third-party deps at opt-level=1 in dev/test profiles. Workspace # members keep opt-level=0 (fast iteration); deps recompile rarely so the @@ -444,6 +447,7 @@ strip = false lto = false incremental = true +codegen-backend = "cranelift" [profile.test.package."*"] opt-level = 1 @@ -452,6 +456,7 @@ inherits = "test" debug = false # Improves compile times strip = "debuginfo" # Improves compile times +codegen-backend = "cranelift" [profile.ci-pr] inherits = "test" ``` Run local build commands with `RUSTUP_TOOLCHAIN=nightly`, for example: ```bash RUSTUP_TOOLCHAIN=nightly make build-debug ``` Set the same toolchain in your [rust-analyzer settings](#rust-analyzer-settings) when using this local patch. ## Services Initialize PostgreSQL, Redis, and pgAdmin from the repository root: ```bash make init-services ``` This starts the containers and initializes the NautilusTrader database schema. To start the containers without reinitializing the schema, run `make start-services`. To start one service, use the Compose file directly: ```bash docker compose -f .docker/docker-compose.yml up -d postgres ``` The development services are: - `postgres`: PostgreSQL with `POSTGRES_USER=nautilus`, `POSTGRES_PASSWORD=pass`, and `POSTGRES_DB=nautilus` by default. - `redis`: Redis server. - `pgadmin`: pgAdmin 4 for database management and administration. :::info Please use this as development environment only. For production, use a proper and more secure setup. ::: Use `make stop-services` to stop the containers without removing their data. Use `make purge-services` only when you intend to delete the development volumes. PostgreSQL-backed tests can each maintain several connections. On a high-core workstation, the local nextest concurrency can exceed the development container's connection limit. Use the CI profile to match CI's lower concurrency: ```bash NEXTEST_PROFILE=ci make cargo-test-extras ``` To retain more local parallelism, set an explicit bounded worker count, for example: ```bash NEXTEST_TEST_THREADS=8 make cargo-test-extras ``` ## Nautilus CLI developer guide The Nautilus CLI is a standalone Rust binary for PostgreSQL administration and other repository operations. It is independent from the Python package installed by `make build-debug` or `make build`. ### Build and select the CLI Install the CLI from the current checkout with: ```bash make install-cli ``` This target runs `cargo install --locked --force` and places `nautilus` in Cargo's binary directory, normally `~/.cargo/bin`. Reinstall it after pulling changes to `crates/cli`, SQL initialization code, or `schema/sql`. An installed CLI can otherwise remain older than the checkout while reading newer schema files from it. Before running repository-dependent commands, check which binary the shell resolves and its version: ```bash command -v nautilus nautilus --version ``` To build and run the CLI directly from the checkout without replacing the installed binary, use: ```bash cargo run --locked --package nautilus-cli --bin nautilus -- --help ``` :::warning On Linux systems with GNOME, `/usr/bin/nautilus` is normally the GNOME file manager. Select the NautilusTrader CLI with one of these methods: - Put `~/.cargo/bin` before `/usr/bin` in `PATH`. - Run `~/.cargo/bin/nautilus` explicitly. - Add `alias nautilus="$HOME/.cargo/bin/nautilus"` to the shell configuration. ::: Windows source installs require GNU Make through MSYS2 or WSL. The nightly workflow also publishes a Windows x86-64 CLI archive. Run `nautilus --help` to view the available command groups. ### Database commands The database commands accept connection settings as command-line arguments or through a `.env` file in the current working directory or one of its parents. The CLI also accepts the corresponding environment variables. | Flag | Environment variable | Purpose | | ------------ | -------------------- | ------------------------------------------------------------- | | `--host` | `POSTGRES_HOST` | Database host. | | `--port` | `POSTGRES_PORT` | Database port. | | `--username` | `POSTGRES_USERNAME` | Connecting administrator, normally the `postgres` role. | | `--password` | `POSTGRES_PASSWORD` | Administrator password and password for the application role. | | `--database` | `POSTGRES_DATABASE` | Database name and application role created during `init`. | | `--schema` | `SCHEMA_DIR` | Directory containing the SQL schema files. | For example: ```dotenv POSTGRES_HOST=localhost POSTGRES_PORT=5432 POSTGRES_USERNAME=postgres POSTGRES_PASSWORD=pass POSTGRES_DATABASE=nautilus ``` `nautilus database init` creates or updates the roles and schema from the SQL files. Pass the schema directory explicitly so renamed clones and worktrees do not depend on checkout path detection: ```bash nautilus database init --schema "$PWD/schema/sql" ``` Use a CLI built from the same checkout as these schema files. The initialization is designed to be re-run, including after an earlier run stopped partway through. :::danger `nautilus database drop` removes the target schema, privileges, role, and stored data. Use it only for a disposable database or after confirming that the data can be deleted. ::: Run `nautilus database --help` for the complete command syntax. ## Rust analyzer settings Rust analyzer is a popular language server for Rust and integrates with many IDEs. Configure its `VIRTUAL_ENV` to use `python/.venv`. If PyO3 analysis cannot locate Python, also provide the `PYO3_PYTHON` and `PYTHONHOME` values from [Configure environment variables](#4-configure-environment-variables). The examples below cover VS Code and AstroNvim. For other settings, see the [rust-analyzer configuration](https://rust-analyzer.github.io/book/configuration.html). ```json tab="VSCode" { "rust-analyzer.restartServerOnConfigChange": true, "rust-analyzer.linkedProjects": [ "Cargo.toml" ], "rust-analyzer.cargo.features": "all", "rust-analyzer.check.workspace": false, "rust-analyzer.check.extraEnv": { "VIRTUAL_ENV": "/python/.venv", "CC": "clang", "CXX": "clang++" }, "rust-analyzer.cargo.extraEnv": { "VIRTUAL_ENV": "/python/.venv", "CC": "clang", "CXX": "clang++" }, "rust-analyzer.runnables.extraEnv": { "VIRTUAL_ENV": "/python/.venv", "CC": "clang", "CXX": "clang++" }, "rust-analyzer.check.features": "all", "rust-analyzer.testExplorer": true } ``` ```lua tab="Neovim (AstroLSP)" config = { rust_analyzer = { settings = { ["rust-analyzer"] = { restartServerOnConfigChange = true, linkedProjects = { "Cargo.toml" }, cargo = { features = "all", extraEnv = { VIRTUAL_ENV = "/python/.venv", CC = "clang", CXX = "clang++", }, }, check = { workspace = false, command = "check", features = "all", extraEnv = { VIRTUAL_ENV = "/python/.venv", CC = "clang", CXX = "clang++", }, }, runnables = { extraEnv = { VIRTUAL_ENV = "/python/.venv", CC = "clang", CXX = "clang++", }, }, testExplorer = true, }, }, }, } ``` # FFI Memory Contract Source: https://nautilustrader.io/docs/latest/developer_guide/ffi/ NautilusTrader exposes a C foreign function interface (FFI) only from `nautilus-core` and `nautilus-model`. Both crates gate the interface behind their `ffi` Cargo feature and keep the exported modules under `crates/core/src/ffi/` and `crates/model/src/ffi/`. Other workspace crates use Rust APIs or PyO3 bindings. The separate `nautilus-plugin` crate defines the public guest plug-in ABI and does not share this memory contract. The rules below are strict. Violating them can cause undefined behavior, including double frees, memory leaks, and invalid pointer access. ## Panic handling Rust panics must never unwind across an `extern "C"` function. Exported functions that can panic must route their implementation through `nautilus_core::ffi::abort_on_panic`, which logs the panic and aborts the process before unwinding crosses the C boundary. ## `CVec` ownership `CVec` is a C-compatible representation of Rust vector allocation metadata. A `CVec` created from `Vec` transfers unique ownership of the allocation to the foreign caller. It is intentionally neither `Copy` nor `Clone` in Rust, but C can still copy its fields, so callers must enforce the same exactly-once ownership rule. | Step | Owner | Action | | ---- | ------- | --------------------------------------------------------------------------------------------- | | 1 | Rust | Convert `Vec` into `CVec`, transferring the allocation to the caller. | | 2 | Foreign | Read the elements without changing `ptr`, `len`, or `cap`. | | 3 | Foreign | Call the matching type-specific `vec_drop_*` function exactly once to release the allocation. | Forgetting the drop leaks the allocation. Dropping the same allocation more than once can corrupt the allocator and crash the process. Empty `CVec` values have `len == 0` and `cap == 0`. Their pointer is an opaque sentinel and must not be dereferenced. Rust consumers must use `CVec::into_vec`, which handles the empty case before it inspects the pointer. Borrowing consumers must use `CVec::as_slice` for the same reason. Both methods are unsafe because the public metadata cannot prove allocation provenance, alignment, initialization, or exclusive ownership. Any exported function that accepts a caller-provided `CVec` and invokes either method must: - Be an `unsafe extern "C" fn`. - Document the caller obligations in a `# Safety` section. - Validate `len`, `cap`, and null-pointer invariants before reconstructing or borrowing data. - Use a concrete element type that matches the original `Vec` allocation. ## Type-specific drop functions There is no generic `cvec_drop`. Reconstructing every allocation as `Vec` gives the allocator the wrong element layout for other types. Each owned vector crossing the boundary requires a drop function for its exact element type, such as `vec_drop_book_levels`, `vec_drop_book_orders`, or `vec_drop_fills`. Add the drop function beside the producer so reviews can verify the pair together. Tests must cover the empty sentinel and any metadata checks implemented by the consumer. ## Borrowed foreign buffers Memory allocated outside Rust must not be reconstructed as `Vec`. Borrow it with `CVec::as_slice`, and copy it with `to_vec()` when Rust needs owned storage. The foreign caller keeps ownership and must release the original buffer with the allocator that created it. ## Opaque pointers Model objects whose layout is not `repr(C)` cross the ABI as opaque owning pointers. The generated header forward-declares the type, so foreign callers handle it only through exported functions. Constructors return an owning `*mut T` from `Box::into_raw`, pure accessors take `&T` or `&mut T` references, and every constructor must have a matching drop function: ```rust #[unsafe(no_mangle)] pub extern "C" fn orderbook_new(id: InstrumentId, book_type: BookType) -> *mut OrderBook { Box::into_raw(Box::new(OrderBook::new(id, book_type))) } /// # Safety /// /// `book` must be a live owning pointer returned by [`orderbook_new`], and must not /// be used after this call. /// /// # Panics /// /// Panics if `book` is null. #[unsafe(no_mangle)] pub unsafe extern "C" fn orderbook_drop(book: *mut OrderBook) { abort_on_panic(|| { assert!(!book.is_null(), "`book` was NULL"); // SAFETY: Caller guarantees `book` was allocated by `orderbook_new` drop(unsafe { Box::from_raw(book) }); }); } ``` The foreign owner must call the drop function exactly once. It must not copy the pointer, consume both copies, or use the pointer after the drop. ## Review checklist For each new or changed FFI export: - Keep the implementation in `nautilus-core` or `nautilus-model`. - Use `repr(C)` for every type whose layout crosses the boundary. - Prevent panics from unwinding across the boundary. - Pair each owned allocation with one type-specific release path. - State complete pointer and ownership obligations in `# Safety` documentation. - Add focused tests for the ownership and validation rules. # Developer Guide Source: https://nautilustrader.io/docs/latest/developer_guide/ Guidance on developing and extending NautilusTrader to meet your trading needs or to contribute improvements back to the project. The core is written in Rust. Python serves as the control plane for strategy logic, configuration, and orchestration. PyO3 bridges the two, exposing Rust functionality to Python with minimal overhead. } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> # Markdown Style Source: https://nautilustrader.io/docs/latest/developer_guide/markdown_style/ Standard revision: 1 This document defines a shared Markdown baseline for repositories that adopt a local copy. Keep copies byte-for-byte identical to the maintained source and put repository-specific additions in a separate local guide. ## Requirement levels Each rule has one of three levels: - **Required:** Applies to every in-scope file. Rules stated as unqualified imperatives such as "Use" and "Do not", or with "must", are Required. - **Preferred:** Identifies the default when more than one valid form exists. Preferred rules use "Prefer" and do not affect conformance. - **Transitional:** Applies to the named construct when it is added or substantially edited, and is labeled **Transitional** or described as transitional. Existing instances may remain until a separate migration. Statements with "may" or "allowed" grant bounded permissions rather than obligations. Any condition limiting that permission is Required. A repository may document narrower local exceptions where its renderer, generated content, or imported material requires them. ## Language and extensions - Use [CommonMark](https://spec.commonmark.org/current/) as the base specification. - Use [GitHub Flavored Markdown](https://github.github.com/gfm/) for tables, task lists, strikethrough, and autolinks. - Use additional front matter, Markdown extensions, or renderer components only when the repository documents and supports them. - Do not introduce syntax that depends on an undocumented renderer extension. The linked specifications are references, not routine prerequisites. Open them only to resolve a concrete parser or renderer ambiguity that the local guide, surrounding content, and markdownlint do not answer. ## Enforcement - Follow the repository's `.markdownlint.jsonc` or equivalent local configuration. - Treat this standard as the intended style and markdownlint as automated enforcement of its mechanical subset. - Treat an undocumented conflict between this standard and the local configuration as drift to resolve, not an implicit exception. - Ensure all authored Markdown in the repository's lint scope passes its configured Markdown check. - Do not hand-edit generated Markdown. Change its source and run the generator. Generated files, imported third-party documents, and renderer fixtures may use documented repository exclusions. ## Headings - Use ATX headings (`#`, `##`, `###`, and so on). - Use one H1 as the first Markdown heading. - Use title case for H1. - Use sentence case for H2 and below. - Maintain a logical hierarchy without skipping heading levels. - Leave one blank line above and below each heading. ## Paragraphs and wrapping - Separate paragraphs with one blank line. - Do not add consecutive blank lines. - Prefer a line-length target of 100-120 characters for prose. - Prefer natural breaks and a longer line over leaving one to three words on the next line. - Allow code blocks, tables, and long link destinations to exceed the target when needed. ## Lists - Use `-` for unordered list items. - Use ordered lists only when sequence matters. - **Transitional:** Use `1.` for every source item in an ordered list so the renderer supplies the displayed numbers. - Keep list indentation and spacing consistent with the local markdownlint configuration. - Leave one blank line before and after a list. Example: ```markdown - First item - Second item 1. First step 1. Second step ``` ## Admonitions - Use only the admonition syntax that the repository documents and renders. - When the repository documents and renders GitHub alerts without a stricter local syntax, use the GitHub blockquote alert form with one of `NOTE`, `TIP`, `IMPORTANT`, `WARNING`, or `CAUTION`. - When support for that extension is absent or unconfirmed, use a portable blockquote with a bold text label. - For a multi-paragraph admonition, prefix every content line and blank continuation line with `>`. A bare blank line ends the admonition. - Keep the type label in the source and rendered output. Do not convey meaning through color or an icon alone. - Preserve established, supported admonition syntax during a narrow edit. Do not convert it only to impose this fallback. GitHub alert: ```markdown > [!WARNING] > Back up the database before running the migration. ``` Portable fallback: ```markdown > **Warning:** Back up the database before running the migration. ``` ## Tables - Use GFM pipe tables for tabular content. - Include leading and trailing pipes. - Align pipe characters vertically. - Pad each column to its widest cell plus one space either side, which matches Prettier's default table output. MD060 checks that pipes line up but ignores cell content, so it accepts a column padded far wider than anything in it. - Pad delimiter cells with spaces (`| ----- |`, not `|-----|`); MD060 does not check this either. - The `normalize markdown table padding` pre-commit hook rewrites tables to this form, so there is no need to count characters by hand. - Prefer left-aligned text columns and right-aligned numeric columns where appropriate. - Keep the delimiter row consistent with the intended rendered alignment. - Avoid HTML tables unless Markdown cannot express the required structure or the repository documents the need. Example: ```markdown | Name | Value | | ----- | ----: | | Alpha | 42 | | Beta | 17 | ``` ## Code - Use backtick-fenced code blocks instead of indented code blocks. - **Transitional:** Specify a language for every opening fence. Use `text` for plain text or output without a more specific grammar. - Use a longer outer fence when documenting fenced Markdown. - Use inline code for commands, file names, functions, types, environment variables, configuration keys, and identifiers. Example: ````markdown ```rust fn main() { println!("Hello"); } ``` ```` Run `cargo test` and edit `.markdownlint.jsonc`. ## Emphasis - Use `*italic*` for emphasis. - Use `**bold**` for strong emphasis. - Avoid emphasis that does not add meaning. ## Thematic breaks Use three hyphens: ```markdown --- ``` ## Links and images - Use descriptive link text. - Prefer inline links. Prefer reference-style links when the same destination appears more than once in a document. - Avoid bare URLs. - Keep internal links relative when the repository's renderer supports them. - Give images useful alternative text. Use empty alternative text only for deliberately decorative images. ## HTML - Prefer portable Markdown syntax over raw HTML. - Use raw HTML only when Markdown cannot express the required result or the repository documents the element or component. - Preserve supported front matter, MDX components, fence attributes, and other repository extensions when editing surrounding content. ## Files - Use UTF-8 encoding. - Use LF line endings. - End each file with one newline. - Do not leave trailing whitespace or use trailing spaces to create Markdown hard line breaks. ## Editing guidance When creating or modifying Markdown: - For narrow edits, preserve surrounding style and use markdownlint without loading the full guide unless both leave a concrete question unanswered. - Before creating or substantially restructuring documentation, consult only the relevant sections of the repository-local Markdown guide. - Preserve the existing document structure unless the task requires a change. - Keep formatting consistent with surrounding content and supported renderer extensions. - Do not reformat unrelated sections. - Run the repository's focused Markdown check when available. ## Transitional adoption ATX headings are enforced through `MD003`. Repeated `1.` ordered-list markers and languages on opening code fences are transitional rules, but `MD029` and `MD040` remain disabled until existing documents receive separate mechanical migrations. Do not broaden a narrow documentation change solely to migrate those existing constructs. # Plugins Source: https://nautilustrader.io/docs/latest/developer_guide/plugins/ The `nautilus-plugin` crate defines the artifact contract for NautilusTrader plug-ins: an independently compiled Rust `cdylib` that identifies itself with a versioned manifest and exchanges values across a C-ABI boundary. The crate covers artifact identity and the boundary primitives only. It does not load, register, or run plug-ins; the loading host is an internal Nautilus deployment detail and is not part of this repository. :::warning The plug-in ABI is early alpha and the contract is unstable. Pin plug-in builds to the matching `nautilus-plugin` version. ::: ## Artifact contract A plug-in is a Rust `cdylib` that exports a single entry symbol, `nautilus_plugin_init`. The `nautilus_plugin!` macro generates that symbol together with the static manifest carrying the build identity: ```rust nautilus_plugin::nautilus_plugin! { name: "example-plugin", vendor: "Nautech", version: env!("CARGO_PKG_VERSION"), } ``` `name` and `version` are required, `vendor` defaults to an empty string. Invoke the macro once per artifact at module scope, set `crate-type = ["cdylib"]` in the artifact's `Cargo.toml`, and depend on the matching `nautilus-plugin` version. ## Manifest compatibility `nautilus_plugin_init` takes an opaque host pointer and returns a `PluginManifest` held in process-lifetime storage, or null when the host pointer is null or the call panics. `PluginManifest::validate` checks the invariants a host relies on before registration. It reports every structural problem it finds and fails when: - `abi_version` does not equal `NAUTILUS_PLUGIN_ABI_VERSION`, or `build_id.schema_version` does not equal `PLUGIN_BUILD_ID_VERSION`. - `plugin_name` or `plugin_version` is empty. - Any manifest string is malformed: a null pointer with non-zero length, or bytes that are not valid UTF-8. - `build_id.precision_mode` or `build_id.fixed_precision` differs from the host build. Precision is validated because it changes model type layout across the boundary. The remaining build identity fields (`nautilus-plugin` version, `rustc` version, target triple, and build profile) are diagnostic. ## Boundary rules Only the `#[repr(C)]` types in `nautilus_plugin::boundary`, and `#[repr(C)]` types built from them, may appear in a signature that crosses the boundary. `String`, `Vec`, and `Box` rely on Rust's unstable ABI and must never cross it. Panics must not unwind across the boundary either; the generated entry symbol catches them and returns null. Under the current ABI, plug-ins are not unloaded, so manifest storage stays valid for the life of the process. See [the crate docs](https://docs.rs/nautilus-plugin) for the boundary and manifest types. # Python Source: https://nautilustrader.io/docs/latest/developer_guide/python/ The [Python](https://www.python.org/) programming language is used for the majority of user-facing code in NautilusTrader. Python provides a rich ecosystem of libraries and frameworks, making it ideal for strategy development, data analysis, and system integration. ## Code style ### PEP-8 The codebase generally follows the PEP-8 style guide. One notable departure is that Python truthiness is not always taken advantage of to check if an argument is `None` for everything other than collections. As per the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html), it's discouraged to use truthiness to check if an argument is/is not `None`, when there is a chance an unexpected object could be passed into the function or method which will yield an unexpected truthiness evaluation (which could result in a logical error type bug). *"Always use if foo is None: (or is not None) to check for a None value. E.g., when testing whether a variable or argument that defaults to None was set to some other value. The other value might be a value that's false in a boolean context!"* :::note Use truthiness to check for empty collections (e.g., `if not my_list:`) rather than comparing explicitly to `None` or empty. ::: We welcome all feedback on where the codebase departs from PEP-8 for no apparent reason. ### Type hints All function and method signatures *must* include type annotations: ```python def __init__(self, config: EMACrossConfig) -> None: def on_bar(self, bar: Bar) -> None: def on_save(self) -> dict[str, bytes]: def on_load(self, state: dict[str, bytes]) -> None: ``` **Union syntax**: Use PEP 604 union syntax for optional types: ```python # Preferred def get_instrument(self, id: InstrumentId) -> Instrument | None: # Avoid def get_instrument(self, id: InstrumentId) -> Optional[Instrument]: ``` **Generic types**: Use Python 3.12 type parameter syntax for reusable functions and classes: ```python def first[T](values: list[T]) -> T: return values[0] ``` ### Docstrings The [NumPy docstring spec](https://numpydoc.readthedocs.io/en/latest/format.html) is used throughout the codebase. This needs to be followed consistently so the docs build correctly. **Python** docstrings should be written in the **imperative mood** - e.g. *"Return a cached client."* This convention aligns with the prevailing style of the Python ecosystem and makes generated documentation feel natural to end-users. #### Private methods Do not add docstrings to private methods (prefixed with `_`): - Docstrings generate public-facing API documentation. - Docstrings on private methods incorrectly imply they are part of the public API. - Private methods are implementation details not intended for end-users. Exceptions where docstrings are acceptable: - Very complex methods with non-trivial logic, multiple steps, or important edge cases. - Methods requiring detailed parameter or return value documentation due to complexity. When a private method needs context (such as a tricky precondition or side effect), prefer a short inline comment (`#`) near the relevant logic rather than a docstring. ### Properties vs methods (PyO3 bindings) When exposing Rust types to Python via PyO3, use `#[getter]` (property) or a plain method based on what the call site communicates, not whether the value can change: - **Property (`#[getter]`):** cheap, side-effect-free, attribute-like view of current state. Scalar fields, predicates, and lightweight derived values belong here even if they change over the object's lifetime. Examples: `status`, `side`, `quantity`, `price`, `is_open`, `has_inputs`, `realized_pnl`, `venue_order_id`. - **Method (no `#[getter]`):** actions, mutations, nontrivial work, allocations/copies, I/O, or anything that takes arguments. Examples: `apply(fill)`, `unrealized_pnl(price)`, `calculate_pnl(...)`. - **Gray area (prefer method):** getters that clone or allocate a collection each call. Using a method signals the cost to the caller. Examples: `events()`, `adjustments()`, `client_order_ids()`, `trade_ids()`. ## Python live callback routing Python live nodes keep one runtime invariant: Tokio worker threads do not run Python code during live trading. `LiveNode::py_run` releases the GIL while the Rust async runtime runs. Worker-side work that must trigger Python uses existing live runner event channels instead of calling `Python::attach` on the worker. Timer callbacks use the time-event channel. The runner drains that channel during startup buffering and the main select loop, then executes callbacks on the live event loop thread. This path is a boundary for unavoidable user Python callback work. It is not a place to move adapter, provider, data, or execution logic into Python. Python adapter modules configure Rust adapters and register factories; Rust owns adapter operations. If worker-side Rust work needs a Python callback, route it through a specific event type that belongs in the live runner. When adding Python-aware live code: - Prefer an existing runner event channel. - Keep callback bodies short because they run synchronously on the live event loop. - Do not call `Python::attach` from Tokio worker tasks in Python live trading. - Do not add adapter business logic in Python to fit callback routing. ### Test naming Use descriptive names that explain the scenario. Keep tests as annotated pytest free functions: ```python def test_write_and_query_option_greeks_round_trip() -> None: ... def test_catalog_loaded_greeks_reach_on_option_greeks() -> None: ... def test_backend_session_rejects_zero_chunk_size() -> None: ... ``` ### Ruff [Ruff](https://astral.sh/ruff) is used to lint the codebase. Its rules are configured in `python/pyproject.toml`, with ignore justifications typically commented. # Python Adapter Interface Source: https://nautilustrader.io/docs/latest/developer_guide/python_adapters/ Use this guide to implement a custom live adapter in a separate Python package, optionally with its own Rust/PyO3 extension. It covers the client interface, registration, and runtime contracts. For the platform overview, see [Adapters](../concepts/adapters.md). For in-tree Rust adapters, see [the Rust adapter guide](adapters.md). Custom clients run on the node's Python event loop and exchange typed objects with the Rust engines through PyO3. Subclass a client from `nautilus_trader.live.clients`: | Base class | Responsibility | | ------------------ | -------------------------------------------------------------------- | | `DataClient` | Data subscriptions and historical requests. | | `MarketDataClient` | Naming specialization of `DataClient`; identical hooks and behavior. | | `ExecutionClient` | Order commands, account updates, and reconciliation reports. | Start with the [deterministic adapter template](../../examples/live/_template/README.md). The [independent Rust/PyO3 package section](#independent-rustpyo3-packages) shows how an extension uses the same interface. ## Registration and configuration ### Define a factory Subclass `DataClientFactory` or `ExecutionClientFactory` from `nautilus_trader.live.clients`. Register either the subclass with a static `create` method or an instance of it. Data factories implement `create(*, name, config, cache, clock)`. Execution factories also receive `trader_id`. The factory returns the appropriate client subclass **synchronously**. The client retains the supplied config and cache objects. Before accepting it, the node validates its registration name and execution identity. Run this example from the repository root to register the template's data client: ```python from nautilus_trader.config import DataClientConfig from nautilus_trader.live import LiveNodeBuilder from nautilus_trader.live.clients import DataClientFactory from examples.live._template.constants import VENUE from examples.live._template.data import TemplateDataClient from examples.live._template.providers import TemplateInstrumentProvider class ExampleFactory(DataClientFactory): @staticmethod def create(*, name, config, cache, clock): return TemplateDataClient( name=name, config=config, cache=cache, clock=clock, venue=VENUE, instrument_provider=TemplateInstrumentProvider(config.instrument_provider), ) builder = LiveNodeBuilder.from_config("example") builder = builder.add_data_client("TEMPLATE", ExampleFactory, DataClientConfig()) ``` The template also supplies an execution client and its factory. Keep node configuration and strategies outside the adapter package. ### Register clients with the node For **direct registration**, use `LiveNodeBuilder.add_data_client(name, factory, config)` or `add_exec_client(...)`. An explicit fourth `routing` argument overrides `config.routing`. For **configuration-based registration**: 1. Put name-to-config mappings in `LiveNodeConfig(data_clients=..., exec_clients=...)`. 1. Supply `data_factories` and `exec_factories` to `LiveNode.build(name, config, ...)` or `LiveNodeBuilder.from_config(name, config, ...)`. Factory lookup checks the complete client name, then its prefix before the first hyphen. A supplied factory takes precedence over an importable config's factory descriptor. Venue and default routing use each config's `RoutingConfig` through the same builder path as direct registration. Both registration paths preserve these rules: - **Native and custom clients can share a node.** Registration distinguishes native factories from Python factories; custom configs do not pass through a native config downcast. - **Duplicate names fail.** Each registered client needs a distinct name. - **Failed builds preserve Python client registrations.** The node disposes clients from the failed attempt, then a retry creates fresh client instances. ### Define adapter configuration Subclass `DataClientConfig` or `ExecutionClientConfig` for adapter fields. The inherited native constructor handles common fields. Declare adapter fields as keyword arguments in a Python `__init__` and accept `**kwargs` for common fields. The factory receives the original subclass instance. Import `ImportableConfig` and `ImportableFactoryConfig` from `nautilus_trader.config` when configs or factories need to be loaded by import path. | Operation | Result | | ---------------------------------------- | ------------------------------------------------------- | | `config.dict()` | Config fields with their Python values retained. | | `config.json()` | JSON bytes. | | `config.to_importable(factory)` | `ImportableConfig` with a `module:qualified_name` path. | | `ImportableFactoryConfig(path).create()` | Factory imported and constructed without arguments. | `ImportableConfig.json()`, `parse()`, and `create()` provide an importable round trip. Classes defined inside functions are not importable. ### Serialize configuration Keep runtime resources out of config objects. Instance fields, including underscore-prefixed fields, participate in serialization; `ClassVar` metadata does not. JSON serialization supports: - Native JSON values. - Common routing and instrument-provider configs. - `Decimal` values encoded as strings. - Domain values with `from_str`. Annotate adapter fields to restore domain values and decimals, including values inside lists, dictionaries, and optional types. `Any` retains the decoded JSON value. Unsupported values raise `TypeError`. :::warning[Serialized credentials] Serializing a config also serializes any credentials it contains. Protect the serialized result. ::: ## Startup, scheduling, and shutdown ### Bind resources at startup Constructors and factories **must not start tasks, capture an event loop, or open network resources**. `client.loop` is `None` during construction. A v1-style factory that declares a `loop` parameter receives `None`. Startup binds every custom client to the actual running loop before `_connect`. Create loop-dependent sessions, locks, and network clients inside that async hook. ### Choose a launch mode | Launch mode | Event loop | Signal ownership | | -------------------------------- | ------------------------------------------------- | ------------------------------------------ | | `node.run()` with custom clients | Creates and drives asyncio on the calling thread. | Restores signal handlers after completion. | | `await node.run_async()` | Uses the host's running loop. | Host application owns its signals. | Both modes preserve the Rust node's startup, reconciliation, maintenance, and shutdown path. Native-only `run()` retains its native execution path. Calling blocking `run()` from an active event loop raises before consuming the node. Use `run_async()` inside an existing loop. :::warning[Database cache backing] Redis/PostgreSQL cache backing is unsupported with custom Python clients in either launch mode. Both modes drive asyncio, where the backing's blocking worker calls would stall the event loop. Native-only nodes can use database backing with `run()`. ::: ### Schedule background work Use `self.create_task(coroutine, name)` for adapter background work. The runtime: - Retains the client and every task until terminal result retrieval. - Defers the first coroutine poll, including with an eager task factory. - Logs unawaited operation failures with the client and operation name. Tasks created directly through asyncio or an external library remain that code's responsibility. Join them during `_disconnect`. ### Process commands in order Commands and subscription changes share a **FIFO queue** with capacity for **1,024 waiting operations per client**. A full queue rejects further admission. An operation failure is logged before the next queued operation runs. Historical requests and awaited reconciliation operations run separately, so network waits do not block command admission. Keep each command hook bounded: it delays later commands on that client. ### Disconnect and drain tasks The node's post-stop grace period precedes disconnection, allowing strategy shutdown cancellations to enter the normal command path. Disconnection then: 1. Closes admission and discards queued commands with diagnostics. 1. Cancels the active command. 1. Calls `_disconnect`, where the adapter closes network resources and joins adapter-owned work. The node bounds disconnection by its configured timeout. The runtime requests cancellation **at most once per task** and shields tasks while draining them, so repeated supervisor cancellation does not interrupt asynchronous cleanup. A deadline reports incomplete cleanup rather than issuing another cancellation. Independently cancelled commands allow the next queued operation to run while admission remains open. :::warning[Cancellation is not completion] Resistant tasks remain supervised, and incomplete cleanup is reported, including when a host closes the loop too early. Finish awaiting the node before closing a host loop. ::: Disposal invalidates output and cache views, so late work cannot emit into a subsequent node. A client instance and its bound provider belong to **one node run**. ## Read-only cache and typed output ### Inspect core state Factories receive `ClientCache`, a **read-only view** of the owning core cache. Instruments, orders, accounts, positions, books, and query collections are owned snapshots. Mutating a returned object does not change core state. No mutable cache, engine, or raw dispatch handle is passed to an adapter. Cache access raises a Python exception when it occurs: - On another thread. - After disposal. - During an incompatible core borrow. The read methods include: | State | Methods | | ---------------------- | --------------------------------------------------------------------- | | Instruments | `instrument`, `instruments`, `instrument_ids` | | Quotes and books | `quote`, `order_book` | | Accounts and positions | `account`, `position`, `positions_open` | | Orders | `order`, `order_list`, `orders`, `orders_open`, `orders_inflight` | | Order queries | `orders_open_count`, `client_order_ids_open`, order identity mappings | | Stored bytes | `get` | Use `quote(instrument_id, index=0)` for quote history. `get` returns a list of byte values, consistent with the v2 cache binding. ### Send typed output Client output methods **enqueue typed values for core processing**. The core remains the sole writer of trading state. | Output | Client methods | | ---------------------- | ------------------------------------------------------------- | | Instrument definitions | `_handle_instrument` | | Streaming data | `_handle_data` | | Historical responses | `_handle_response` | | Execution events | `generate_account_state`, `generate_order_*`, `_handle_event` | | Reconciliation reports | `_handle_report` | `_handle_data` accepts these types from `nautilus_trader.model`: - **Quotes, trades, and bars**: `QuoteTick`, `TradeTick`, `Bar`. - **Order books**: `OrderBookDelta`, `OrderBookDeltas`, `OrderBookDepth10`. - **Prices and funding**: `MarkPriceUpdate`, `IndexPriceUpdate`, `FundingRateUpdate`. - **Options and instrument events**: `OptionGreeks`, `InstrumentStatus`, `InstrumentClose`. - **Custom data**: `CustomData`. `_handle_report` accepts `OrderStatusReport`, `FillReport`, `PositionStatusReport`, or `ExecutionMassStatus` from `nautilus_trader.model`. Its optional `fills` list accompanies an `OrderStatusReport` only. Report account identity, mass-report client/venue identity, and nested report ownership must match the registered client. ## Data client hooks ### Subscriptions and requests Each hook is async and receives an owned, frozen command or request from `nautilus_trader.live`. Nested params are copied. Override the hooks for the venue's supported capabilities; unimplemented hooks raise `NotImplementedError`. Track venue subscription acknowledgments and reconnect replay in adapter-local state. | Family | Subscribe/unsubscribe suffix | Historical request suffix | | ----------------- | --------------------------------------- | -------------------------------------------- | | Custom data | No suffix | `data` | | Instruments | `instruments`, `instrument` | `instruments`, `instrument` | | Order books | `book_deltas`, `book_depth10` | `book_snapshot`, `book_deltas`, `book_depth` | | Quotes and trades | `quotes`, `trades` | `quotes`, `trades` | | Reference prices | `mark_prices`, `index_prices` | *Not supported* | | Funding | `funding_rates` | `funding_rates` | | Bars | `bars` | `bars` | | Instrument events | `instrument_status`, `instrument_close` | *Not supported* | | Options | `option_greeks` | `option_chain_reference_price` | For example, quote subscriptions call `_subscribe_quotes(command)` and `_unsubscribe_quotes(command)`; quote history calls `_request_quotes(request)`. ### Historical responses Preserve request IDs, params, time bounds, and payload identity when constructing the matching typed response. Response classes in `nautilus_trader.live` are: - **Custom data and instruments**: `CustomDataResponse`, `InstrumentResponse`, `InstrumentsResponse`. - **Order books**: `BookResponse`, `BookDeltasResponse`, `BookDepthResponse`. - **Quotes and trades**: `QuotesResponse`, `TradesResponse`. - **Funding and bars**: `FundingRatesResponse`, `BarsResponse`. - **Options**: `OptionChainReferencePriceResponse`. An **empty response still completes the adapter's response path**. Windowed requests expose UTC `start`/`end` datetimes and exact `start_ns`/`end_ns`; response bounds use integer nanoseconds. `RequestBookSnapshot` and `RequestOptionChainReferencePrice` have no time-window fields. ### Instrument providers `InstrumentProvider` stores adapter-local instruments and currencies. Override `load_all_async`, `load_ids_async`, or `load_async`. `initialize(reload=False)` uses `InstrumentProviderConfig` and retries after a failed load. The synchronous loading methods depend on the provider's binding: - **Bound to a client**: schedule supervised work. - **Unbound, outside an active event loop**: run the load synchronously. - **Unbound, inside an active event loop**: await the async method instead. Send loaded instruments through the client output to populate the core cache. ## Execution client hooks ### Construct the client Pass the factory's `name`, `config`, `cache`, `clock`, and `trader_id` to `ExecutionClient`. Execution clients also require these non-optional values from `nautilus_trader.model`: | Argument | Type | | -------------- | ------------- | | `venue` | `Venue` | | `account_id` | `AccountId` | | `account_type` | `AccountType` | | `oms_type` | `OmsType` | `base_currency` and `instrument_provider` are optional. Data clients can use `venue=None`. ### Generate reconciliation reports Implement the reconciliation hooks: - `_generate_order_status_report` - `_generate_order_status_reports` - `_generate_fill_reports` - `_generate_position_status_reports` Return typed reports, lists, or the optional result prescribed by the method. `_generate_mass_status(lookback_mins)` defaults to native composition of the bulk methods using the owning node's clock; an override returns an `ExecutionMassStatus`. **Propagated report failures fail startup reconciliation.** ### Handle order commands Override the hooks supported by the venue: - **Submit**: `_submit_order`, `_submit_order_list`. - **Modify**: `_modify_order`. - **Cancel**: `_cancel_order`, `_cancel_all_orders`. - **Query**: `_query_account`, `_query_order`. `_batch_modify_orders` and `_batch_cancel_orders` default to ordered calls of the corresponding single-order hook. Override them for a venue batch endpoint. Commands preserve native identifiers and params, correlation/causation IDs, timestamps, and nested orders or batch members. ### Make synchronous decisions and receive notifications `_handles_order_venue`, `_provides_bulk_position_coverage`, and `_calculate_commission` are synchronous decision hooks. They **must return promptly without I/O**. Commission returns `Money` or `None` to use the core fallback. `position_reconciliation_tolerance` accepts a nonnegative `Decimal` at construction. `_on_instrument` and `_register_external_order` are async notifications admitted to the command queue. Registration timestamps come from reconciliation metadata, not necessarily the order's initialization event. ## Migration from v1 The interface restores live adapter capabilities using v2 names and ownership rules. It does not make an unchanged Cython adapter source-compatible. | V1 surface | V2 replacement | Migration detail | | ----------------------------------- | ----------------------------------- | ------------------------------------------- | | `LiveDataClient` | `DataClient` | Import from `nautilus_trader.live.clients`. | | `LiveMarketDataClient` | `MarketDataClient` | Same module. | | `LiveExecutionClient` | `ExecutionClient` | Same module. | | `quote_ticks` / `trade_ticks` hooks | `quotes` / `trades` hooks | Typed v2 commands. | | `order_book_*` hooks | `book_*` hooks | Depth subscription uses `book_depth10`. | | `_request` | `_request_data` | Custom-data request. | | `_handle_*` history methods | `_handle_response(typed_response)` | Preserve correlation and bounds. | | `_send_*` execution methods | `_handle_event` / `_handle_report` | Validates owner identity. | | Cache writes | Queued instruments and events | Core applies changes. | | Subscription tracking methods | Adapter-local subscription state | Update from venue acknowledgments. | | Constructor event loop | `client.loop` after startup binding | Create resources in `_connect`. | | `cancel_pending_tasks` | Supervised node disconnection | Cancellation is not terminal completion. | | `run_after_delay` | Coroutine with `asyncio.sleep` | Schedule through `create_task`. | ### Remaining differences and limitations - **Forward prices**: use `RequestOptionChainReferencePrice` and its matching response for a v2 option series. - **DeFi**: native block/pool subscriptions and pool snapshots are outside this Python interface. - **Historical requests**: the [known migration limitations](../../MIGRATION_V2.md#known-limitations) for request joining/completion still apply. - **Custom publication and subscription**: the same migration limitations apply. Clients emit `CustomData` through typed data output; they do not expose component `publish_message` or topic subscription methods. - **Database cache backing**: unsupported with custom Python clients in either launch mode. See [startup](#startup-scheduling-and-shutdown) and [hosted event loops](../concepts/live.md#hosted-event-loops). - **Revised bars**: config retains `handle_revised_bars`, but the v2 core lacks the v1 bar revision marker and revision overwrite behavior. - **Networking**: this interface does not restore removed HTTP/WebSocket bindings. ## Independent Rust/PyO3 packages An adapter can ship its venue implementation in a separate Rust/PyO3 extension. Its Python facade subclasses the same client bases and registers through the same factories as a pure Python adapter. Compile the extension against PyO3 and import NautilusTrader model classes from the installed wheel. :::warning[Use the installed wheel's model classes] All exchanged domain values must be instances of those classes. Do not link a second set of Nautilus model pyclasses or exchange native Rust trait objects across extension modules. The integration boundary is the Python protocol, not a shared Rust ABI. ::: ### Package layout and dependencies Keep the adapter in its own project, with this layout: ```text external-adapter/ Cargo.toml pyproject.toml src/lib.rs external_adapter/__init__.py ``` Configure the library as a Python extension. For example, `Cargo.toml` can contain: ```toml [package] name = "external-adapter" version = "0.1.0" edition = "2024" publish = false [lib] name = "_backend" crate-type = ["cdylib"] [dependencies] pyo3 = { version = "=0.29.2", features = ["extension-module"] } [workspace] ``` The empty workspace keeps Cargo resolution independent if the project sits beneath another Cargo workspace. Keep the adapter's dependency lockfile and audit policy in its own project. Configure the Python package in `pyproject.toml`: ```toml [build-system] requires = ["maturin==1.15.0"] build-backend = "maturin" [project] name = "external-adapter" version = "0.1.0" requires-python = ">=3.12,<3.15" dependencies = ["nautilus-trader"] [tool.maturin] module-name = "external_adapter._backend" python-source = "." ``` These build versions match the demonstrated PyO3 boundary. Set the NautilusTrader dependency range to the releases covered by your adapter's tests. ### Exchange wheel-owned objects Use `Bound<'_, PyAny>` for incoming commands and `Py` for objects returned to Python. Import classes while attached to Python, read typed command attributes, and construct output with those imported classes. This method illustrates the conversion inside a `#[pymethods]` implementation: ```rust fn quote(&self, py: Python<'_>, command: &Bound<'_, PyAny>) -> PyResult> { let model = py.import("nautilus_trader.model")?; let values = PyDict::new(py); values.set_item("instrument_id", command.getattr("instrument_id")?)?; for (name, value) in [("bid_price", "1.12345"), ("ask_price", "1.12349")] { values.set_item( name, model.getattr("Price")?.call_method1("from_str", (value,))?, )?; } for (name, value) in [("bid_size", "17000"), ("ask_size", "23000")] { values.set_item( name, model.getattr("Quantity")?.call_method1("from_str", (value,))?, )?; } values.set_item("ts_event", 19)?; values.set_item("ts_init", 29)?; Ok(model.getattr("QuoteTick")?.call((), Some(&values))?.unbind()) } ``` Import `pyo3::{prelude::*, types::PyDict}` and expose the containing class from a `#[pymodule]` function named `_backend`. The example values are deterministic; a venue implementation supplies its received prices, sizes, and nanosecond timestamps. Construct prices, quantities, and money from exact decimal values or strings without a floating-point round trip. ### Delegate from Python client hooks The Python facade imports the backend and delegates from its client hooks. For a backend class named `Backend` with the method above, the quote hook is: ```python from external_adapter._backend import Backend async def _subscribe_quotes(self, command): self._handle_data(Backend().quote(command)) ``` Place that hook on a `MarketDataClient` subclass. Publish the instrument definition before its market data. Implement connection and disconnection hooks, and supply a `DataClientFactory` that constructs the subclass using the node-provided name, config, cache, and clock. The [Python template](../../examples/live/_template/README.md) demonstrates those hooks and factory registration. A backend with persistent venue state should be retained by its client. Execution adapters follow the same pattern with `ExecutionClient` and `ExecutionClientFactory`. Read `command.order` and emit submission, acceptance, rejection, and fill events through the client's `generate_*` methods. Return the documented report types from reconciliation hooks. Preserve client/account identity, request correlation, exact commission values, and event order; calling from Rust does not change these contracts. ### Lifecycle and cache ownership Keep Python calls and cache access on the node's owner thread and event loop. The GIL permits Python object access; it does not grant another thread permission to use the node's cache or output capability. The supplied cache is read-only, and returned mutable snapshots do not mutate the core. Send typed output through the client so the synchronous core applies state changes. The extension follows the same lifecycle as a Python adapter: - **Construction**: retain configuration without starting network work. - **Connection**: create loop-bound resources in `_connect`. - **Background work**: schedule Python tasks through `self.create_task`. - **Disconnection**: close resources in `_disconnect`. Stop any Rust workers or network tasks owned by the extension and retrieve their terminal results. Do not retain usable cache/output capabilities beyond node teardown. Propagate failures as Python exceptions so the client runtime can supervise them. ### Build and verify installed wheels Install maturin into a compatible CPython build interpreter first. If Cargo would discover a different interpreter, select the build interpreter with `PYO3_PYTHON`. From the adapter's own project, build a wheel with bounded Cargo concurrency: ```bash CARGO_BUILD_JOBS=4 python -m maturin build --out dist ``` Install the adapter wheel and a standard, non-editable NautilusTrader wheel into a fresh environment. Use absolute wheel paths when running these commands from outside both source trees: ```bash python -m venv /tmp/adapter-check /tmp/adapter-check/bin/python -m pip install /absolute/path/nautilus_trader.whl /absolute/path/adapter.whl /tmp/adapter-check/bin/python -I /absolute/path/verify_adapter.py ``` Replace the wheel placeholders with the complete generated wheel filenames. On Windows, use the virtual environment's `Scripts/python.exe` path. Your verifier should check: - **Import isolation**: `nautilus_trader`, the adapter package, and its backend resolve beneath `sys.prefix`. Editable imports can hide packaging or extension-identity defects. - **Launch modes**: both `node.run()` and hosted `run_async()` work. - **Data delivery**: a quote reaches the strategy and core cache. - **Reconciliation**: reconciliation completes. - **Execution**: a submitted order produces the expected fill quantity, price, commission, and final cached order state. - **Shutdown**: resources close and further adapter output is prevented. Run this installed-wheel check when changing the adapter protocol or the extension's supported NautilusTrader versions. # Releases Source: https://nautilustrader.io/docs/latest/developer_guide/releases/ This guide covers the release process and the standards for writing release notes. ## Overview NautilusTrader uses a three-branch model: - **`develop`**: active development; publishes dev wheels to Cloudflare R2 on every push. - **`nightly`**: pre-release testing; publishes all supported pre-release wheels and CLI binaries. - **`master`**: stable releases; triggers the full release pipeline. Merging a release commit to `master` automatically tags the version from `python/pyproject.toml`, creates a draft GitHub release, uploads release assets, publishes Cargo crates to crates.io, publishes wheels and sdist to PyPI, publishes the GitHub release, builds Docker images, and triggers a docs rebuild. ## Stable release workflow The `build` workflow treats the GitHub release as the anchor for stable releases. It creates the release as a draft first, uploads the wheel and sdist assets to that draft release, and only then publishes those packages to package indexes. The workflow publishes the GitHub release only after the registry verification and final integrity assets are complete. ```mermaid flowchart TD push["Push to master"] wheels["Build wheel artifacts
Linux x86/ARM, macOS, Windows"] audits["Release gates
Rust suite + cargo-deny + cargo-vet
Cargo publish + docs/features preflights"] security["security-audit
Zizmor + supply chain"] tag["tag-release
Create tag and draft GitHub release"] wheel_assets["publish-wheels-master
Upload wheels to GitHub release and R2
release env"] build_sdist["build-sdist
Build sdist workflow artifact"] sdist_asset["upload-sdist-release
Upload sdist to GitHub release"] crates["publish-cargo-crates
crates.io Trusted Publishing
release env"] wheel_pypi["publish-wheels-pypi
Attest and publish wheels to PyPI
release env"] sdist_pypi["publish-sdist-pypi
Attest and publish sdist to PyPI
release env"] integrity["publish-release-integrity
Checksums and registry verification
Attestation siblings and cleanup"] publish_release["publish-github-release
Publish draft release
Verify release attestation"] push --> wheels push --> audits push --> security wheels --> tag audits --> tag security --> tag tag --> build_sdist build_sdist --> sdist_asset tag --> sdist_asset tag --> wheel_assets wheels --> wheel_assets sdist_asset --> wheel_assets wheel_assets --> wheel_pypi wheel_assets --> crates wheel_pypi --> sdist_pypi sdist_asset --> sdist_pypi crates --> integrity wheel_pypi --> integrity sdist_pypi --> integrity tag --> integrity integrity --> publish_release tag --> publish_release ``` Keep these sequencing rules intact when editing `.github/workflows/build.yml`: - The draft GitHub release must exist before any release asset upload or package registry publish. - `tag-release` must depend on `security-audit` so stable release tagging cannot proceed after an audit failure. - Wheel and sdist assets must be attached to the GitHub release before package index publishing starts (`packages.nautechsystems.io`, PyPI, crates.io). - PyPI and crates.io Trusted Publishing jobs must keep `environment: release` and `id-token: write`; those registrations depend on the `release` environment. - Non-OIDC integrity and asset-upload jobs should avoid `environment: release` unless they need release environment secrets or approvals. - `publish-release-integrity` must run after PyPI and crates.io publishing. It generates the release manifest first, verifies registries against that manifest, then attaches final integrity assets only after verification passes. - `publish-github-release` must be the final stable release job. GitHub recommends creating a draft release, attaching all assets, then publishing the draft before enabling release immutability. Once GitHub release immutability is enabled for the repo, published release assets and the release tag cannot be changed; only the title and release notes remain editable. The job verifies the final draft asset set before publishing and verifies GitHub's release attestation after publishing the draft. ## Versioning The project maintains two version numbers: | File | Scope | | ------------------------ | -------------- | | `python/pyproject.toml` | Python package | | `Cargo.toml` (workspace) | Rust crates | These are bumped independently. The Python version drives the `v` release tag. Versions ending in `aN`, `bN`, or `rcN` create a GitHub pre-release; final versions create a normal release. ## Crates.io publishing The `build` workflow publishes Cargo crates from the `publish-cargo-crates` job. The job uses crates.io Trusted Publishing through GitHub Actions OIDC, so it does not use a persistent cargo token. Configure each crate on crates.io with: | Field | Value | | ----------- | ----------------- | | Owner | `nautechsystems` | | Repository | `nautilus_trader` | | Workflow | `build.yml` | | Environment | `release` | Enable Trusted Publishing Only for crates after their trusted publisher is configured. Crates that have never been published still need an initial manual publish before crates.io allows the trusted publisher configuration. Do not use `cargo publish --workspace` for CI releases. The release job runs `scripts/ci/publish-cargo-crates.sh`, which publishes crates one at a time in dependency order, skips versions already present on crates.io, and waits for each new version to appear in the crates.io API and sparse index before publishing dependents. The script fails before uploading if a publishable crate depends on a local `publish = false` crate that is absent from crates.io. Optional local dependencies count as blockers because publishing a public feature that resolves to an absent crate would leave that feature unusable. Post-publish verification treats an existing crate version as `previously_published` only when crates.io shows it was trusted-published by this repository. It still fails for user-published crate versions unless `CRATES_IO_MANUAL_PUBLISH_EXCEPTIONS` names each recovered `crate@version` entry for emergency token-publish recovery. Accepted manual entries are recorded in `crates-manifest.json` with `release_status: "manual_token_publish"`, and malformed or unused exception entries fail the job. Wrong trusted-publishing repositories and checksum or sparse-index mismatches also fail. ## Release checklist ### Pre-release (on `develop`) - [ ] Finalize `RELEASES.md`: review all items, remove empty sections - [ ] Ensure versions are set in `python/pyproject.toml` and the `Cargo.toml` workspace - [ ] Ensure crates.io Trusted Publishing is configured for every crate that CI publishes: `bash scripts/ci/check-crates-io-trusted-publishing.sh` - [ ] Ensure all CI checks pass on `develop` ### Release - [ ] Merge `develop` into `nightly`, verify nightly CI passes - [ ] Merge `nightly` into `master` - [ ] Verify the `build` workflow completes: - Wheels built for Linux x86/ARM, macOS, Windows - `cargo-deny` and `cargo-vet` pass - `security-audit` passes its Zizmor and supply-chain checks - Release docs/features and Cargo publish preflights pass before tagging - Tag and draft GitHub release created - Wheels and sdist attached to the GitHub release before package registry publishing - Cargo crates published to crates.io or skipped because the version already exists - Wheels and sdist published to PyPI - Registry verification passes before release checksums, crates manifest, and attestation siblings are attached - GitHub release published after all release assets and integrity assets are attached - [ ] Verify the `docker` workflow completes (images built and pushed) - [ ] Verify the `build-docs` workflow completes (docs rebuild triggered) ### Post-release (on `develop`) - [ ] Update the release date in `RELEASES.md` for the published version - [ ] Add horizontal separator `---` below the completed release - [ ] Add the next version template at the top of `RELEASES.md` (see below) - [ ] Bump `python/pyproject.toml` version to the next release number - [ ] Bump crate versions in tutorial and how-to `Cargo.toml` snippets (`docs/concepts/rust.md`, `docs/how_to/run_rust_backtest.md`, `docs/how_to/run_rust_live_trading.md`) ## Release notes This section documents the standards for writing release notes in `RELEASES.md`. ### Sections Use the following sections in this order: 1. Enhancements 2. Breaking Changes 3. Security 4. Fixes 5. Internal Improvements 6. Documentation Updates 7. Deprecations Omit sections that have no items for a given release. ### Enhancements New features and user-visible improvements. **Format**: ```markdown - Added `subscribe_order_fills(...)` and `unsubscribe_order_fills(...)` for `Actor` - Added BitMEX conditional orders support - Added support for `OrderBookDepth10` requests (#2955), thanks @faysou ``` **Guidelines**: - Start with "Added". - Use backticks for code elements. - Be specific about what was added, not how. ### Breaking Changes Changes that may break existing code. **Format**: ```markdown - Removed `nautilus_trader.analysis.statistics` subpackage - must import from `nautilus_trader.analysis` - Renamed `BinanceAccountType.USDT_FUTURE` to `USDT_FUTURES` - Changed `start` parameter to required for `Actor` data request methods ``` **Guidelines**: - Start with "Removed", "Renamed", or "Changed". - Explain migration path briefly. ### Security Security hardening and fixes that prevent crashes, undefined behavior, or data corruption. Includes significant hardening improvements elevated from Internal Improvements. **Format**: ```markdown - Fixed divide-by-zero and overflow bugs in model crate that could cause crashes - Fixed core arithmetic operations to reject NaN/Infinity values and improve overflow handling ``` **Guidelines**: - Include overflow/underflow fixes, memory safety improvements, FFI guards, data integrity fixes. - Focus on user impact: what could have happened. - Exclude routine dependency updates, minor hardening, or test-only fixes. - Omit this section entirely if there are no security items for the release. ### Fixes Bug fixes that improve correctness but don't qualify as security issues. **Format**: ```markdown - Fixed reduce-only order panic when quantity exceeds position - Fixed Binance order status parsing for external orders (#3006), thanks for reporting @bmlquant ``` **Guidelines**: - Start with "Fixed". ### Internal Improvements Implementation details and infrastructure changes. **Format**: ```markdown - Added ARM64 support to Docker builds - Ported `PortfolioAnalyzer` to Rust - Improved clock and timer thread safety - Upgraded Rust (MSRV) to 1.90.0 - Upgraded `pyo3` crates to v0.26.0 ``` **Guidelines**: - Use "Added", "Implemented", "Improved", "Optimized", "Upgraded", "Refined", "Standardized". - Include version numbers for dependency upgrades. ### Documentation Updates Changes to guides and examples. **Format**: ```markdown - Added rate limit tables with links to official docs - Improved dark and light themes for readability - Fixed broken links ``` ### Deprecations Features marked for removal. **Format**: ```markdown - Deprecated `some_config_option`; disable (`False`) to maintain consistent behavior. Will be removed in future version ``` **Guidelines**: - Explain migration path and provide alternatives. ## Attribution - Credit external contributors: `thanks @username` or `thanks for reporting @username`. - Include issue/PR numbers for community contributions and complex features: `(#1234)`. ## Style - Use sentence case (capitalize first word only). - Do not end with periods. - Use backticks for code elements. - Focus on **what** changed, not how. **Be specific**: ```markdown Bad: Improved Binance adapter Good: Improved Binance fill handling when instrument not cached ``` ## Security classification Include in Security if the change addresses: - Memory safety (overflow, underflow, divide-by-zero that threatens stability). - Undefined behavior or crashes that could corrupt state. - Data integrity (NaN/Infinity propagation, race conditions leading to corruption). - Input validation preventing injection or exploitation (SQL injection, command injection, path traversal). - Build hardening (non-exec stack, FFI guards). - Significant hardening that users should know about. Otherwise use Fixes (for logic bugs and panics) or Internal Improvements (for minor hardening). Note: Plain logic panics belong in Fixes unless they threaten system stability or data corruption. ## Examples **Security** (could cause crashes/corruption): ```markdown - Fixed divide-by-zero in margin calculations that could crash the engine - Fixed integer overflow in model arithmetic that could crash the process ``` **Fixes** (incorrect but safe): ```markdown - Fixed Binance order status parsing for external orders - Fixed position purge logic to prevent purging re-opened position ``` **Enhancements** (user-facing): ```markdown - Added BitMEX conditional orders support ``` **Internal** (implementation): ```markdown - Implemented BitMEX ping/pong handling ``` ## Release notes template ```markdown # NautilusTrader Beta Released on TBD (UTC). ### Enhancements ### Breaking Changes ### Security ### Fixes ### Internal Improvements ### Documentation Updates ### Deprecations --- ``` # Runtime Conformance Contract Source: https://nautilustrader.io/docs/latest/developer_guide/runtime_conformance/ Use this reference to locate implementation boundaries and representative checks for selected [design principles](design_principles.md). The source baseline is commit `46f87cd1b7af576495418761bbf11db23e89124c`. Source links are relative to this document's revision; use that baseline when reproducing this snapshot. The entries describe Rust source and test coverage. They do not certify every adapter, Python entry point, configuration, or failure mode. The named tests are source references, not a record of a test run. ## Evidence and outcome provenance The [execution policies](../concepts/execution/policies.md#terminal-reconciliation-provenance) distinguish venue evidence from local policy resolution. In the Rust live execution manager, `check_inflight_orders` generates a rejection with reason `INFLIGHT_TIMEOUT` for a submitted order when the configured retry limit expires. Pending updates and cancellations instead generate `OrderCanceled`. These events carry `reconciliation=true`. - **Implementation**: [Execution manager](../../crates/live/src/execution/manager.rs), `check_inflight_orders`. - **Representative checks**: [Manager integration tests](../../crates/live/tests/integration/manager.rs), `test_inflight_order_generates_rejection_after_max_retries`, `test_inflight_pending_update_generates_canceled`, and `test_inflight_pending_cancel_generates_canceled`. - **Limit**: Retry exhaustion does not establish a venue outcome. The reconciliation flag alone does not distinguish a venue report from a local policy resolution, and `OrderCanceled` has no reason field. Consumers need the associated inputs and logs to retain that distinction. ## Callback ordering and ownership The [callback dispatch contract](callback_dispatch.md) requires publication order across recipients and exclusive component access. Private Rust primitives reserve publication order and reject overlapping checked access to an allocation. Production dispatch does not use these primitives. - **Implementation**: [Dispatch](../../crates/common/src/actor/dispatch.rs), `PublicationScope` and `drain`; [allocation access](../../crates/common/src/actor/access.rs), `AllocationGuard`. - **Representative checks**: `nested_publication_reserves_all_outer_recipients` in the dispatch module checks outer-recipient ordering across a nested publication. `test_actor_and_component_views_share_access` in the access module checks exclusion across views. - **Limit**: These checks do not establish production callback ordering or ownership safety. Runtime integration must end enclosing mutable borrows before draining and preserve native, Python, and dynamic-backend lifecycle eligibility. Unchecked access remains outside the private allocation guards. ## Recovery For a Rust live node with execution reconciliation enabled, startup performs reconciliation before starting trader components. A reconciliation error aborts startup. The startup integration test below supplies terminal order and fill reports through a test execution client and checks the recovered quantity, price, trade identity, commission, position quantity, and terminal status. - **Implementation**: [Live node](../../crates/live/src/node/mod.rs), `perform_startup_reconciliation` and its callers. - **Representative check**: [Node integration tests](../../crates/live/tests/integration/node.rs), `test_live_node_startup_recovers_terminal_fill_exactly`. - **Limit**: This check covers report-based startup recovery, not backing-store durability, supervisor restart, or arbitrary panic recovery. Reconciliation can be disabled; event-store replay also skips live client connection and reconciliation. Venue completeness remains subject to the [reconciliation policies](../concepts/execution/policies.md#reconciliation-authority). ## Overload handling The [live runner](../concepts/live.md#dispatch-priority-and-overload-behavior) uses unbounded message channels. Polling priority does not impose producer backpressure or a queue-depth limit. The private callback dispatcher separately enforces retained-count, known-storage, and callback-chain limits. - **Implementation**: [Runner](../../crates/live/src/runner.rs), `AsyncRunner::new` and `recv`; [dispatch](../../crates/common/src/actor/dispatch.rs), admission accounting and `drain`. - **Representative checks**: `test_recv_processes_system_event_before_command` in the runner module checks priority for that channel pair. `event_count_limit_latches_after_exact_capacity` and `progress_limit_persists_between_bounded_drains` in the dispatch module check private limits. - **Limit**: A polling-order test does not prove bounded latency or progress under sustained load. Private callback limits do not bound live runner queues or total process memory. Production overflow handling and safe drain boundaries remain integration requirements; queue monitoring supplies operational signals without automatically throttling feeds or stopping trading. # Rust Source: https://nautilustrader.io/docs/latest/developer_guide/rust/ This page defines NautilusTrader conventions for Rust source, Cargo manifests, PyO3 bindings, and tests. NautilusTrader uses Rust for its mission-critical core because the language combines a strong type system, an ownership model, and predictable performance. Safe Rust prevents data races and many memory errors at compile time. `unsafe` code must make explicit the invariants that the compiler cannot check. `rustfmt` and the workspace lints own general Rust style; the rules below supplement them. ## Sources of truth | Concern | Source | | -------------------------- | -------------------------------------------------- | | Formatting and imports. | `rustfmt.toml`. | | Workspace lints. | `Cargo.toml` and `clippy.toml`. | | Cargo layout. | `.pre-commit-hooks/check_cargo_conventions.sh`. | | Rust layout. | `.pre-commit-hooks/check_formatting_rs.sh`. | | Nautilus type conventions. | `.pre-commit-hooks/check_nautilus_conventions.sh`. | | Async boundaries. | `.pre-commit-hooks/check_tokio_usage.sh`. | | DST boundaries. | `.pre-commit-hooks/check_dst_conventions.sh`. | | PyO3 bindings. | `.pre-commit-hooks/check_pyo3_conventions.sh`. | | Anyhow. | `.pre-commit-hooks/check_anyhow_usage.sh`. | | Error names. | `.pre-commit-hooks/check_error_conventions.sh`. | | Logging. | `.pre-commit-hooks/check_logging_conventions.sh`. | | Rustdoc contracts. | `.pre-commit-hooks/check_docs_conventions.sh`. | | Test style. | `.pre-commit-hooks/check_testing_conventions.sh`. | | Workspace test selection. | `scripts/ci/check-workspace-test-coverage.sh`. | Match nearby code when the tools do not settle a choice. Change generator inputs and regenerate outputs instead of editing generated files directly. ## Module layout Arrange each hand-written module so its primary behavior appears before its supporting implementation details. ### Module roots and declarations Preserve the surrounding module-root style when adding child modules. In each contiguous declaration block in a `mod.rs` file, order out-of-line module declarations by these sections: 1. `#[macro_use]` modules. 1. Public modules. 1. Restricted modules such as `pub(crate)`. 1. Non-test `#[cfg(...)]`-gated modules. 1. Private modules. 1. Test-only modules. Alphabetize declarations within each section and leave one blank line between sections. The formatting hook enforces this order in `mod.rs` files. Use the narrowest visibility that serves the caller. The workspace denies unreachable `pub` items. ### Imports Keep imports at the top of the file or module. Use a local import only when its narrow scope materially improves clarity. ### Constants and global state Group module-wide constants and global state near the top of the module. Put `const` items before `static` and `thread_local!` declarations. Keep a narrowly used constant or static next to its consumers instead. ### Primary types and implementations Keep the primary type and its inherent implementation near the top of a module. ### Enums Place enums by role rather than collecting them in one module-wide block: - Keep a primary or public enum with the module's primary types. - Keep a small state enum next to the global whose state it represents. - Place a private supporting enum below its first caller. ### Supporting definitions Place private functions and types below their callers. In adapters, place private route types, decision enums, and parsing functions below the main client implementations. ### Box-style banner comments Do not add box-style banner or separator comments to divide module contents. Use modules and implementation blocks to express structure. The standard copyright and license header is the exception. ## Cargo manifests ### Dependencies and sections - Use workspace inheritance for shared dependencies, for example `serde = { workspace = true }`. Pin a version in a crate only when the dependency is not workspace-managed. - Separate dependency groups with a blank line and alphabetize each group. Manifests normally group internal `nautilus-*` crates, required external crates, and optional external crates, but preserve a manifest's meaningful local groups. - Keep optional crates in their own blank-line group, except when every crate in the group is a `nautilus-*` crate. - Keep the standard section order: package, lints, library, features, `cargo-machete` metadata, docs.rs metadata, dependencies, development dependencies, build dependencies, benches, binaries, examples, and tests. - Add `[lints] workspace = true` to every workspace crate with a library or binary target. - Keep adapter dependencies in the `# Adapter dependencies` section of the workspace `Cargo.toml`. Core crates must not depend on entries from that section. - Keep related dependency families compatible. The Cargo convention hook checks the known constraints, including `capnp` with `capnpc`, `arrow` with `parquet`, and `dydx-proto` with `prost` and `tonic`. - List only declared dependencies under `[package.metadata.cargo-machete] ignored`. - Remove a root `[workspace.dependencies]` entry when no crate uses it. Cargo tools kept only for CI are exempt from this check. - Remove a root `[workspace.package]` field when no crate inherits it. - List each `[workspace] members` entry as a literal path. The convention hook resolves member manifests directly and does not expand Cargo glob members. - Obtain `libfuzzer-sys` in adapter crates through `nautilus-live`; do not add it directly to an adapter manifest. When `crates/pyo3/Cargo.toml` groups adapters separately, keep its `# Adapters` block below the core internal crates. ### Package fields Crate `[package]` sections use this canonical prefix. Cargo infers `README.md` next to the manifest, so omit `readme`. The other fields shown are required. ```toml [package] name = "nautilus-example" version.workspace = true edition.workspace = true rust-version.workspace = true authors.workspace = true license.workspace = true description = "Example crate for NautilusTrader" categories.workspace = true keywords.workspace = true documentation.workspace = true repository.workspace = true homepage.workspace = true ``` Place the optional `publish`, `build`, and `include` fields after `homepage.workspace`. ### Features - Preserve the crate's existing default feature contract. Most core crates have empty defaults, while many adapter crates enable `high-precision` by default. - Include `"python"` in every `extension-module` feature that builds a Python artifact. Keep it next to `"pyo3/extension-module"`. - Propagate `high-precision` to dependent Nautilus crates that store or construct fixed-point domain values. - Document each public non-default feature once, in alphabetical order, under `## Feature flags` in the crate README and `# Feature Flags` in the crate-level library docs. ### Targets - Use snake_case filenames for `bin/` sources and kebab-case executable names, for example `path = "bin/ws_data.rs"` and `name = "hyperliquid-ws-data"`. - Set `doc = false` on binary and example targets. Also set `test = false` on binary targets. - Order library crate types as `rlib`, `staticlib`, then `cdylib` when a crate produces more than one type. ## File header requirements Copy the standard copyright and license header from a neighboring hand-written Rust file. Generated files retain their generator header instead. The copyright hook checks the year. Change a generator input and rerun the generator instead of editing generated Rust, C headers, Python stubs, or wrapper doc comments. ## Formatting and attributes `rustfmt` groups standard library, external crate, and local imports, then alphabetizes each group. Run the formatter instead of ordering imports by hand. Leave one blank line: - Between functions, including tests. - Above each `///` or `//!` doc comment. - Above standalone `if`, `match`, `for`, `while`, and `loop` expressions. - Above task spawn calls, including `spawn_local` and `spawn_blocking`. - Before a `let` that contains a multiline braced expression, unless it starts a block. - Before a multiline struct literal or a direct qualified `Type::new(...)` call, unless it starts a block. - After a statement that contains a multiline braced expression, before the next statement in the same block. The control-flow and spawn rules do not apply when the expression starts a block, continues the previous operation, or has an attached comment or attribute. Wrapped calls and method chains alone do not need a separator. Construction spacing also applies to assignments, explicit returns, and calls followed by `?` or `.await`. Construction must span multiple lines after rustfmt layout; one-line initializers stay together. Keep comments and attributes attached to their statement when inserting a separator. Skip macro bodies and `rustfmt::skip` regions. Apply the multiline statement rules around changed code; leave unrelated code alone. Add only blank lines that `rustfmt` preserves. The formatting hook: - Checks existing control-flow, spawn, and module-ordering rules at changed boundaries. - Compares staged and unstaged changes against `HEAD`, or against the merge base with `CHANGED_BASE_SHA` when set. - Falls back to checking all tracked Rust files when a CI base is unavailable. - Reads complete changed files for context and includes lines used by exemption checks when selecting diagnostics. - Does not modify files. These remain review conventions: - Multiline statement spacing. - Construction spacing. - `spawn_local` and `spawn_blocking` spacing. Keep changed code readable for a human reader, and add further blank lines when those separators still leave a dense block hard to follow. Use inline format arguments for existing variables: ```rust anyhow::bail!("Failed to subtract {n} months from {datetime}"); ``` Add `#[must_use]` to constructors, accessors, pure conversions, and consuming `with_*` methods when discarding the returned value is almost certainly a mistake. A return type such as `Result` already carries its own `must_use` annotation. When suppressing `missing_panics_doc` or `missing_errors_doc`, include a reason that explains why the lint does not apply: ```rust #[allow( clippy::missing_errors_doc, reason = "result type is retained for API compatibility but no error is returned", )] ``` ## Type qualification | Item | Convention | | ----------------------------- | --------------------------------------------------------------------------- | | `anyhow`. | Import only `anyhow::Context`; fully qualify macros and `Result`. | | Nautilus domain types. | Import the type, then use its short name. | | Tokio time, sync, and tasks. | Fully qualify the path; use `std::time::Duration`. | | `Debug` and `Display`. | Import the trait. | | `Formatter` and `fmt::Result` | Use `std::fmt::Formatter` and `std::fmt::Result` at the implementation. | | Nautilus macros. | Import `nautilus_actor!` or `nautilus_strategy!`, then call it unqualified. | Use `debug_struct(stringify!(TypeName))` in manual `Debug` implementations. Use `// nautilus-import-ok` only where a macro, generated path, or conditional import requires a fully qualified Nautilus type. Put the marker on the affected line or directly above the narrow block. ## Error handling and contracts ### Design by contract Use the narrowest mechanism that expresses the contract: | Situation | Mechanism | | -------------------------------------------------- | -------------------------------------------- | | Compile-time state or ownership rule. | Types, lifetimes, newtypes, and visibility. | | Public input precondition. | `check_*` from `nautilus_core::correctness`. | | Validated value construction. | `new_checked()` with a `new()` wrapper. | | Recoverable parse, I/O, or network failure. | `Result`. | | Internal invariant safe to omit in release builds. | `debug_assert!`, covered by a targeted test. | | Soundness or always-on invariant. | `assert!` or a checked error path. | Place an assertion where the code first relies on the invariant. Never use `debug_assert!` for public input validation or a soundness condition because release builds remove it. Prefix debug assertion messages with `Invariant:` and state the positive rule. The shared `Condition failed: ...` prefix marks a caller input violation; `Invariant: ...` marks an internal contract bug. ### Error boundaries Choose the error type at the API boundary: | Boundary | Return type | | ------------------------------------- | ----------------------------------- | | Reusable library or domain API. | A typed `Result`. | | Application or adapter orchestration. | `anyhow::Result`. | | Public input validation. | `CorrectnessResult` when suited. | - Define a typed error with `thiserror` when callers inspect or recover from the failure. - Bind error patterns and closures as `e`, not `err` or `error`. - Use `anyhow::bail!` for an early return from an `anyhow::Result` function. Use `anyhow::anyhow!` when an error value is required, such as inside `ok_or_else`. - Start `.context()` messages with lowercase text so chained errors read naturally. Preserve the capitalization of a leading proper noun or acronym. - Do not use `", got"` in an error or assertion. Use `", was"`, `", received"`, or `", found"` according to the context. ```rust parse_timestamp(value).context("failed to parse timestamp")?; connect().context("BitMEX websocket did not become active")?; ``` ### Panic policy Panics are valid for internal invariant violations that callers cannot reasonably recover from. Keep an API infallible when returning `Result` would only require callers to handle a programming defect. Use an always-on assertion for an invariant that protects soundness or prevents unsafe code from continuing with invalid state. Return `Result` at boundaries that accept untrusted input or can fail because of configuration, I/O, external data, task cancellation, or resource lifecycle. These are operational failures even when a specific call site expects them never to occur. For validated values, follow the [`new()` and `new_checked()` constructor pattern](#constructor-patterns): expose a fallible path for untrusted values and use the convenience wrapper where its caller contract permits a panic. Document reachable panics in public Rust APIs, and make each panic message name the violated invariant. Treat findings for these lints from `make clippy-strict-audit` as review prompts: - `clippy::panic` - `clippy::unwrap_used` - `clippy::expect_used` The audit also reports `clippy::panic_in_result_fn`, which the required workspace Clippy gate already enforces. Remove a panic when the failure is recoverable; retain a justified invariant panic, with a scoped lint reason when needed. The audit uses forced warnings, so its totals include deliberate sites with local lint allowances and diagnostics from macro expansions. ### Failure contract examples APIs with a documented panic contract use panics for: - Programmer errors (logic bugs, incorrect API usage). - Data that violates fundamental invariants (negative timestamps, NaN prices). - Arithmetic that would silently produce incorrect results. APIs return `Result` or `Option` when callers, including downstream crates, can handle a failure or absence, including: - Expected runtime failures (network errors, file I/O). - Business logic validation (order constraints, risk limits). - User input validation. The API determines how an invalid operation fails: ```rust let later = timestamp + duration_ns; // Panics on overflow. let price = Price::new_checked(f64::NAN, precision); // Returns Err. let later = timestamp.checked_add(duration_ns); // Returns None on overflow. ``` This policy is implemented throughout the core types (`UnixNanos`, `Price`, `Quantity`, etc.) and helps NautilusTrader maintain strong data correctness for production trading. The repository release profile sets `panic = "abort"`, so a panic terminates the process for a supervisor or orchestration system to handle. Downstream Rust binaries control their own release profile. ## Logging - Fully qualify log macros, for example `log::debug!` and `log::info!`. - Start messages with a capitalized word and omit terminal periods. - Use `// log-period-ok` on the call or within three lines above it when a terminal period is part of the logged value rather than sentence punctuation. - Keep connection and client lifecycle, reconnection, reconciliation, and mass-status summaries at `INFO`. - Keep subscription detail, per-order confirmations, instrument counts, authentication, and WebSocket internals at `DEBUG`. - Log a warning or error when an unexpected, user-actionable, or data-loss condition is handled locally. Do not log an error that continues to propagate through `?` or `anyhow::bail!`. - Leave a blank line above a log call unless it is the first line of the function. - Do not write to stdout or stderr or terminate the process from production library code. Binaries, examples, benches, tests, adapters, the CLI, and testkit may control their process when that is part of their role. ## Async code Synchronous core crates do not take Tokio as a regular dependency. The `common` crate keeps Tokio optional. The Tokio convention hook owns the exact crate list. Adapter production code uses the shared runtime because calls may arrive from Python threads without a thread-local Tokio context: ```rust use nautilus_common::live::get_runtime; get_runtime().spawn(async move { run_client().await; }); ``` - Import `get_runtime` through `nautilus_common::live`, not `live::runtime`. - Use `get_runtime().block_on()` when synchronous adapter code must call an async function. - Use the runtime supplied by `#[tokio::test]` in tests; `tokio::spawn()` is valid there. - Put `// tokio-import-ok` on an import or spawn line only when the fully qualified form or shared runtime cannot serve that site. - Install a custom runtime before `LiveNode::build()` or any adapter use. Build a multi-threaded runtime with all drivers enabled. - `set_runtime()` bypasses the default initializer, including Python initialization. With the `python` feature enabled, initialize Python before installing a custom runtime or keep the default runtime. Code on the deterministic simulation path follows the [DST determinism contract](../concepts/dst.md#determinism-contract). Route clocks, random values, task spawning, and network access through the project seams. Use `biased;` in `tokio::select!` blocks on that path. ## Runtime ownership and access ### Cache order access `Cache` hides its `SharedCell` order storage behind lifetime-scoped accessors. Use `order_ref()` or `try_order_ref()` for scoped reads, `order_mut()` for an exclusive write borrow, and `order_owned()` or `try_order_owned()` when a snapshot must cross a boundary. The `try_*` forms return `OrderLookupError` when the order is absent. `order_mut()` requires `&mut Cache`, so adapter-facing `CacheView` code cannot mutate orders. Drop an `OrderRef` or `OrderRefMut` before dispatching events or taking a borrow that can re-enter the same order, then look up the order again for post-event state. ### Runtime invariants The actor registry, component registry, and message bus use `thread_local!` storage. Objects registered on one thread are not visible from another. `LiveNodeHandle` is the intended cross-thread control surface. Keep an `ActorRef` within one synchronous scope. Do not store it in a struct, hold it across `.await`, or send it to another thread. Capture the actor ID in long-lived callbacks and look up the actor each time the callback runs. The component registry rejects aliased mutable access with a scoped borrow guard. Do not make component lifecycle operations re-entrant. ## Construction and conversion ### Constructor patterns Validated value types pair a fallible `new_checked()` with a convenience `new()`: ```rust pub fn new_checked>(value: T) -> CorrectnessResult { let value = value.as_ref(); check_valid_string_ascii(value, stringify!(value))?; Ok(Self(Ustr::from(value))) } pub fn new>(value: T) -> Self { Self::new_checked(value).expect_display(FAILED) } ``` Use the shared `FAILED` constant with `CorrectnessResultExt::expect_display` so panic messages use the standard `Condition failed: ...` prefix. Document the error on `new_checked()` and the panic on `new()`. For types with long constructors dominated by optional fields, use a `bon` builder instead of public positional `new()` and `new_checked()` methods. Put `#[bon::bon]` on the inherent implementation and make the builder's finish method delegate to a private validation function or perform the validation directly. Keep one validation and defaulting path. ```rust #[builder(start_fn = builder, finish_fn = build)] pub fn build_checked(/* same inputs as new_checked */) -> CorrectnessResult { Self::new_checked(/* forward inputs */) } ``` Required fields remain required in `bon` typestate. Optional fields remain omittable, and `build()` returns the same `CorrectnessResult` as the internal validation path. ### Conversion patterns - Implement `From` only for infallible, complete conversions. - Implement `TryFrom` when the source can contain an invalid or unrepresentable value. - Implement `FromStr` for string parsing. - Treat generic `From>` implementations that panic as compatibility surfaces. Do not copy that pattern into new APIs. - Keep venue wire enums distinct from Nautilus domain enums. Use idiomatic Rust variant names, express the wire spelling with `serde` or `strum` attributes, and convert explicitly at the boundary. - Deserialize adapter payloads into wire models before constructing domain objects. Keep parsing and validation in conversion functions instead of embedding venue wire details in domain types. ### Domain numeric types Preserve discrete financial values as decimals from ingestion: | Value | Type and construction | | ---------------------------------------- | -------------------------------------------------------- | | Price or quantity. | `Price::from_decimal_dp` or `Quantity::from_decimal_dp`. | | Money, fee, margin, or balance. | `Decimal`, then `Money::from_decimal` or `Money::zero`. | | Continuous signal ratio or timing curve. | `f64` when decimal precision has no domain meaning. | Do not route wire values through `f64` constructors. In tests, compare `.as_decimal()` with `dec!(value)`. ## Identifier storage ### How string interning works String interning stores one shared copy of each distinct string in a central cache. Repeated values refer to the same cached bytes instead of allocating another copy. Small handles make the values cheap to copy and compare, while a cached hash avoids reading the full string again during hashing. NautilusTrader uses `Ustr` for its interned identifier components. Each `Ustr` is a pointer-sized `Copy` handle with a precomputed hash and stable direct string access. Composite types such as `InstrumentId` preserve the same cheap copy semantics by storing these handles. ### Reclamation boundary The string cache retains every unique value for the process lifetime. This retention keeps copied handles and returned string slices valid without reference counting, access guards, or explicit lifetime parameters on identifier types. Process teardown is the normal reclamation boundary. These guarantees rule out safe reclamation of individual entries. Rust can copy a `Copy` value without executing code, so an atomic reference count cannot observe every copy. Designs that add reclamation change the identifier contract: - Reference counting requires `Clone` and `Drop`, which removes `Copy` from identifiers and types that contain them. - Borrowed or epoch-protected storage requires lifetimes or access guards at string access points. - Generational handles permit reclamation but make lookup fallible and invalidate stale handles. - A global cache reset is safe only at a proven quiescent point after all handles, references, and foreign pointers have been destroyed and no task or thread can retain one. ### Storage boundaries Interning is best suited to identifiers drawn from a bounded process-scoped universe and values that repeat enough to benefit from deduplication. Identifiers whose distinct values can grow with every order, trade, or message increase the cache for the process lifetime. Fixed-capacity inline storage retains `Copy` when the external protocol supplies a suitable maximum. `TradeId`, for example, uses a 36-character `StackStr`. Owned or reference-counted storage provides dynamic capacity when reclamation matters more than `Copy`. The domain model also contains compatibility exceptions. `ClientOrderId`, `VenueOrderId`, `PositionId`, and `OrderListId` remain `Ustr`-backed and therefore retain every distinct value. Identifier storage participates in the supported by-value C ABI, so a broader redesign depends on conversion-based bindings replacing raw layout sharing. The storage boundary includes an up-front estimate of every unique value and all intermediate strings interned during parsing. The cache is shared by every `Ustr` use in the process, so its memory cost is the aggregate set rather than a separate budget for each identifier type. ### Polymarket scale example A Polymarket instrument symbol combines a 66-byte condition ID with a 77- or 78-byte token ID, for a 144- or 145-byte interned symbol. With the 64-bit `ustr` 1.1.0 layout, 600,000 unique `InstrumentId` values require roughly 150 MiB for the retained identifier values, cache lookup table, and reserved string storage. The Polymarket parsing path also interns each raw token ID and each condition ID. For 600,000 instruments from about 300,000 markets, these entries raise the estimate to roughly 300 MiB before instrument objects, descriptions, maps, and other metadata. The estimate assumes unique instrument and token IDs and includes capacity reserved by the cache's geometric allocator, so it is not an exact resident-set measurement. NautilusTrader accepts this bounded cost to preserve `Copy`, stable direct access, and global deduplication across the instrument universe. Unbounded streams of unique external IDs remain outside this storage model. ## Collections Choose a hash collection by iteration semantics and trust boundary: | Requirement | Collection | | ---------------------------------------------- | ------------------------- | | Observable insertion-order iteration. | `IndexMap` or `IndexSet`. | | Hot lookup with no observable iteration order. | `AHashMap` or `AHashSet`. | | Keys chosen by an untrusted third party. | `HashMap` or `HashSet`. | | External API requires a standard collection. | `HashMap` or `HashSet`. | Default `AHashMap` and `AHashSet` instances randomize their hasher state, so their iteration order is not stable. When iteration order affects emitted events, commands and traffic sent to a venue, persisted output, or random number consumption, make the order deterministic. An iteration whose order no caller observes does not need stabilization. Use `IndexMap` or `IndexSet` when insertion order is both deterministic and the required sequence. Otherwise, keep the hash collection and sort at the point of use. Prefer sorting when ordered traversal is infrequent or order-preserving removal is hot. `shift_remove` preserves relative order in O(n) average time; `swap_remove` runs in O(1) average time but can move the last entry. `AHash` is not cryptographically secure. Use `HashMap` or `HashSet` where untrusted keys make hash-flooding resistance part of the security boundary. ## Documentation ### Coverage and tone - Add doc comments to public items. Document private behavior with a normal comment only when non-obvious context prevents a likely misreading. - Add module documentation to public modules and modules with a non-obvious contract. Do not add boilerplate to a private leaf module. - Use the indicative mood: "Returns the account ID", not "Return the account ID". - End public field and enum variant documentation with a period. - Match documentation density to neighboring items. Do not add filler comments to make a bare block look uniform. - Put important context for private fields in the type-level documentation instead of documenting each field. Document public functions when their contract is not fully clear from the type and name. Cover fallible conditions, panic conditions, safety obligations, and non-obvious input semantics. ### Rustdoc sections Use Title Case for Rustdoc section headings: - `# Examples` - `# Errors` - `# Panics` - `# Safety` - `# Notes` - `# Thread Safety` - `# Feature Flags` Use one sentence for a single error or panic condition: ```rust /// # Errors /// /// Returns an error if the currency conversion fails. ``` Use bullets with terminating periods for multiple conditions: ```rust /// # Errors /// /// Returns an error if: /// - The market price cannot be found. /// - The conversion rate calculation fails. ``` Use `# Errors` only on a function that returns `Result`, `PyResult`, or `Option`. Use `# Panics` only when the function can panic, and remove the section instead of saying that the function does not panic. The documentation hook recognizes direct panic sites in a `Result`-returning function. Put `// panics-doc-ok` immediately above the doc block when a called function supplies the documented panic. Use `// errors-doc-ok` in the same position only for a special error contract that the signature check cannot recognize. ### Doc examples `make cargo-test-doc` compiles and runs doc examples. Annotate every fence: | Fence | Behavior | Use for | | -------------- | ------------------------- | ------------------------------------------------------ | | `rust` | Compiled and run. | Self-contained examples with no external dependencies. | | `rust,no_run` | Compiled, not run. | Examples needing a catalog, network, or venue. | | `ignore` | Neither compiled nor run. | Pseudocode; state why it cannot compile. | | `compile_fail` | Must fail to compile. | Demonstrating a rejected usage. | | `text` | Not code. | Directory trees, output, or diagrams. | | `bash`, `json` | Not Rust. | Commands or payloads. | Prefer `no_run` to `ignore` when only the runtime dependency is unavailable. Prefix setup lines with `#` when they must compile but would obscure the rendered example. Put examples on public items. Rustdoc also collects fences from private item docs, but those examples cannot import the item they document. ## Python bindings ### PyO3 names and errors - Prefix a Rust function renamed with `#[pyo3(name = "...")]` with `py_`. - When a binding needs a Rust-only wrapper type, prefix it with `Py` and expose the Python name without that prefix. - Use `nautilus_trader.adapters.` for public adapter stub metadata. Runtime module paths use `nautilus_trader._libnautilus.`. - Convert standard Python exceptions with `to_pyvalue_err`, `to_pytype_err`, `to_pyruntime_err`, `to_pykey_err`, `to_pyexception`, or `to_pynotimplemented_err` from `nautilus_core::python`. The submodules registered in `crates/pyo3/src/lib.rs` are public API. Do not add an internal crate as a new Python submodule as a refactor side effect. Register an intentional class in an existing submodule when that preserves the public package structure. A deliberate submodule change also updates the allowlist in `check_nautilus_conventions.sh`. Keep each submodule registration as `let n = ""` followed by one `pyo3::wrap_pymodule!()` call. The name must match the final component of the target path. ### PyO3 enums Nautilus domain integer enums use `frozen`, `eq`, `eq_int`, `from_py_object`, and `rename_all = "SCREAMING_SNAKE_CASE"`. Do not add PyO3's `hash` attribute to an `eq_int` enum. Its generated hash differs from Python's hash for the equal integer discriminant and breaks the rule that equal values have equal hashes. Return the discriminant directly instead: ```rust #[pymethods] impl MyEnum { const fn __hash__(&self) -> isize { *self as isize } } ``` ### Type stub annotations Every Python-exposed type and function needs the matching `pyo3-stub-gen` annotation: | PyO3 construct | Stub annotation | | ----------------- | ----------------------------------------------- | | `#[pyclass]` | `pyo3_stub_gen::derive::gen_stub_pyclass`. | | Enum `#[pyclass]` | `pyo3_stub_gen::derive::gen_stub_pyclass_enum`. | | `#[pymethods]` | `pyo3_stub_gen::derive::gen_stub_pymethods`. | | `#[pyfunction]` | `pyo3_stub_gen::derive::gen_stub_pyfunction`. | - Put class and enum stub annotations in `#[cfg_attr(feature = "python", ...)]` directly below the runtime `pyo3::pyclass` attribute. - Put `gen_stub_pymethods` directly below `#[pymethods]`. - Put `gen_stub_pyfunction` after doc comments and directly above `#[pyfunction]`. - Set the stub `module` to the package from which Python imports the object. - Add `pyo3-stub-gen` as an optional dependency and include it in the `python` feature. ### Generated Python artifacts The Python surface commits generated `.pyi` files under `python/nautilus_trader/` and generated wrapper doc comments under `crates/**/src/python/`. Regenerate both with: ```bash make py-stubs ``` Run the target after changing a Python-exposed Rust item, its stub annotation, its core doc comments, or adapter feature wiring. Commit every changed generated artifact with the source change. The stub generator removes `extension-module` before invoking Cargo. Add a feature that is enabled only through `extension-module` explicitly to `cargo_features` in `python/generate_stubs.py`; otherwise its exported types disappear from the generated stubs. The Interactive Brokers `gateway` feature is the model. The Python targets accept the uv minor series defined by `required-version` in `python/pyproject.toml`. `make sync`, `make py-stubs`, and `make build-debug` stop when the installed version falls outside that range. Run `make update-uv` to install the exact project version from `.nautilus-engineering/tools.toml`. Do not edit wrapper `///` comments in `crates/**/src/python/`. Edit the core Rust item docs and regenerate. The doc sync: - Preserves `# Errors` and `# Safety`. - Drops `# Panics` so a panic contract does not cross the Python API boundary. - Removes Rust intra-doc links. - Converts Rust `::` paths to Python `.` paths. ### Rust-Python object ownership `Py` owns a reference to a Python object. Clone it with `Py::clone_ref` while attached to the interpreter or with `nautilus_core::python::clone_py_object`. An extra `Arc>` is normally unnecessary because `Py` already provides shared ownership. Cloning a Python reference does not break a cycle. Use a Python weak reference when a back-reference must not keep its target alive. ## Testing conventions - Use `mod tests` for ordinary inline tests. - Use `#[rstest]` instead of `#[test]`, including for a non-parameterized test. - Use `#[tokio::test]` for a non-parameterized async test. - Keep `#[cfg(test)]` on test modules and test-only files. Do not add test behavior or interfaces to production code. - Store JSON fixtures under the crate's `test_data/` directory and load them with `include_str!`. - Use distinct, non-default inputs and exact expected values. Assert every stable field. - Group assertions after setup and actions unless the test checks stepwise state changes. - Do not use Arrange, Act, Assert separator comments. Every workspace member appears exactly once in `CORE_CRATES`, `ADAPTER_CRATES`, or `NO_TEST_CRATES` in the `Makefile`. Add a crate to `NO_TEST_CRATES` only when it defines no Rust test target and contains no `#[test]`, `#[rstest]`, or `#[test_case]` function. Parametrize cases when the same behavior applies to several inputs: ```rust #[rstest] #[case("AUDUSD", false)] #[case("CL.FUT", true)] fn test_symbol_is_composite(#[case] input: &str, #[case] expected: bool) { assert_eq!(Symbol::new(input).is_composite(), expected); } ``` ### Test specs Events with many constructor arguments use a fluent `bon` spec next to the event under `events/order/spec/`. Gate the module with `#[cfg(any(test, feature = "test-support"))]` so downstream tests can opt in without adding the spec to production builds. - Derive `bon::Builder` with `finish_fn = into_spec`. - Give required fields deterministic valid defaults. Leave optional fields as `Option`. - Default event IDs with `test_uuid()` from `crate::stubs`. - Implement `build()` by forwarding through the production constructor. - Return the event directly because spec defaults are valid by construction. - Pin every default in one test in the spec module. Override only the fields relevant to the caller: ```rust let fill = OrderFilledSpec::builder() .last_qty(Quantity::from(50_000)) .trade_id(TradeId::from("TRADE-1")) .build(); ``` Under plain `cargo test`, call `reset_test_uuid_rng()` before a test that compares UUID sequences. `cargo nextest` starts each test in a fresh process, so the sequence resets automatically. ### Property-based tests Use `proptest` when an invariant spans an input class that examples cannot cover. Keep strategies near the property suite and combine ranges with explicit edge cases. Prefix property names with `prop_` and retain `#[rstest]` on tests inside `proptest!`. A large suite may use a `property_tests` module or a dedicated file, but the repository also keeps focused properties beside their unit tests. ## Unsafe Rust Unsafe code must make its proof obligations reviewable: - Give each unsafe function a `# Safety` section that states the caller's complete obligations. - Put a `SAFETY:` comment directly above each unsafe operation and explain why its preconditions hold at that site. - Wrap each unsafe operation in its own `unsafe { ... }` block. Workspace crates deny `unsafe_op_in_unsafe_fn`. - Use always-on checks for null, alignment, provenance, and other soundness conditions. - Add targeted tests for observable behavior around unsafe code. Tests support the proof but do not establish soundness. - Treat an unsafe `Send` or `Sync` implementation as a proof over all reachable state, aliases, callbacks, generic parameters, safe methods, cloning, and destruction. For raw vectors crossing FFI, follow the [FFI memory contract](ffi.md). Foreign code owns the allocation after transfer and must call the matching `vec_drop_*` function exactly once. ## Other generated artifacts ### FFI bindings and precision Only `nautilus-core` and `nautilus-model` expose an `ffi` feature. Check both crates directly when changing their C ABI: ```bash cargo check -q -p nautilus-core --features ffi cargo check -q -p nautilus-model --features ffi,python,high-precision ``` The crate-local `cbindgen.toml` files define the header layout for native consumers. Do not add an `ffi` feature or `src/ffi` module to another workspace crate. Run `make check-cbindgen-abi` to generate both headers, verify their public names and compatibility enum values, and compile a C11 consumer against them. The scheduled nightly tests run this check. ### Cap'n Proto schemas Schema files live under `crates/serialization/schemas/capnp/`, and generated Rust lives under `crates/serialization/generated/capnp/`. - Add fields at the end. - Do not remove fields or reuse field numbers. Mark obsolete fields as deprecated in comments. - Regenerate with `make regen-capnp`. - Review `git diff crates/serialization/generated/capnp`. - Run `make check-capnp-schemas` to verify the checked-in output. - Test the serialization crate with `make cargo-test-crate-nautilus-serialization`. Install the pinned compiler as described in [Environment setup](environment_setup.md#capn-proto). Generated bindings stay checked in so docs.rs can build without the compiler. # Security Architecture Source: https://nautilustrader.io/docs/latest/developer_guide/security/ This page describes the security model for the NautilusTrader release pipeline. It explains how release artifacts are built, published, attested, and verified. Use this page with: - [Supply chain policy](https://nautilustrader.io/security/supply-chain/), which states the public dependency and release integrity commitments. - [Releases](releases.md), which documents the release workflow and checklist. - [Security Policy](../../SECURITY.md), which gives consumer-facing verification commands. - [GitHub Actions overview](../../.github/OVERVIEW.md#security), which documents CI/CD controls. ## Security goals The release pipeline has four goals: - Build every official artifact from a reviewed repository commit. - Publish Python and Rust packages without long-lived package registry tokens. - Attach checksums, manifests, and provenance before publishing the GitHub release. - Give users enough public data to verify that downloaded artifacts match the release. The GitHub release anchors package integrity. Stable releases attach wheel and sdist assets to a draft GitHub release before any package index publish starts. The pipeline publishes package indexes, verifies those indexes against the GitHub release assets, attaches final integrity assets, then publishes the GitHub release. ## Threat model The pipeline defends against: - Compromised or mutable third-party GitHub Actions, by pinning actions to commit SHAs. - Accidental release from the wrong workflow, branch, or environment, by binding OIDC publishers to `nautechsystems/nautilus_trader`, `build.yml`, and the `release` environment. - Long-lived package registry token theft, by using PyPI and crates.io Trusted Publishing. - Registry propagation lag or partial re-runs, by making publish and verify scripts idempotent and retry-tolerant. - Registry substitution or upload drift, by comparing PyPI and crates.io artifacts against release manifests and registry metadata. - Silent manual crate recovery, by requiring explicit `CRATES_IO_MANUAL_PUBLISH_EXCEPTIONS` entries and recording those exceptions in `crates-manifest.json`. The pipeline does not defend against: - A malicious maintainer with permission to change release workflows and approve releases. - A compromise of GitHub, PyPI, crates.io, or Sigstore that can forge the trust roots users rely on. - A compromised end-user machine before verification runs. - Runtime compromise of an exchange, broker, data provider, or user trading strategy. - Bit-identical rebuild drift for wheels and sdists. The current guarantee is provenance and digest verification, not reproducible builds. ## Trust roots - GitHub repository rules protect reviewed source, release branches, and release tags. Protected `master` and immutable `v*` release tags are the relevant records. - GitHub Actions OIDC issuer provides short-lived workflow identities from `https://token.actions.githubusercontent.com`. - GitHub `release` environment gates package publishing and release approvals. The environment restricts deployment to `master` and requires reviewer approval. - PyPI Trusted Publishing publishes wheels and sdist without a persistent token. It binds to repository `nautechsystems/nautilus_trader`, workflow `build.yml`, and environment `release`. - crates.io Trusted Publishing publishes Rust crates without a persistent token. It binds to owner `nautechsystems`, repository `nautilus_trader`, workflow `build.yml`, and environment `release`. - Sigstore Fulcio, Rekor, and TUF bind artifacts to OIDC identities and the transparency log. GitHub artifact attestations, PyPI publish attestations, and Docker cosign signatures rely on this root. - GitHub release immutability prevents post-publish asset and tag replacement. Published release assets and the release tag become immutable. ## Release flow ```mermaid flowchart TD source["Reviewed commit on master"] gates["Release gates
Rust suite + cargo-deny + cargo-vet
Cargo publish + docs/features preflights"] wheels["Build wheels"] draft["Create tag and draft GitHub release"] sdist["Build sdist"] assets["Attach wheels and sdist to draft release"] registries["Publish PyPI and crates.io
Trusted Publishing"] verify["Verify registries against release assets"] integrity["Attach SHA256SUMS, manifests,
Sigstore bundles, DSSE envelopes"] publish["Publish GitHub release"] release_attest["Verify GitHub release attestation"] docker["Build, sign, and attest Docker images"] source --> gates source --> wheels gates --> draft wheels --> draft draft --> sdist sdist --> assets wheels --> assets assets --> registries registries --> verify verify --> integrity integrity --> publish publish --> release_attest source --> docker ``` The Docker workflow is separate from the package release workflow, but it follows the same identity model: image signatures and SBOM attestations bind the image digest to the expected GitHub Actions workflow identity. ## Artifact records - Python wheels are published to GitHub Releases, PyPI, and the Nautech Systems package index (`packages.nautechsystems.io`). `SHA256SUMS`, per-asset `.sha256` files, and `dist-manifest.json` record integrity. GitHub artifact attestations, PyPI publish attestations, `.sigstore` bundles, and `.intoto.jsonl` envelopes record provenance. - Python sdists are published to GitHub Releases and PyPI. They use the same integrity and provenance records as wheels but are not published to the wheel-only package index. - Rust crates are published to crates.io. The crates.io checksum and `crates-manifest.json` record integrity. crates.io `trustpub_data` records provenance unless an explicit manual exception is present. - Docker images are published to GitHub Container Registry. The image digest is the integrity record. Sigstore cosign signatures and SPDX SBOM attestations record provenance. - The GitHub release record is published through GitHub Releases. Published release assets and the immutable tag record integrity. The GitHub release attestation records provenance. ## Consumer verification map Detailed commands live in [Verifying releases](https://github.com/nautechsystems/nautilus_trader/blob/master/SECURITY.md#verifying-releases). The checks below show the public data each consumer should verify. ### Python wheels and sdist Verify: - The artifact digest matches `SHA256SUMS`, the per-asset `.sha256` file, or `dist-manifest.json`. - The GitHub artifact attestation identity matches `nautechsystems/nautilus_trader/.github/workflows/build.yml` on `master` or `nightly`. - The PyPI publish attestation reports repository `nautechsystems/nautilus_trader`, workflow `build.yml`, and environment `release`. Example: ```bash : "${VERSION:?Set VERSION to the Python package version}" : "${ARTIFACT:?Set ARTIFACT to the release asset filename}" TAG="v$VERSION" REPO=nautechsystems/nautilus_trader ISSUER=https://token.actions.githubusercontent.com IDENTITY='^https://github\.com/nautechsystems/nautilus_trader/\.github/workflows/build\.yml@refs/heads/(master|nightly)$' gh release download "$TAG" --repo "$REPO" --pattern "$ARTIFACT" --pattern "$ARTIFACT.sha256" sha256sum -c "$ARTIFACT.sha256" gh attestation verify "$ARTIFACT" \ --repo "$REPO" \ --cert-identity-regex "$IDENTITY" \ --cert-oidc-issuer "$ISSUER" ``` ### PyPI publish provenance Verify: - PyPI file hashes match `dist-manifest.json`. - PyPI provenance exposes the expected GitHub publisher identity. - `pypi-attestations verify` accepts the downloaded file URL. Example: ```bash : "${VERSION:?Set VERSION to the Python package version}" : "${ARTIFACT:?Set ARTIFACT to the release asset filename}" PYPI_URL=$(curl -sS "https://pypi.org/pypi/nautilus_trader/$VERSION/json" | \ jq -r --arg artifact "$ARTIFACT" '.urls[] | select(.filename == $artifact) | .url') uv run --no-project --no-build --with pypi-attestations -- \ pypi-attestations verify pypi \ --repository https://github.com/nautechsystems/nautilus_trader \ "$PYPI_URL" ``` ### Rust crates Verify: - The crates.io version checksum matches the downloaded `.crate` file. - `trustpub_data.provider` is `github`. - `trustpub_data.repository` is `nautechsystems/nautilus_trader`. - `published_by` is `null`, unless `crates-manifest.json` records an explicit `manual_token_publish` exception. Example: ```bash CRATE=${CRATE:-nautilus-core} : "${VERSION:?Set VERSION to the crate version}" REPO=nautechsystems/nautilus_trader VERSION_JSON=$(curl -sS "https://crates.io/api/v1/crates/$CRATE/versions" | \ jq -c --arg version "$VERSION" '.versions[] | select(.num == $version)') CRATE_SHA256=$(printf '%s\n' "$VERSION_JSON" | jq -r '.checksum') printf '%s\n' "$VERSION_JSON" | jq -e --arg repo "$REPO" \ '.trustpub_data.provider == "github" and .trustpub_data.repository == $repo and .published_by == null' curl -sSL "https://static.crates.io/crates/$CRATE/$CRATE-$VERSION.crate" -o "$CRATE-$VERSION.crate" test "$(sha256sum "$CRATE-$VERSION.crate" | cut -d ' ' -f 1)" = "$CRATE_SHA256" ``` ### Docker images Verify: - The mutable tag resolves to the digest you intend to run. - The cosign signature identity matches the Docker workflow. - The SPDX SBOM attestation is bound to the same image digest. Example: ```bash export IMAGE_BASE=ghcr.io/nautechsystems/nautilus_trader export DIGEST=$(crane digest "$IMAGE_BASE:latest") export IMAGE=$IMAGE_BASE@$DIGEST export ISSUER=https://token.actions.githubusercontent.com export IDENTITY='^https://github\.com/nautechsystems/nautilus_trader/\.github/workflows/docker\.yml@refs/heads/(master|nightly)$' cosign verify "$IMAGE" --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" cosign verify-attestation \ --type https://spdx.dev/Document/v2.3 \ "$IMAGE" \ --certificate-identity-regexp "$IDENTITY" \ --certificate-oidc-issuer "$ISSUER" ``` ## Manual recovery posture Normal releases use Trusted Publishing only. Manual package publishing is a last-resort recovery path after a partial release. Rules for manual recovery: - Prefer re-running the failed job or workflow when a registry or Sigstore verifier fails. - Do not replace a release tag or GitHub release assets after publication. - Do not silently accept manually published crates. - If a crate must be recovered with a token, list each `crate@version` in `CRATES_IO_MANUAL_PUBLISH_EXCEPTIONS`. - Record the exception in release notes and in `crates-manifest.json` with `release_status: "manual_token_publish"`. No routine release path depends on a long-lived PyPI or crates.io token. ## Incident response posture - PyPI publisher drift is detected by the PyPI provenance verifier. Stop publishing, fix the PyPI Trusted Publisher, and rerun verification. - crates.io publisher drift is detected by the trusted-publishing check or registry verifier. Fix crate publisher settings and rerun. Use a manual exception only for partial recovery. - GitHub release asset mismatch is detected by checksum or manifest verification. Stop the release before publication, or publish an advisory if assets already shipped. - Sigstore, Rekor, or TUF lag is detected by retryable transparency errors. Retry with bounded backoff, then pause release sealing if lag persists. - Sigstore trust root concern appears when attestation verification becomes ambiguous. Pause releases, verify against registry records, and rotate trust roots when supported. - Workflow identity mismatch is detected by GitHub, PyPI, or cosign identity checks. Treat it as configuration drift or compromise until reviewed. - Manual crate publish exceptions are detected when crates.io shows `published_by` instead of `trustpub_data`. Record the explicit exception, document affected crates, and preserve the audit trail. ## SLSA posture Python release artifacts carry build provenance through GitHub artifact attestations and PyPI publish attestations. Docker images carry Sigstore signatures and SPDX SBOM attestations. Rust crates rely on crates.io Trusted Publishing metadata and the release `crates-manifest.json`. This page does not assert a named SLSA level for all artifact classes. Any future SLSA level claim must cite this architecture, name the artifact classes it covers, and include CI validation that the published provenance parses as the claimed predicate type. # Shell Scripts Source: https://nautilustrader.io/docs/latest/developer_guide/shell/ This page defines when to create a shell script and how to name, write, invoke, and test it. Bash is the default. Use POSIX `sh` only when a supported caller cannot rely on Bash being installed. The policy applies to shell files throughout the repository, including `scripts/`, `scripts/ci/`, `.pre-commit-hooks/`, and component directories. ## When to write a script Before adding a script, search for an existing script, Make target, or installed tool that already provides the behavior. Add a script when it gives procedural logic one testable, linted source of truth. | Location | Owns | Delegates | | -------------- | -------------------------------------------------------------------- | ------------------------------------------------------ | | GitHub Actions | Events, permissions, matrices, runners, secrets, and hosted actions | Multi-step shell behavior | | `Makefile` | Discoverable tasks, dependencies, variables, and concurrency limits | Non-trivial control flow | | Shell script | Validation, reusable command sequences, retries, and transformations | Workflow orchestration and build dependency management | Prefer a script when: - A command sequence is used by more than one workflow, Make target, or developer task. - An inline GitHub Actions step contains branches, loops, retries, or failure handling worth testing outside the workflow. - A Make recipe needs enough procedural logic that quoting, error propagation, or platform behavior becomes hard to review. - A repeated maintenance or release task benefits from ShellCheck, `shfmt`, and focused tests. Keep a command inline when it is short, used once, and clearer in its caller. Do not wrap one stable command only to add another file. When extracting GitHub Actions logic, keep expressions such as `${{ github.ref }}`, permissions, secret selection, and runner selection in the workflow. Pass ordinary values to the script through arguments or environment variables. The script's exit status must remain the step's exit status. ## Choose the shell and extension The extension identifies the shell language, not whether the file is executable or sourceable. | Extension | Interpreter | Use | | --------- | ----------- | ------------------------------------------------------------------------ | | `.bash` | Bash | Default for new scripts and sourceable Bash files. | | `.sh` | POSIX `sh` | Only when a supported caller has a real requirement to run without Bash. | Use `#!/usr/bin/env bash` for `.bash` files and `#!/usr/bin/env sh` for `.sh` files. Keep the shebang even when Make or GitHub Actions invokes the file through `bash` or `sh`, because tools and direct callers use it to identify the interpreter. Bash is preferred for normal development, Make, and CI scripts because the repository already depends on it and its features make non-trivial shell code clearer. These features include `pipefail`, `[[ ... ]]`, arrays, process substitution, and function-local variables. Use POSIX `sh` for a small bootstrap or wrapper only when avoiding a Bash dependency is part of its supported interface. Test the script under the target `/bin/sh`; simple syntax alone does not prove POSIX compatibility. Existing filenames predate this extension policy, so some `.sh` files contain Bash. Treat their shebangs as the source of truth. Do not rename an existing script only to change its extension. Apply the policy to new files and to scoped renames that already update every call site and document. ## Define the portability target A script must support every platform on which its callers run. Unless its purpose states a narrower target, write it for Linux, macOS, and Windows through Git Bash, MSYS2, or WSL. Use Bash 3.2 as the default language floor because it is available on supported macOS systems. Avoid Bash 4+ features unless every caller provisions a newer version. Common Bash 4+ features and portable alternatives include: | Feature | Bash version | Portable alternative | | --------------------------------- | ------------ | ---------------------------------- | | Associative arrays (`declare -A`) | 4.0+ | Files, simple arrays, or functions | | `readarray` / `mapfile` | 4.0+ | `while read` loops | | `${var,,}` / `${var^^}` | 4.0+ | `tr` for case conversion | A CI-only script may use a newer Bash version or platform-specific tool when every workflow caller guarantees that environment. Document the constraint near the code that depends on it. A path under `scripts/ci/` does not by itself make a script Linux-only because CI also uses macOS and Windows runners. ### System utilities Prefer options supported by both GNU and BSD utilities. When no common form exists, detect the capability or operating system and implement both forms. | Operation | GNU form | BSD or macOS form | Portable approach | | ----------------- | -------------- | ----------------- | --------------------------------------------------------- | | In-place `sed` | `sed -i` | `sed -i ''` | Use a backup suffix such as `sed -i.bak`, then remove it. | | File size | `stat -c '%s'` | `stat -f '%z'` | Try or select the supported form. | | SHA-256 | `sha256sum` | `shasum -a 256` | Detect the command and keep output handling equal. | | Canonical path | `readlink -f` | No common form | Avoid it or resolve from a known directory with `pwd`. | | Extended matching | `grep -P` | No common form | Use `grep -E` when it expresses the same pattern. | | Nanosecond time | `date +%N` | No common form | Use an existing run ID or `$RANDOM` for cache busting. | Quote paths and expansions so spaces do not change argument boundaries. Do not assume filesystem paths are case-sensitive. Use repository-relative paths only after resolving the repository root from the script location, not from the caller's working directory. Use only commands installed by the documented development or runner setup. If an optional command is necessary, check for it with `command -v` and report how to install or replace it. The repository stores text with LF endings through `.gitattributes`; do not add platform-specific line endings. ## Place and name scripts - Put general development and maintenance commands under `scripts/`. - Put workflow-specific build, test, publication, and verification commands under `scripts/ci/`. - Put repository checks invoked by pre-commit under `.pre-commit-hooks/`. - Keep component-specific scripts beside the component when moving them to `scripts/` would hide their ownership. - Under `scripts/` and in component directories, use lowercase kebab-case. Name a regression script `test-.bash` or `test-.sh` to keep the tested pair together. - Under `.pre-commit-hooks/`, use lowercase snake_case and match the existing `check_*` and `test_check_*` name families. Keep each script focused on one task, and extend an existing script when new logic shares the same responsibility. ## Structure scripts for reliable execution Standalone Bash scripts should start with: ```bash #!/usr/bin/env bash set -euo pipefail ``` Use `set -eu` in a standalone POSIX `sh` script. POSIX does not define `pipefail`, so check pipeline behavior explicitly when a failure must propagate. If a script cannot use these options, explain the specific control flow that makes an option unsafe. Follow these requirements: - Validate required arguments and environment variables before changing state. Print concise usage text and exit nonzero for invalid input. - Quote parameter expansions. Use Bash arrays for argument lists instead of building a command string, and do not use `eval`. - Keep machine-readable output on standard output and diagnostics on standard error when callers capture the result. - Do not end routine status output with a terminating period. Keep punctuation when the output is a complete explanatory or diagnostic sentence. - Create temporary files with `mktemp`, register cleanup with `trap`, and constrain cleanup to the exact paths created by the script. - Bound retries, report the final failure, and return a nonzero status when the requested operation does not complete. - Do not print secrets or enable command tracing around credentials. Pass secrets through the environment or the tool's supported secret input. - Use comments only for constraints or behavior that the commands do not make clear. Give a standalone script executable permissions when users or tools call it as `./path`. A sourceable file does not need executable permissions. Prefer executing a script in a child process; source a file only when the caller must share its functions or shell state. A sourceable Bash file must not call `exit` or change the caller's shell options. Return errors from functions and let the caller choose its error policy. Use `${BASH_SOURCE[0]}` instead of `$0` to resolve the source file's location. When a script needs several functions, define `main` first, place called functions below their callers, and invoke `main "$@"` after all definitions. This keeps the task visible at the top while ensuring every function exists before execution starts. ## Integrate with Make and GitHub Actions Make targets should provide the stable, discoverable command that developers run. Keep target dependencies, build variables, and concurrency limits in the Makefile, then invoke the script with an explicit interpreter. For example, the `check-generated-drift` target delegates its procedural work to `scripts/ci/check-generated-drift.bash`. GitHub Actions should provide workflow context through named environment variables and call the same script used locally where practical. The build workflow follows this boundary: it owns the Python matrix condition and `TARGET_DIR`, then invokes `scripts/ci/check-generated-drift.bash`. Keep GitHub-specific output files such as `$GITHUB_OUTPUT` and `$GITHUB_ENV` at the workflow boundary when the script is also a local command. A script dedicated to GitHub Actions may write them when that integration is its stated purpose. ## Format and lint The pre-commit configuration formats shell files with `shfmt` using two-space indentation, indented case branches, consistent redirect spacing, and the Bash parser. ShellCheck then checks quoting, expansion, control flow, portability, and common command errors. The pinned hooks in `.pre-commit-config.yaml` are the source of truth for tool versions and options. Run both hooks against the exact changed scripts: ```bash prek run shfmt --files scripts/ci/test-wheel.bash prek run shellcheck --files scripts/ci/test-wheel.bash ``` `shfmt` updates files in place. Review its changes before running ShellCheck. ShellCheck selects the language from the shebang, so a `.sh` file with `#!/usr/bin/env sh` receives POSIX checks even though the formatter uses the common Bash parser. Keep ShellCheck suppressions on the narrowest applicable line. Add a nearby reason when the constraint is not clear from the code, and do not disable a check for the whole repository to silence one script. The repository also rejects executable files without shebangs, mixed line endings, trailing whitespace, and unresolved merge markers. These checks complement ShellCheck; they do not replace a runtime test. ## Test behavior Run the smallest test that exercises the changed branches and failure paths. For logic that can regress independently of a workflow, add a companion shell test. This includes parsing, multi-branch decisions, retries and cleanup, policy checks, and material external side effects. A domain-level suite may cover cooperating scripts, and a thin wrapper does not need a one-to-one test when that suite invokes it and proves its behavior. Each companion test: - Creates isolated state under `mktemp -d` and removes it on exit. - Supplies fake external commands through a temporary `PATH` instead of changing production code. - Uses distinct inputs and exact output, exit status, and side-effect assertions. - Covers success, invalid input, dependency failure, and cleanup when those paths exist. - Fails when a required test command is unavailable; a passing skip does not validate behavior. - Runs from `make test-scripts`, which is the script test inventory used by CI. When a script has callers on multiple operating systems, exercise platform-sensitive changes on each caller's relevant CI matrix. A Linux test plus clean ShellCheck output does not prove macOS or Windows behavior. ## Review checklist - The behavior is not already available from a script, Make target, installed tool, or simple native workflow feature. - GitHub Actions, Make, and the script own the right parts of the workflow. - The extension, shebang, executable mode, and documented portability target agree. - The script runs independently of the caller's working directory. - Arguments, failures, temporary files, retries, output, and secrets have explicit handling. - Focused behavior tests, `shfmt`, and ShellCheck pass for the final changed files. - Every call site, workflow path filter, and document uses the final filename. # Data Testing Spec Source: https://nautilustrader.io/docs/latest/developer_guide/spec_data_testing/ This section defines a rigorous test matrix for validating adapter data functionality using the Rust `DataTester` actor. Python exposes it as a built-in actor configured through `nautilus_trader.testkit.DataTesterConfig`; Rust code imports it from `nautilus_testkit::testers`. Each test case is identified by a prefixed ID (e.g. TC-D01) and grouped by functionality. **Each adapter must pass the subset of tests matching its supported data types.** Test groups are ordered from least derived to most derived data: instruments and raw book data first, then quotes, trades, bars, and derivatives data. An adapter that passes groups 1-4 is considered baseline data compliant. Document adapter-specific data behavior (custom channels, throttling, snapshot semantics, etc.) in the adapter's own guide, not here. ## Prerequisites Before running data tests: - Target instrument available and loadable via the instrument provider. - API credentials set via environment variables (`{VENUE}_API_KEY`, `{VENUE}_API_SECRET`) when the venue requires authentication for the data being tested. - If the venue offers a demo/testnet mode, use credentials created for that environment. Demo and production API keys are typically separate and not interchangeable; using the wrong credentials produces authentication errors (e.g. HTTP 401). **Python node setup**: Use `nautilus_trader.live.LiveNode`. Call `LiveNode.builder(...)` when you need to register adapter client factories before the node is built. ```python from nautilus_trader.common import Environment from nautilus_trader.config import LiveDataEngineConfig from nautilus_trader.live import LiveNode from nautilus_trader.model import TraderId from nautilus_trader.testkit import DataTesterConfig node = ( LiveNode.builder("TESTER-001", TraderId("TESTER-001"), Environment.SANDBOX) .with_data_engine_config(LiveDataEngineConfig(time_bars_build_with_no_updates=False)) .add_data_client(None, adapter_data_client_factory, data_client_config) .build() ) tester_config = DataTesterConfig( client_id=client_id, instrument_ids=[instrument_id], subscribe_quotes=True, ) node.add_builtin_actor("DataTester", tester_config) # Register remaining components, then start or run ``` **Rust node setup** (reference: `crates/adapters/{adapter}/examples/node_data_tester.rs`): ```rust use nautilus_testkit::testers::{DataTester, DataTesterConfig}; let tester_config = DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_quotes(true) .build()?; let tester = DataTester::new(tester_config); node.add_actor(tester)?; node.run().await?; ``` ## Timestamp scale Nautilus stores `ts_event` and `ts_init` as Unix nanoseconds (`UnixNanos`). Every data message that carries those fields must use that scale, not leftover seconds, milliseconds, or microseconds. - A value below `10^16` is not a plausible Unix-nanosecond timestamp (`10^16` ns is about 116 days after 1970-01-01) and usually means the adapter left the venue scale unconverted. - Second-precision venue times that were converted correctly end in `000000000` and still pass: that is coarse precision, not a scale error. - Live stream `ts_event` should be near wall-clock time for the session. Historical request results may be older and still valid if the scale is nanoseconds. - `ts_init` is the local clock when Nautilus created the object. Small `ts_event` > `ts_init` skew is possible when the venue clock is ahead. `DataTester` warns when `ts_event` or `ts_init` fails the scale check on instruments, quotes, trades, bars, book deltas, book depth, mark and index prices, funding rates, instrument status and close, option greeks, and historical batches of those types. It does not check reconstructed books in `on_book`. Treat a warning as a failure for the case that produced the message. --- Each group below begins with a summary table, followed by detailed test cards. Test IDs use spaced numbering to allow insertion without renumbering. --- ## Group 1: Instruments Verify instrument loading and subscription before testing market data streams. | TC | Name | Description | Skip when | | ------ | ------------------------ | --------------------------------- | ------------------ | | TC-D01 | Request instruments | Load all instruments for a venue. | Never. | | TC-D02 | Subscribe instrument | Subscribe to instrument updates. | No instrument sub. | | TC-D03 | Load specific instrument | Load a single instrument by ID. | Never. | ### TC-D01: Request instruments | Field | Value | | ------------------ | --------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected. | | **Action** | DataTester requests all instruments for the venue on start. | | **Event sequence** | `on_instruments` callback receives instrument list. | | **Pass criteria** | At least one instrument received; each has valid symbol, price precision, and size increment. | | **Skip when** | Never. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], request_instruments=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .request_instruments(true) .build()? ``` ### TC-D02: Subscribe instrument | Field | Value | | ------------------ | --------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester subscribes to instrument updates. | | **Event sequence** | `on_instrument` callback receives instrument. | | **Pass criteria** | Instrument received with correct `instrument_id`, valid fields. | | **Skip when** | Adapter does not support instrument subscriptions. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_instrument=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_instrument(true) .build()? ``` ### TC-D03: Load specific instrument | Field | Value | | ------------------ | -------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected. | | **Action** | Load a specific instrument by `InstrumentId` via the instrument provider. | | **Event sequence** | Instrument available in cache after load. | | **Pass criteria** | Instrument loaded with correct ID, price precision, size increment, and trading rules. | | **Skip when** | Never. | **Considerations:** - This tests the instrument provider's `load` / `load_async` method directly. - Verify the instrument is cached and available via `self.cache.instrument(instrument_id)`. --- ## Group 2: Order book Test order book subscription modes and snapshot requests. | TC | Name | Description | Skip when | | ------ | -------------------------- | ----------------------------------- | ----------------- | | TC-D10 | Subscribe book deltas | Stream `OrderBookDeltas` updates. | No book support. | | TC-D11 | Subscribe book at interval | Periodic `OrderBook` snapshots. | No book support. | | TC-D12 | Subscribe book depth | `OrderBookDepth10` snapshots. | No book depth. | | TC-D13 | Request book snapshot | One-time book snapshot request. | No book snapshot. | | TC-D14 | Managed book from deltas | Build local book from delta stream. | No book support. | Python uses `BookType.L2_MBP` for these scenarios. The Rust builder can override `book_type` when an adapter requires a different book representation. ### TC-D10: Subscribe book deltas | Field | Value | | ------------------ | -------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester subscribes to order book deltas. | | **Event sequence** | `OrderBookDeltas` events received in `on_book_deltas`. | | **Pass criteria** | Deltas received with valid instrument ID; at least one delta contains bid/ask updates. | | **Skip when** | Adapter does not support order book data. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_book_deltas=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_book_deltas(true) .book_type(BookType::L2_MBP) .build()? ``` ### TC-D11: Subscribe book at interval | Field | Value | | ------------------ | ----------------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester subscribes to periodic order book snapshots. | | **Event sequence** | `OrderBook` events received in `on_book` at configured interval. | | **Pass criteria** | Book snapshots received with bid/ask levels; updates arrive at approximately the configured interval. | | **Skip when** | Adapter does not support order book data. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_book_at_interval=True, book_depth=10, book_interval_ms=1000, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_book_at_interval(true) .book_type(BookType::L2_MBP) .book_depth(10) .book_interval_ms(1000) .build()? ``` ### TC-D12: Subscribe book depth | Field | Value | | ------------------ | ------------------------------------------------------------------------------------ | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester subscribes to `OrderBookDepth10` snapshots. | | **Event sequence** | `OrderBookDepth10` events received in `on_book_depth`. | | **Pass criteria** | Depth snapshots received with up to 10 bid/ask levels; prices are correctly ordered. | | **Skip when** | Adapter does not support book depth subscriptions. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_book_depth=True, book_depth=10, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_book_depth(true) .book_type(BookType::L2_MBP) .book_depth(10) .build()? ``` ### TC-D13: Request book snapshot | Field | Value | | ------------------ | ------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester requests a one-time order book snapshot. | | **Event sequence** | Book snapshot received via historical data callback. | | **Pass criteria** | Snapshot contains bid/ask levels with valid prices and sizes. | | **Skip when** | Adapter does not support book snapshot requests. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], request_book_snapshot=True, book_depth=10, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .request_book_snapshot(true) .book_depth(10) .build()? ``` ### TC-D14: Managed book from deltas | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, book deltas streaming. | | **Action** | DataTester subscribes to deltas with `manage_book=True`; builds local order book from the delta stream. | | **Event sequence** | `OrderBookDeltas` applied to local `OrderBook`; book logged with configured depth. | | **Pass criteria** | Local book builds correctly from deltas; bid levels descend, ask levels ascend; book is not empty after initial snapshot. | | **Skip when** | Adapter does not support order book data. | **Considerations:** - The managed book applies each delta to an `OrderBook` instance maintained by the actor. - Use `book_levels_to_print` to control logging verbosity. **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_book_deltas=True, manage_book=True, book_levels_to_print=10, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_book_deltas(true) .manage_book(true) .book_type(BookType::L2_MBP) .build()? ``` `DataTesterConfig` exposes `request_book_deltas`, but `DataTester` does not issue that historical request. Test an adapter's historical book delta support through a custom actor until the tester implements the request path. --- ## Group 3: Quotes Test quote tick subscriptions and historical requests. | TC | Name | Description | Skip when | | ------ | ------------------------- | ------------------------------------------- | --------------------- | | TC-D20 | Subscribe quotes | Verify `QuoteTick` events flow after start. | Never. | | TC-D21 | Request historical quotes | Request historical quote ticks. | No historical quotes. | ### TC-D20: Subscribe quotes | Field | Value | | ------------------ | --------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester subscribes to quotes on start. | | **Event sequence** | `QuoteTick` events received in `on_quote`. | | **Pass criteria** | At least one `QuoteTick` received with valid bid/ask prices and sizes; bid < ask. | | **Skip when** | Never. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_quotes=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_quotes(true) .build()? ``` ### TC-D21: Request historical quotes | Field | Value | | ------------------ | ----------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester requests historical quote ticks. | | **Event sequence** | Historical quote batches received via `on_historical_quotes`. | | **Pass criteria** | Quotes received with valid timestamps, bid/ask prices, and sizes. | | **Skip when** | Adapter does not support historical quote requests. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], request_quotes=True, ) ``` --- ## Group 4: Trades Test trade tick subscriptions and historical requests. | TC | Name | Description | Skip when | | ------ | ------------------------- | ------------------------------------------- | --------------------- | | TC-D30 | Subscribe trades | Verify `TradeTick` events flow after start. | Never. | | TC-D31 | Request historical trades | Request historical trade ticks. | No historical trades. | ### TC-D30: Subscribe trades | Field | Value | | ------------------ | ----------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester subscribes to trades on start. | | **Event sequence** | `TradeTick` events received in `on_trade`. | | **Pass criteria** | At least one `TradeTick` received with valid price, size, and aggressor side. | | **Skip when** | Never. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_trades=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_trades(true) .build()? ``` ### TC-D31: Request historical trades | Field | Value | | ------------------ | -------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester requests historical trade ticks. | | **Event sequence** | Historical trade batches received via `on_historical_trades`. | | **Pass criteria** | Trades received with valid timestamps, prices, sizes, and trade IDs. | | **Skip when** | Adapter does not support historical trade requests. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], request_trades=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .request_trades(true) .build()? ``` --- ## Group 5: Bars Test bar subscriptions and historical requests. | TC | Name | Description | Skip when | | ------ | ----------------------- | ------------------------------------- | ------------------- | | TC-D40 | Subscribe bars | Verify `Bar` events flow after start. | No bar support. | | TC-D41 | Request historical bars | Request historical OHLCV bars. | No historical bars. | ### TC-D40: Subscribe bars | Field | Value | | ------------------ | ---------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, bar type configured. | | **Action** | DataTester subscribes to bars for a configured `BarType`. | | **Event sequence** | `Bar` events received in `on_bar`. | | **Pass criteria** | At least one `Bar` received with valid OHLCV values; high >= low, high >= open, high >= close. | | **Skip when** | Adapter does not support bar subscriptions. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], bar_types=[BarType.from_str("BTCUSDT-PERP.VENUE-1-MINUTE-LAST-EXTERNAL")], subscribe_bars=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .bar_types(vec![bar_type]) .subscribe_bars(true) .build()? ``` ### TC-D41: Request historical bars | Field | Value | | ------------------ | --------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, bar type configured. | | **Action** | DataTester requests historical bars for a configured `BarType`. | | **Event sequence** | Historical bars received via callback. | | **Pass criteria** | Bars received with valid OHLCV values and ascending timestamps. | | **Skip when** | Adapter does not support historical bar requests. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], bar_types=[BarType.from_str("BTCUSDT-PERP.VENUE-1-MINUTE-LAST-EXTERNAL")], request_bars=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .bar_types(vec![bar_type]) .request_bars(true) .build()? ``` --- ## Group 6: Derivatives data Test derivatives-specific data streams: mark prices, index prices, and funding rates. | TC | Name | Description | Skip when | | ------ | -------------------------------- | ----------------------------- | ----------------- | | TC-D50 | Subscribe mark prices | `MarkPriceUpdate` events. | Not a derivative. | | TC-D51 | Subscribe index prices | `IndexPriceUpdate` events. | Not a derivative. | | TC-D52 | Subscribe funding rates | `FundingRateUpdate` events. | Not a perpetual. | | TC-D53 | Request historical funding rates | Historical funding rate data. | Not a perpetual. | ### TC-D50: Subscribe mark prices | Field | Value | | ------------------ | -------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, derivative instrument loaded. | | **Action** | DataTester subscribes to mark price updates. | | **Event sequence** | `MarkPriceUpdate` events received in `on_mark_price`. | | **Pass criteria** | At least one `MarkPriceUpdate` received with valid instrument ID and mark price. | | **Skip when** | Instrument is not a derivative, or adapter does not provide mark prices. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_mark_prices=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_mark_prices(true) .build()? ``` ### TC-D51: Subscribe index prices | Field | Value | | ------------------ | ---------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, derivative instrument loaded. | | **Action** | DataTester subscribes to index price updates. | | **Event sequence** | `IndexPriceUpdate` events received in `on_index_price`. | | **Pass criteria** | At least one `IndexPriceUpdate` received with valid instrument ID and index price. | | **Skip when** | Instrument is not a derivative, or adapter does not provide index prices. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_index_prices=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_index_prices(true) .build()? ``` ### TC-D52: Subscribe funding rates | Field | Value | | ------------------ | ---------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, perpetual instrument loaded. | | **Action** | DataTester subscribes to funding rate updates. | | **Event sequence** | `FundingRateUpdate` events received in `on_funding_rate`. | | **Pass criteria** | At least one `FundingRateUpdate` received with valid instrument ID and rate. | | **Skip when** | Instrument is not a perpetual, or adapter does not provide funding rates. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_funding_rates=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_funding_rates(true) .build()? ``` ### TC-D53: Request historical funding rates | Field | Value | | ------------------ | -------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, perpetual instrument loaded. | | **Action** | DataTester requests historical funding rates (default 7-day lookback). | | **Event sequence** | Historical funding rates received via callback. | | **Pass criteria** | Funding rates received with valid timestamps and rate values. | | **Skip when** | Instrument is not a perpetual, or adapter does not support historical funding rate requests. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], request_funding_rates=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .request_funding_rates(true) .build()? ``` --- ## Group 7: Instrument status Test instrument status and close event subscriptions. | TC | Name | Description | Skip when | | ------ | --------------------------- | -------------------------- | ------------------ | | TC-D60 | Subscribe instrument status | `InstrumentStatus` events. | No status support. | | TC-D61 | Subscribe instrument close | `InstrumentClose` events. | No close support. | ### TC-D60: Subscribe instrument status | Field | Value | | ------------------ | ------------------------------------------------------------------------ | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester subscribes to instrument status updates. | | **Event sequence** | `InstrumentStatus` events received in `on_instrument_status`. | | **Pass criteria** | Status events received with valid `MarketStatusAction` (e.g. `Trading`). | | **Skip when** | Adapter does not support instrument status subscriptions. | **Considerations:** - Status events may only fire on state changes (e.g. trading halt -> resume). - During normal trading hours, a `Trading` status may be received on subscribe. **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_instrument_status=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_instrument_status(true) .build()? ``` ### TC-D61: Subscribe instrument close | Field | Value | | ------------------ | ----------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded. | | **Action** | DataTester subscribes to instrument close events. | | **Event sequence** | `InstrumentClose` events received in `on_instrument_close`. | | **Pass criteria** | Close event received with valid close price and close type. | | **Skip when** | Adapter does not support instrument close subscriptions. | **Considerations:** - Close events typically fire at end-of-session for traditional markets. - May not fire for 24/7 crypto venues unless the adapter synthesizes a daily close. **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_instrument_close=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_instrument_close(true) .build()? ``` --- ## Group 8: Option greeks Test option greeks and option chain subscriptions. | TC | Name | Description | Skip when | | ------ | ----------------------- | -------------------------------------------- | ------------------ | | TC-D62 | Subscribe option greeks | `OptionGreeks` data for a single instrument. | No greeks support. | | TC-D63 | Subscribe option chain | `OptionChainSlice` snapshots for a series. | No chain support. | ### TC-D62: Subscribe option greeks | Field | Value | | ------------------ | ------------------------------------------------------------ | | **Prerequisite** | Adapter connected, option instrument loaded. | | **Action** | DataTester subscribes to option greeks updates. | | **Event sequence** | `OptionGreeks` events received in `on_option_greeks`. | | **Pass criteria** | Greeks received with valid delta, gamma, vega, theta values. | | **Skip when** | Adapter does not support option greeks subscriptions. | **Considerations:** - Greeks are only available for option instruments. - Values depend on the venue's pricing model and may update on every quote change. - Some venues (Bybit, Deribit) subscribe per instrument; OKX subscribes per instrument family and filters to the requested instruments. - `rho` may be zero when the venue does not provide it (Bybit, OKX). - `underlying_price` and `open_interest` may be `None` depending on the venue channel. **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_option_greeks=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_option_greeks(true) .build()? ``` ### TC-D63: Subscribe option chain | Field | Value | | ------------------ | ------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, option instruments loaded for the series. | | **Action** | DataTester subscribes to option chain snapshots for a series. | | **Event sequence** | `OptionChainSlice` snapshots received in `on_option_chain`. | | **Pass criteria** | Chain snapshot contains greeks for instruments matching the series. | | **Skip when** | Adapter does not support option chain subscriptions. | **Considerations:** - Option chain subscriptions are managed by the DataEngine, which creates per-instrument quote and greeks subscriptions internally. - Dynamic strike ranges require an ATM price before instrument subscriptions begin. The DataEngine requests an initial reference price and otherwise waits for live option Greeks. - Not yet configurable via `DataTesterConfig`; requires manual actor setup with `subscribe_option_chain` and an `OptionSeriesId`. --- ## Group 9: Lifecycle Test actor lifecycle behavior: unsubscribe handling, retirement cleanup, and custom parameters. | TC | Name | Description | Skip when | | ------ | ----------------------- | ----------------------------------------------- | ----------------- | | TC-D70 | Unsubscribe on stop | Unsubscribe from data feeds on actor stop. | No unsub support. | | TC-D71 | Custom subscribe params | Adapter-specific subscription parameters. | N/A. | | TC-D72 | Custom request params | Adapter-specific request parameters. | N/A. | | TC-D73 | Retirement cleanup | Release an actor's retained data subscriptions. | N/A. | ### TC-D70: Unsubscribe on stop | Field | Value | | ------------------ | ---------------------------------------------------------------- | | **Prerequisite** | Active data subscriptions (quotes, trades, book). | | **Action** | Stop the actor with `can_unsubscribe=True` (default). | | **Event sequence** | Data subscriptions removed; no further data events received. | | **Pass criteria** | Clean unsubscribe; no errors in logs; no data events after stop. | | **Skip when** | Adapter does not support unsubscribe. | **Python config:** ```python DataTesterConfig( instrument_ids=[instrument_id], subscribe_quotes=True, subscribe_trades=True, can_unsubscribe=True, ) ``` **Rust config:** ```rust DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_quotes(true) .subscribe_trades(true) .can_unsubscribe(true) .build()? ``` ### TC-D71: Custom subscribe params | Field | Value | | ------------------ | ---------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, adapter accepts additional subscription parameters. | | **Action** | Subscribe with adapter-specific `subscribe_params`. | | **Event sequence** | Subscription established with custom parameters applied. | | **Pass criteria** | Data flows with adapter-specific parameters in effect. | | **Skip when** | N/A (adapter-specific). | **Rust config:** ```rust use nautilus_core::Params; use serde_json::json; let mut subscribe_params = Params::new(); subscribe_params.insert("key".to_string(), json!("value")); DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .subscribe_quotes(true) .subscribe_params(subscribe_params) .build()? ``` **Considerations:** - `subscribe_params` is opaque to the DataTester and passed through to the adapter. - The Python `DataTesterConfig` constructor does not expose this Rust-only field. - Consult the adapter's guide for supported parameters. ### TC-D72: Custom request params | Field | Value | | ------------------ | -------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, adapter accepts additional request parameters. | | **Action** | Request data with adapter-specific `request_params`. | | **Event sequence** | Request fulfilled with custom parameters applied. | | **Pass criteria** | Historical data received with adapter-specific parameters in effect. | | **Skip when** | N/A (adapter-specific). | **Rust config:** ```rust use nautilus_core::Params; use serde_json::json; let mut request_params = Params::new(); request_params.insert("key".to_string(), json!("value")); DataTesterConfig::builder() .client_id(client_id) .instrument_ids(vec![instrument_id]) .request_quotes(true) .request_params(request_params) .build()? ``` **Considerations:** - `request_params` is opaque to the DataTester and passed through to the adapter. - The Python `DataTesterConfig` constructor does not expose this Rust-only field. - Consult the adapter's guide for supported parameters. ### TC-D73: Retirement cleanup | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | An actor has venue-backed subscriptions; two actors share an internally aggregated bar. | | **Action** | Retire the first actor, then retire the second actor through the trader. | | **Event sequence** | `on_dispose` completes; unsubscribe commands are sent; the actor is deregistered. | | **Pass criteria** | The first retirement keeps shared data active; the final retirement releases the retained route and leaves no retired actor handlers. | | **Skip when** | N/A. | The shared bar must remain active after the first actor retires and stop after the final actor retires. **Considerations:** - `DataTesterConfig` does not cover multi-actor retirement. Create two actors manually, then remove them through Python `Controller.remove_actor` or Rust `Trader::remove_actor`. - If `on_dispose` fails, the actor must remain registered with its subscriptions intact so a later retirement can release them without invoking the failed hook again. - A failed `on_stop` or `on_fault` must not block retirement: disposal and deregistration must still complete from the corresponding transitional state. --- ## DataTester configuration reference The Python constructor accepts the parameters below. Defaults are resolved values after construction. Historical quote, trade, and bar requests use a one-hour lookback; funding rate requests use seven days. The lookback is not configurable through `DataTesterConfig`. | Parameter | Type | Default | Affects groups | | ----------------------------- | -------------------- | ------- | --------------- | | `actor_id` | `ActorId?` | `None` | All | | `client_id` | `ClientId?` | `None` | All | | `instrument_ids` | `list[InstrumentId]` | `[]` | All | | `bar_types` | `list[BarType]?` | `None` | 5 | | `subscribe_book_deltas` | `bool` | `False` | 2 | | `subscribe_book_depth` | `bool` | `False` | 2 | | `subscribe_book_at_interval` | `bool` | `False` | 2 | | `subscribe_quotes` | `bool` | `False` | 3 | | `subscribe_trades` | `bool` | `False` | 4 | | `subscribe_mark_prices` | `bool` | `False` | 6 | | `subscribe_index_prices` | `bool` | `False` | 6 | | `subscribe_funding_rates` | `bool` | `False` | 6 | | `subscribe_bars` | `bool` | `False` | 5 | | `subscribe_instrument` | `bool` | `False` | 1 | | `subscribe_instrument_status` | `bool` | `False` | 7 | | `subscribe_instrument_close` | `bool` | `False` | 7 | | `subscribe_option_greeks` | `bool` | `False` | 8 | | `can_unsubscribe` | `bool` | `True` | 9 | | `request_instruments` | `bool` | `False` | 1 | | `request_book_snapshot` | `bool` | `False` | 2 | | `request_book_deltas` | `bool` | `False` | Not implemented | | `request_quotes` | `bool` | `False` | 3 | | `request_trades` | `bool` | `False` | 4 | | `request_bars` | `bool` | `False` | 5 | | `request_funding_rates` | `bool` | `False` | 6 | | `book_depth` | `PositiveInt?` | `None` | 2 | | `book_interval_ms` | `PositiveInt` | `1000` | 2 | | `book_levels_to_print` | `PositiveInt` | `10` | 2 | | `manage_book` | `bool` | `True` | 2 | | `log_data` | `bool` | `True` | All | | `stats_interval_secs` | `int` | `5` | All | | `log_events` | `bool` | `True` | All | | `log_commands` | `bool` | `True` | All | The Rust builder also exposes these parameters: | Parameter | Type | Default | Affects groups | | ------------------ | ---------- | -------- | -------------- | | `book_type` | `BookType` | `L2_MBP` | 2 | | `subscribe_params` | `Params?` | `None` | 9 | | `request_params` | `Params?` | `None` | 9 | --- # Execution Testing Spec Source: https://nautilustrader.io/docs/latest/developer_guide/spec_exec_testing/ This section defines a rigorous test matrix for validating adapter execution functionality using the Rust `ExecTester` strategy. Python exposes it as a built-in strategy configured through `nautilus_trader.testkit.ExecTesterConfig`; Rust code imports it from `nautilus_testkit::testers`. Each test case is identified by a prefixed ID (e.g. TC-E01) and grouped by functionality. **Each adapter must pass the subset of tests matching its supported capabilities.** Tests progress from simple (single market order) to complex (brackets, modification chains, rejection handling). An adapter that passes groups 1-5 is considered baseline compliant. Data connectivity should be verified first using the [Data Testing Spec](spec_data_testing.md). Document adapter-specific behavior (how a venue simulates market orders, handles TIF options, etc.) in the adapter's own guide, not here. Each adapter guide should include a capability matrix showing which order types, time-in-force options, actions, and flags it supports. ## Prerequisites Before running execution tests: - Demo/testnet account with valid API credentials (preferred, not required). - Account funded with sufficient margin for the test instrument and quantities. - Target instrument available and loadable via the instrument provider. - Environment variables set: `{VENUE}_API_KEY`, `{VENUE}_API_SECRET` (or sandbox variants). - If the venue offers a demo/testnet mode, use credentials created for that environment. Demo and production API keys are typically separate and not interchangeable; using the wrong credentials produces authentication errors (e.g. HTTP 401). - Risk engine bypassed (`LiveRiskEngineConfig(bypass=True)`) to avoid interference. - Reconciliation enabled to verify state consistency. **Python node setup**: Use `nautilus_trader.live.LiveNode`. Call `LiveNode.builder(...)` when you need to register adapter client factories before the node is built. ```python from nautilus_trader.common import Environment from nautilus_trader.config import LiveExecutionEngineConfig from nautilus_trader.config import LiveRiskEngineConfig from nautilus_trader.live import LiveNode from nautilus_trader.model import TraderId from nautilus_trader.testkit import ExecTesterConfig node = ( LiveNode.builder("TESTER-001", TraderId("TESTER-001"), Environment.SANDBOX) .with_risk_engine_config(LiveRiskEngineConfig(bypass=True)) .with_exec_engine_config(LiveExecutionEngineConfig(reconciliation=True)) .add_exec_client(None, adapter_exec_client_factory, exec_client_config) .build() ) tester_config = ExecTesterConfig( instrument_id=instrument_id, client_id=client_id, order_qty=order_qty, ) node.add_builtin_strategy("ExecTester", tester_config) # Register remaining components, then start or run ``` **Rust node setup** (reference: `crates/adapters/{adapter}/examples/node_exec_tester.rs`): ```rust use nautilus_testkit::testers::{ExecTester, ExecTesterConfig}; use nautilus_trading::strategy::StrategyConfig; let tester_config = ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(order_qty) .build()?; let tester = ExecTester::new(tester_config); node.add_strategy(tester)?; node.run().await?; ``` ## Timestamp scale Nautilus stores `ts_event` and `ts_init` as Unix nanoseconds (`UnixNanos`). Every execution message must use that scale: order events, fills, account state, and reconciliation reports (`OrderStatusReport`, `FillReport`, `PositionStatusReport`, `ExecutionMassStatus`). - A value below `10^16` is not a plausible Unix-nanosecond timestamp (`10^16` ns is about 116 days after 1970-01-01) and usually means the adapter left the venue scale unconverted. - Second-precision venue times that were converted correctly end in `000000000` and still pass: that is coarse precision, not a scale error. - Live stream `ts_event` should be near wall-clock time for the session. Reconciliation reports may be older and still valid if the scale is nanoseconds. - `ts_init` is the local clock when Nautilus created the object. Small `ts_event` > `ts_init` skew is possible when the venue clock is ahead. `ExecTester` warns when received market-data or order-event timestamps fail the scale check. Inspect report timestamps on the reports themselves. Treat a scale warning or leftover seconds, milliseconds, or microseconds on any execution message as a failure. ## Basic smoke test A quick sanity check that can run at any time, for example after adapter changes or between development iterations. The tester opens a position with a market order on start, places a buy and sell post-only limit order, waits 30 seconds, then stops (cancelling open orders and closing the position). **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.001"), open_position_on_start_qty=Decimal("0.001"), enable_limit_buys=True, enable_limit_sells=True, use_post_only=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.001")) .open_position_on_start_qty(dec!(0.001)) .enable_limit_buys(true) .enable_limit_sells(true) .use_post_only(true) .build()? ``` **Expected behavior:** 1. On start: market order fills, opening a position. 2. Two limit orders placed at `tob_offset_ticks` away from best bid/ask (default 500 ticks). 3. Strategy idles for 30 seconds. Check logs for errors, rejected orders, or disconnections. 4. On stop: open limit orders cancelled, position closed with a market order. **Pass criteria:** No errors in logs, position opened and closed cleanly, and limit orders acknowledged by the venue. --- Each group below begins with a summary table, followed by detailed test cards. Test IDs use spaced numbering to allow insertion without renumbering. --- ## Group 1: Market orders Test market order submission and fills. Market orders should execute immediately. | TC | Name | Description | Skip when | | ------ | ----------------------------- | --------------------------------------------------- | ------------------ | | TC-E01 | Market BUY - submit and fill | Open long position via market buy. | No market orders. | | TC-E02 | Market SELL - submit and fill | Open short position via market sell. | No market orders. | | TC-E03 | Market order with IOC TIF | Market order explicitly using IOC time in force. | No IOC. | | TC-E04 | Market order with FOK TIF | Market order explicitly using FOK time in force. | No FOK. | | TC-E05 | Market order with quote qty | Market order using quote currency quantity. | No quote quantity. | | TC-E06 | Close position via market | Close an open position with a market order on stop. | No market orders. | ### TC-E01: Market BUY - submit and fill | Field | Value | | ------------------ | ---------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, market data flowing, no open position. | | **Action** | ExecTester opens a long position via `open_position_on_start_qty`. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled`. | | **Pass criteria** | Position opened with side=LONG, quantity matches config, fill price within market range, `AccountState` updated. | | **Skip when** | Adapter does not support market orders. | **Considerations:** - Some adapters simulate market orders as aggressive limit IOC orders (check adapter guide). - The event sequence from the strategy's perspective should be identical regardless of the venue mechanism. - Fill price should be within the recent bid/ask spread. - Partial fills are valid; verify the cumulative filled quantity matches the order quantity. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), open_position_on_start_qty=Decimal("0.01"), enable_limit_buys=False, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .open_position_on_start_qty(Decimal::new(1, 2)) .enable_limit_buys(false) .enable_limit_sells(false) .build()? ``` ### TC-E02: Market SELL - submit and fill | Field | Value | | ------------------ | ----------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, market data flowing, no open position. | | **Action** | ExecTester opens a short position via negative `open_position_on_start_qty`. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled`. | | **Pass criteria** | Position opened with side=SHORT, quantity matches config, fill price within market range. | | **Skip when** | Adapter does not support market orders or short selling. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), open_position_on_start_qty=Decimal("-0.01"), enable_limit_buys=False, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .open_position_on_start_qty(Decimal::new(-1, 2)) .enable_limit_buys(false) .enable_limit_sells(false) .build()? ``` ### TC-E03: Market order with IOC TIF | Field | Value | | ------------------ | --------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, market data flowing. | | **Action** | Open position with `open_position_time_in_force=IOC`. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled`. | | **Pass criteria** | Same as TC-E01; the IOC TIF is explicitly set on the order. | | **Skip when** | No IOC support. | **Considerations:** - Some adapters simulate market orders as aggressive limit IOC. An unfilled simulated market maps to `OrderCanceled`, not `OrderRejected`. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), open_position_on_start_qty=Decimal("0.01"), open_position_time_in_force=TimeInForce.IOC, enable_limit_buys=False, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .open_position_on_start_qty(Decimal::new(1, 2)) .enable_limit_buys(false) .enable_limit_sells(false) .open_position_time_in_force(TimeInForce::Ioc) .build()? ``` ### TC-E04: Market order with FOK TIF | Field | Value | | ------------------ | --------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, market data flowing. | | **Action** | Open position with `open_position_time_in_force=FOK`. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled`. | | **Pass criteria** | Same as TC-E01; the FOK TIF is explicitly set on the order. | | **Skip when** | No FOK support. | **Considerations:** - FOK requires the entire quantity to be fillable immediately or the order is canceled. - Use small test quantities so book depth is sufficient for a complete fill. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), open_position_on_start_qty=Decimal("0.01"), open_position_time_in_force=TimeInForce.FOK, enable_limit_buys=False, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .open_position_on_start_qty(Decimal::new(1, 2)) .enable_limit_buys(false) .enable_limit_sells(false) .open_position_time_in_force(TimeInForce::Fok) .build()? ``` ### TC-E05: Market order with quote quantity | Field | Value | | ------------------ | -------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, adapter supports quote quantity. | | **Action** | Open position with `use_quote_quantity=True`, quantity in quote currency. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled`. | | **Pass criteria** | Order submitted with quote currency quantity; fill quantity is in base currency. | | **Skip when** | Adapter does not support quote quantity orders. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("100.0"), # Quote currency amount open_position_on_start_qty=Decimal("100.0"), use_quote_quantity=True, enable_limit_buys=False, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("100")) .open_position_on_start_qty(Decimal::from(100)) .use_quote_quantity(true) .enable_limit_buys(false) .enable_limit_sells(false) .build()? ``` ### TC-E06: Close position via market order on stop | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------- | | **Prerequisite** | Open position from TC-E01 or TC-E02. | | **Action** | Stop the strategy; ExecTester closes position via market order. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled` (closing order). | | **Pass criteria** | Position closed, or only the exact sub-precision residual remains; no open orders remain. | | **Skip when** | Adapter does not support market orders. | **Considerations:** - This test naturally follows TC-E01 or TC-E02 as part of the same session. - `close_positions_on_stop=True` is the default. - The closing order should be on the opposite side of the position. - Set `close_positions_qty_precision` when the venue accepts fewer size decimals than the instrument. The tester closes only that venue-fillable quantity and logs any exact residual. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), open_position_on_start_qty=Decimal("0.01"), close_positions_on_stop=True, enable_limit_buys=False, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .open_position_on_start_qty(Decimal::new(1, 2)) .close_positions_on_stop(true) .enable_limit_buys(false) .enable_limit_sells(false) .build()? ``` --- ## Group 2: Limit orders Test limit order submission, acceptance, and behavior across time-in-force options. | TC | Name | Description | Skip when | | ------ | ------------------------- | ------------------------------------------------ | --------- | | TC-E10 | Limit BUY GTC | Place GTC limit buy below TOB, verify accepted. | Never. | | TC-E11 | Limit SELL GTC | Place GTC limit sell above TOB, verify accepted. | Never. | | TC-E12 | Limit BUY and SELL pair | Both sides simultaneously, verify both accepted. | Never. | | TC-E13 | Limit IOC aggressive fill | Limit IOC at aggressive price, expect fill. | No IOC. | | TC-E14 | Limit IOC passive no fill | Limit IOC away from market, expect cancel. | No IOC. | | TC-E15 | Limit FOK fill | Limit FOK at aggressive price, expect fill. | No FOK. | | TC-E16 | Limit FOK no fill | Limit FOK away from market, expect cancel. | No FOK. | | TC-E17 | Limit GTD | Limit with expiry time, verify accepted. | No GTD. | | TC-E18 | Limit GTD expiry | Verify documented terminal event at expiry. | No GTD. | | TC-E19 | Limit DAY | Limit with DAY TIF, verify accepted. | No DAY. | ### TC-E10: Limit BUY GTC - submit and accept | Field | Value | | ------------------ | --------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester places a limit buy at `best_bid - tob_offset_ticks`. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order is open on the venue with correct price, quantity, side=BUY, TIF=GTC. | | **Skip when** | Never. | **Considerations:** - The `tob_offset_ticks` (default 500) places the order well away from the market to avoid accidental fills. - Verify the order appears in the cache with `OrderStatus.ACCEPTED`. - The order should remain open until explicitly canceled. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(false) .build()? ``` ### TC-E11: Limit SELL GTC - submit and accept | Field | Value | | ------------------ | ---------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester places a limit sell at `best_ask + tob_offset_ticks`. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order is open on the venue with correct price, quantity, side=SELL, TIF=GTC. | | **Skip when** | Never. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=False, enable_limit_sells=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(false) .enable_limit_sells(true) .build()? ``` ### TC-E12: Limit BUY and SELL pair | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------ | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester places both a limit buy and limit sell. | | **Event sequence** | Two independent sequences: each `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Both orders open on venue, buy below bid, sell above ask. | | **Skip when** | Never. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(true) .build()? ``` ### TC-E13: Limit IOC aggressive fill | Field | Value | | ------------------ | --------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | Submit a limit buy IOC at or above the best ask (aggressive price). | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled`. | | **Pass criteria** | Order fills immediately; position opened. | | **Skip when** | Adapter does not support IOC TIF. | **Considerations:** - This test requires manual order creation or adapter-specific configuration, as the ExecTester's default limit order placement uses GTC TIF. - IOC orders that don't fill immediately are canceled by the venue. ### TC-E14: Limit IOC passive - no fill | Field | Value | | ------------------ | ----------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | Submit a limit buy IOC well below the market (passive price). | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderCanceled`. | | **Pass criteria** | Order is immediately canceled by venue with no fill. | | **Skip when** | Adapter does not support IOC TIF. | **Considerations:** - The venue should cancel the unfilled IOC order; verify `OrderCanceled` (not `OrderExpired` or `OrderRejected`). ### TC-E15: Limit FOK fill | Field | Value | | ------------------ | ---------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing, sufficient book depth. | | **Action** | Submit a limit buy FOK at aggressive price with quantity within top-of-book depth. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled`. | | **Pass criteria** | Order fills completely in a single fill event. | | **Skip when** | Adapter does not support FOK TIF. | **Considerations:** - FOK requires the entire quantity to be fillable; use small quantities so book depth is sufficient. ### TC-E16: Limit FOK no fill | Field | Value | | ------------------ | ----------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | Submit a limit buy FOK at passive price (well below market). | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderCanceled`. | | **Pass criteria** | Order is immediately canceled by venue with no fill. | | **Skip when** | Adapter does not support FOK TIF. | ### TC-E17: Limit GTD - submit and accept | Field | Value | | ------------------ | --------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | Place limit buy with `order_expire_time_delta_mins` set (e.g., 60 minutes). | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order accepted with GTD TIF and correct expiry timestamp. | | **Skip when** | Adapter does not support GTD TIF. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), order_expire_time_delta_mins=60, enable_limit_buys=True, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(false) .order_expire_time_delta_mins(60) .build()? ``` ### TC-E18: Limit GTD expiry | Field | Value | | ------------------ | --------------------------------------------------------------------------- | | **Prerequisite** | Open GTD limit order from TC-E17 using the shortest venue-supported expiry. | | **Action** | Wait for the GTD expiry time to elapse. | | **Event sequence** | `OrderExpired` by default, or the terminal event documented by the adapter. | | **Pass criteria** | Order reaches the adapter-documented terminal status at venue expiry. | | **Skip when** | Adapter does not support GTD TIF. | **Considerations:** - Use the shortest expiry accepted by the venue; do not assume one or two minutes is valid. - Some venues report GTD expiry as a cancel. Preserve and verify the documented adapter mapping instead of normalizing every venue to `OrderExpired`. ### TC-E19: Limit DAY - submit and accept | Field | Value | | ------------------ | ---------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, market is in trading hours. | | **Action** | Submit limit buy with DAY TIF. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order accepted with DAY TIF; will be automatically canceled at end of trading day. | | **Skip when** | Adapter does not support DAY TIF. | **Considerations:** - DAY orders may behave differently on 24/7 crypto venues vs traditional markets. - Verify behavior when submitted outside trading hours (if applicable). --- ## Group 3: Stop and conditional orders Test stop and conditional order types. These orders rest on the venue until a trigger condition is met. Adapters that support venue-native conditional orders should also verify that open trigger orders appear in restart reconciliation, not only in the normal open-order endpoint. | TC | Name | Description | Skip when | | ------ | -------------------- | ------------------------------------------- | ----------------- | | TC-E20 | StopMarket BUY | Stop buy above ask, verify accepted. | No `STOP_MARKET`. | | TC-E21 | StopMarket SELL | Stop sell below bid, verify accepted. | No `STOP_MARKET`. | | TC-E22 | StopLimit BUY | Stop-limit buy with trigger + limit price. | No `STOP_LIMIT`. | | TC-E23 | StopLimit SELL | Stop-limit sell with trigger + limit price. | No `STOP_LIMIT`. | | TC-E24 | MarketIfTouched BUY | MIT buy below bid. | No `MIT`. | | TC-E25 | MarketIfTouched SELL | MIT sell above ask. | No `MIT`. | | TC-E26 | LimitIfTouched BUY | LIT buy with trigger + limit price. | No `LIT`. | | TC-E27 | LimitIfTouched SELL | LIT sell with trigger + limit price. | No `LIT`. | ### TC-E20: StopMarket BUY | Field | Value | | ------------------ | --------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester places a stop-market buy above the current ask. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Stop order accepted on venue with correct trigger price and side=BUY. | | **Skip when** | Adapter does not support `StopMarket` orders. | **Considerations:** - The trigger price should be above the current ask by `stop_offset_ticks`. - The order should NOT trigger immediately (trigger price is above market). - For venues with long-lived trigger signatures, verify the trigger-order signing expiry uses the venue's trigger-order window rather than the normal order expiry. - Verifying trigger and fill requires the market to move, which may not happen during the test. - After acceptance, restart or force reconciliation and verify the order still appears as an open order report when the venue keeps trigger orders in a separate endpoint. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=False, enable_limit_sells=False, enable_stop_buys=True, enable_stop_sells=False, stop_order_type=OrderType.STOP_MARKET, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(false) .enable_limit_sells(false) .enable_stop_buys(true) .enable_stop_sells(false) .stop_order_type(OrderType::StopMarket) .build()? ``` ### TC-E21: StopMarket SELL | Field | Value | | ------------------ | ---------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester places a stop-market sell below the current bid. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Stop order accepted on venue with correct trigger price and side=SELL. | | **Skip when** | Adapter does not support `StopMarket` orders. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=False, enable_limit_sells=False, enable_stop_buys=False, enable_stop_sells=True, stop_order_type=OrderType.STOP_MARKET, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(false) .enable_limit_sells(false) .enable_stop_buys(false) .enable_stop_sells(true) .stop_order_type(OrderType::StopMarket) .build()? ``` ### TC-E22: StopLimit BUY | Field | Value | | ------------------ | --------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester places a stop-limit buy with trigger price above ask and limit offset. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Stop-limit order accepted with correct trigger price, limit price, and side=BUY. | | **Skip when** | Adapter does not support `StopLimit` orders. | **Considerations:** - Requires `stop_limit_offset_ticks` to be set for the limit price offset from the trigger price. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=False, enable_limit_sells=False, enable_stop_buys=True, enable_stop_sells=False, stop_order_type=OrderType.STOP_LIMIT, stop_limit_offset_ticks=50, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(false) .enable_limit_sells(false) .enable_stop_buys(true) .enable_stop_sells(false) .stop_order_type(OrderType::StopLimit) .stop_limit_offset_ticks(50) .build()? ``` ### TC-E23: StopLimit SELL | Field | Value | | ------------------ | --------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester places a stop-limit sell with trigger price below bid. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Stop-limit order accepted with correct trigger price, limit price, and side=SELL. | | **Skip when** | Adapter does not support `StopLimit` orders. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=False, enable_limit_sells=False, enable_stop_buys=False, enable_stop_sells=True, stop_order_type=OrderType.STOP_LIMIT, stop_limit_offset_ticks=50, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(false) .enable_limit_sells(false) .enable_stop_buys(false) .enable_stop_sells(true) .stop_order_type(OrderType::StopLimit) .stop_limit_offset_ticks(50) .build()? ``` ### TC-E24: MarketIfTouched BUY | Field | Value | | ------------------ | ---------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | Place MIT buy with trigger below current bid (buy on dip). | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | MIT order accepted on venue with correct trigger price. | | **Skip when** | Adapter does not support `MarketIfTouched` orders. | ### TC-E25: MarketIfTouched SELL | Field | Value | | ------------------ | -------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | Place MIT sell with trigger above current ask (sell on rally). | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | MIT order accepted on venue with correct trigger price. | | **Skip when** | Adapter does not support `MarketIfTouched` orders. | ### TC-E26: LimitIfTouched BUY | Field | Value | | ------------------ | -------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | Place LIT buy with trigger below bid and limit price offset. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | LIT order accepted with correct trigger price and limit price. | | **Skip when** | Adapter does not support `LimitIfTouched` orders. | ### TC-E27: LimitIfTouched SELL | Field | Value | | ------------------ | -------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | Place LIT sell with trigger above ask and limit price offset. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | LIT order accepted with correct trigger price and limit price. | | **Skip when** | Adapter does not support `LimitIfTouched` orders. | --- ## Group 4: Order modification Test order modification (amend) and cancel-replace workflows. | TC | Name | Description | Skip when | | ------ | ------------------------- | ---------------------------------------------- | ------------------------ | | TC-E30 | Modify limit BUY price | Amend open limit buy to new price. | No modify support. | | TC-E31 | Modify limit SELL price | Amend open limit sell to new price. | No modify support. | | TC-E32 | Cancel-replace limit BUY | Cancel and resubmit limit buy at new price. | Never. | | TC-E33 | Cancel-replace limit SELL | Cancel and resubmit limit sell at new price. | Never. | | TC-E34 | Modify stop trigger price | Amend stop order trigger price. | No modify or no stop. | | TC-E35 | Cancel-replace stop order | Cancel and resubmit stop at new trigger price. | No stop orders. | | TC-E36 | Modify rejected | Modify on unsupported adapter. | Adapter supports modify. | `trigger_limit_order_maintenance_once=True` requires exactly one of the limit-order modify or cancel-replace modes and at least one enabled limit side. It is incompatible with bracket entries, `test_modify_rejected`, and two-sided batch limit submission. Configuration validation rejects these combinations. ### TC-E30: Modify limit BUY price | Field | Value | | ------------------ | ---------------------------------------------------------------------------- | | **Prerequisite** | Open GTC limit buy from TC-E10. | | **Action** | ExecTester modifies the open limit buy to a new price. | | **Event sequence** | `OrderPendingUpdate` -> `OrderUpdated`. | | **Pass criteria** | `OrderUpdated` event logged with the new price; order exits `PendingUpdate`. | | **Skip when** | Adapter does not support order modification. | **Considerations:** - `trigger_limit_order_maintenance_once=True` amends to a different valid price directly from the first `OrderAccepted`, preferring one tick more passive. A quiet book is not inconclusive. - The one-shot workflow stops further limit maintenance for that side after it starts, so the tester does not immediately amend the order back to the TOB target. - Without that flag, the modify waits for the order price to drift from the target TOB offset. No movement and no `OrderUpdated` is inconclusive, not a failure. - Verify the `OrderUpdated` log shows the expected price. If the event never arrives after a forced or drift-triggered amend, the order stays in `PendingUpdate` and the tester stops modifying it. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=False, modify_orders_to_maintain_tob_offset=True, trigger_limit_order_maintenance_once=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(false) .modify_orders_to_maintain_tob_offset(true) .trigger_limit_order_maintenance_once(true) .build()? ``` ### TC-E31: Modify limit SELL price | Field | Value | | ------------------ | ---------------------------------------------------------------------------- | | **Prerequisite** | Open GTC limit sell from TC-E11. | | **Action** | ExecTester modifies the open limit sell to a new price. | | **Event sequence** | `OrderPendingUpdate` -> `OrderUpdated`. | | **Pass criteria** | `OrderUpdated` event logged with the new price; order exits `PendingUpdate`. | | **Skip when** | Adapter does not support order modification. | **Considerations:** - Same one-shot trigger and quiet-book guidance as TC-E30. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=False, enable_limit_sells=True, modify_orders_to_maintain_tob_offset=True, trigger_limit_order_maintenance_once=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(false) .enable_limit_sells(true) .modify_orders_to_maintain_tob_offset(true) .trigger_limit_order_maintenance_once(true) .build()? ``` ### TC-E32: Cancel-replace limit BUY | Field | Value | | ------------------ | ----------------------------------------------------------------------------------------------------- | | **Prerequisite** | Open GTC limit buy. | | **Action** | ExecTester cancels and resubmits the limit buy at a new price. | | **Event sequence** | `OrderPendingCancel` -> `OrderCanceled` -> `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Original order canceled, new order accepted at updated price. | | **Skip when** | Never (cancel-replace is always available). | **Considerations:** - This is the universal alternative when the adapter does not support native modify. - Two distinct orders in the cache: the canceled original and the new replacement. - `trigger_limit_order_maintenance_once=True` requests one cancel directly from the first `OrderAccepted`, then submits the replacement only after `OrderCanceled`. - Without the one-shot trigger, cancel-replace waits for TOB drift. No movement and no replacement is inconclusive, not a failure. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=False, cancel_replace_orders_to_maintain_tob_offset=True, trigger_limit_order_maintenance_once=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(false) .cancel_replace_orders_to_maintain_tob_offset(true) .trigger_limit_order_maintenance_once(true) .build()? ``` ### TC-E33: Cancel-replace limit SELL | Field | Value | | ------------------ | ----------------------------------------------------------------------------------------------------- | | **Prerequisite** | Open GTC limit sell. | | **Action** | ExecTester cancels and resubmits the limit sell at a new price. | | **Event sequence** | `OrderPendingCancel` -> `OrderCanceled` -> `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Original order canceled, new order accepted at updated price. | | **Skip when** | Never. | **Considerations:** - Same one-shot trigger and quiet-book guidance as TC-E32. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=False, enable_limit_sells=True, cancel_replace_orders_to_maintain_tob_offset=True, trigger_limit_order_maintenance_once=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(false) .enable_limit_sells(true) .cancel_replace_orders_to_maintain_tob_offset(true) .trigger_limit_order_maintenance_once(true) .build()? ``` ### TC-E34: Modify stop trigger price | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------ | | **Prerequisite** | Open stop order from TC-E20 or TC-E22. | | **Action** | ExecTester modifies stop trigger price as market moves (`modify_stop_orders_to_maintain_offset=True`). | | **Event sequence** | `OrderPendingUpdate` -> `OrderUpdated`. | | **Pass criteria** | `OrderUpdated` event logged with the new trigger price; order exits `PendingUpdate`. | | **Skip when** | Adapter does not support native stop modify, or no stop order support. | **Considerations:** - Some venues allow limit-order modify but reject trigger-order replace. For those adapters, skip TC-E34 and run TC-E35 instead. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_stop_buys=True, modify_stop_orders_to_maintain_offset=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_stop_buys(true) .modify_stop_orders_to_maintain_offset(true) .build()? ``` ### TC-E35: Cancel-replace stop order | Field | Value | | ------------------ | ----------------------------------------------------------------------------------------------------- | | **Prerequisite** | Open stop order. | | **Action** | ExecTester cancels and resubmits stop at new trigger price. | | **Event sequence** | `OrderPendingCancel` -> `OrderCanceled` -> `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Original stop canceled, new stop accepted at updated trigger price. | | **Skip when** | No stop order support. | **Considerations:** - This is the required path for venues that do not support native trigger-order replace. - After the new stop is accepted, restart or force reconciliation and verify exactly one current trigger order remains. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_stop_buys=True, cancel_replace_stop_orders_to_maintain_offset=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_stop_buys(true) .cancel_replace_stop_orders_to_maintain_offset(true) .build()? ``` ### TC-E36: Modify rejected | Field | Value | | ------------------ | ---------------------------------------------------------------------------------------------------- | | **Prerequisite** | Open limit order, adapter does NOT support modify. | | **Action** | Attempt to modify the order (programmatically, not via ExecTester auto-maintain). | | **Event sequence** | `OrderModifyRejected`. | | **Pass criteria** | Modify attempt results in `OrderModifyRejected` event with reason; original order remains unchanged. | | **Skip when** | Adapter supports order modification. | **Considerations:** - This tests the adapter's rejection path, not the ExecTester's cancel-replace logic. - The rejection reason should indicate that modification is not supported. --- ## Group 5: Order cancellation Test order cancellation workflows. | TC | Name | Description | Skip when | | ------ | -------------------------- | ------------------------------------------------- | ---------------- | | TC-E40 | Cancel single limit order | Cancel an open limit order. | Never. | | TC-E41 | Cancel all on stop | Strategy stop cancels all open orders (default). | Never. | | TC-E42 | Individual cancels on stop | Cancel orders one-by-one on stop. | Never. | | TC-E43 | Batch cancel on stop | Cancel orders via batch API on stop. | No batch cancel. | | TC-E44 | Cancel already-canceled | Verify documented rejection or idempotent result. | Never. | ### TC-E40: Cancel single limit order | Field | Value | | ------------------ | --------------------------------------------------------------- | | **Prerequisite** | Open GTC limit order from TC-E10 or TC-E11. | | **Action** | Stop the strategy; ExecTester cancels the open limit order. | | **Event sequence** | `OrderPendingCancel` -> `OrderCanceled`. | | **Pass criteria** | Order status transitions to CANCELED; no open orders remaining. | | **Skip when** | Never. | **Considerations:** - `cancel_orders_on_stop=True` (default) triggers cancellation when the strategy stops. - Verify the `OrderCanceled` event contains the correct `venue_order_id`. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=False, cancel_orders_on_stop=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(false) .cancel_orders_on_stop(true) .build()? ``` ### TC-E41: Cancel all on stop | Field | Value | | ------------------ | -------------------------------------------------------------- | | **Prerequisite** | Multiple open orders (limit buy + limit sell from TC-E12). | | **Action** | Stop the strategy with `cancel_orders_on_stop=True` (default). | | **Event sequence** | For each order: `OrderPendingCancel` -> `OrderCanceled`. | | **Pass criteria** | All open orders canceled; no open orders remaining. | | **Skip when** | Never. | **Considerations:** - Default stop uses `cancel_all`. Some venues implement that as account-wide. Confirm the adapter's `CancelAllOrders` scope before running beside other open orders. Use `use_individual_cancels_on_stop` or `use_batch_cancel_on_stop` to isolate. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=True, cancel_orders_on_stop=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(true) .cancel_orders_on_stop(true) .build()? ``` ### TC-E42: Individual cancels on stop | Field | Value | | ------------------ | ------------------------------------------------------------------- | | **Prerequisite** | Multiple open orders. | | **Action** | Stop with `use_individual_cancels_on_stop=True`. | | **Event sequence** | Individual `OrderPendingCancel` -> `OrderCanceled` for each order. | | **Pass criteria** | Each order canceled individually; all orders reach CANCELED status. | | **Skip when** | Never. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=True, use_individual_cancels_on_stop=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(true) .use_individual_cancels_on_stop(true) .build()? ``` ### TC-E43: Batch cancel on stop | Field | Value | | ------------------ | ------------------------------------------------------------------------ | | **Prerequisite** | Multiple open orders, adapter supports batch cancel. | | **Action** | Stop with `use_batch_cancel_on_stop=True`. | | **Event sequence** | Batch `OrderPendingCancel` -> `OrderCanceled` for all orders. | | **Pass criteria** | All orders canceled via single batch request; all reach CANCELED status. | | **Skip when** | Adapter does not support batch cancel. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=True, use_batch_cancel_on_stop=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(true) .use_batch_cancel_on_stop(true) .build()? ``` ### TC-E44: Cancel already-canceled order | Field | Value | | ------------------ | ------------------------------------------------------------------------------------- | | **Prerequisite** | A previously canceled order (from TC-E40). | | **Action** | Attempt to cancel the same order again. | | **Event sequence** | `OrderCancelRejected` by default, or no new event for a documented idempotent result. | | **Pass criteria** | Result matches the adapter contract without a duplicate terminal event. | | **Skip when** | Never. | **Considerations:** - The default contract tests the adapter's error handling for invalid cancel requests. The rejection reason should indicate the order is not in a cancelable state. - `Strategy.cancel_order` and the order manager drop cancels for locally closed orders, so `ExecTester` cannot reach the adapter after TC-E40. A second `cancel_order` that never leaves the process is inconclusive. - A venue may treat an already-terminal cancel as idempotent. Document that disposition and use an adapter-focused test that submits `CancelOrder` with the venue order ID after the order is terminal. --- ## Group 6: Bracket orders Test bracket order submission (entry + take-profit + stop-loss). | TC | Name | Description | Skip when | | ------ | ---------------------------- | ----------------------------------------------- | ------------------- | | TC-E50 | Bracket BUY | Entry limit buy + TP limit sell + SL stop sell. | No bracket support. | | TC-E51 | Bracket SELL | Entry limit sell + TP limit buy + SL stop buy. | No bracket support. | | TC-E52 | Bracket entry fill activates | Verify TP/SL become active after entry fill. | No bracket support. | | TC-E53 | Bracket with post-only entry | Entry order uses post-only flag. | No bracket or PO. | ### TC-E50: Bracket BUY | Field | Value | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester submits a bracket order: limit buy entry + take-profit sell + stop-loss sell. | | **Event sequence** | Entry: `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`; TP and SL: `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Three orders created and accepted: entry below bid, TP above ask, SL below entry. | | **Skip when** | Adapter does not support bracket orders. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_brackets=True, bracket_entry_order_type=OrderType.LIMIT, bracket_offset_ticks=500, enable_limit_buys=True, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_brackets(true) .bracket_entry_order_type(OrderType::Limit) .bracket_offset_ticks(500) .enable_limit_buys(true) .enable_limit_sells(false) .build()? ``` ### TC-E51: Bracket SELL | Field | Value | | ------------------ | --------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester submits bracket: limit sell entry + TP buy + SL buy. | | **Event sequence** | Same pattern as TC-E50 but for sell side. | | **Pass criteria** | Three orders created and accepted on sell side. | | **Skip when** | Adapter does not support bracket orders. | ### TC-E52: Bracket entry fill activates TP/SL | Field | Value | | ------------------ | --------------------------------------------------------------------- | | **Prerequisite** | Bracket order from TC-E50 where entry order fills. | | **Action** | Entry order fills; verify contingent TP and SL orders activate. | | **Event sequence** | Entry: `OrderFilled`; TP and SL transition from contingent to active. | | **Pass criteria** | After entry fill, TP and SL orders are live on the venue. | | **Skip when** | Adapter does not support bracket orders. | **Considerations:** - This requires the entry order to actually fill, which may need aggressive pricing. - The TP/SL activation mechanism varies by venue (some activate immediately, some are OCA groups). ### TC-E53: Bracket with post-only entry | Field | Value | | ------------------ | ----------------------------------------------------------------------- | | **Prerequisite** | Adapter supports brackets and post-only. | | **Action** | Submit bracket with `use_post_only=True` (applied to entry and TP). | | **Event sequence** | Same as TC-E50 with post-only flag on entry. | | **Pass criteria** | Entry and TP orders accepted as post-only (maker); SL is not post-only. | | **Skip when** | No bracket support or no post-only support. | --- ## Group 7: Order flags Test order-level flags and special parameters. | TC | Name | Description | Skip when | | ------ | ------------------- | -------------------------------------------- | -------------------- | | TC-E60 | PostOnly accepted | Limit with post-only, placed away from TOB. | No post-only. | | TC-E61 | ReduceOnly on close | Close position with reduce-only flag. | No reduce-only. | | TC-E62 | Display quantity | Iceberg order with visible quantity < total. | No display quantity. | | TC-E63 | Custom order params | Adapter-specific params via `order_params`. | N/A. | ### TC-E60: PostOnly accepted | Field | Value | | ------------------ | ----------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester places limit buy with `use_post_only=True` at passive price. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order accepted as a maker order; post-only flag acknowledged by venue. | | **Skip when** | Adapter does not support post-only flag. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=False, use_post_only=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(false) .use_post_only(true) .build()? ``` ### TC-E61: ReduceOnly on close | Field | Value | | ------------------ | ---------------------------------------------------------------------------------------------- | | **Prerequisite** | Open position (from TC-E01). | | **Action** | Stop strategy with `reduce_only_on_stop=True`; closing order uses reduce-only flag. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled` (with reduce-only). | | **Pass criteria** | Closing order has reduce-only flag; position fully closed. | | **Skip when** | Adapter does not support reduce-only flag. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), open_position_on_start_qty=Decimal("0.01"), reduce_only_on_stop=True, close_positions_on_stop=True, enable_limit_buys=False, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .open_position_on_start_qty(Decimal::new(1, 2)) .reduce_only_on_stop(true) .close_positions_on_stop(true) .enable_limit_buys(false) .enable_limit_sells(false) .build()? ``` ### TC-E62: Display quantity (iceberg) | Field | Value | | ------------------ | ------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, adapter supports display quantity. | | **Action** | Place limit order with `order_display_qty` < `order_qty`. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order accepted with display quantity set; only display qty visible on the book. | | **Skip when** | Adapter does not support display quantity / iceberg orders. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("1.0"), order_display_qty=Quantity.from_str("0.1"), enable_limit_buys=True, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("1.0")) .enable_limit_buys(true) .enable_limit_sells(false) .order_display_qty(Quantity::from("0.1")) .build()? ``` ### TC-E63: Custom order params | Field | Value | | ------------------ | ---------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, adapter accepts additional parameters. | | **Action** | Place order with `order_params` dict containing adapter-specific parameters. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order accepted; adapter-specific parameters passed through to venue. | | **Skip when** | N/A (adapter-specific). | **Considerations:** - The `order_params` dict is opaque to the ExecTester and passed through to the adapter. - Consult the adapter's guide for supported parameters. --- ## Group 8: Rejection handling Test that the adapter correctly handles and reports order rejections. | TC | Name | Description | Skip when | | ------ | ---------------------- | ------------------------------------------------ | --------------- | | TC-E70 | PostOnly rejection | Post-only order that would cross the spread. | No post-only. | | TC-E71 | ReduceOnly rejection | Reduce-only order with no position to reduce. | No reduce-only. | | TC-E72 | Unsupported order type | Submit order type not supported by adapter. | Never. | | TC-E73 | Unsupported TIF | Submit order with unsupported time in force. | Never. | | TC-E74 | Ambiguous submit fail | Transport, timeout, or send failure on submit. | No mock path. | | TC-E75 | Ambiguous cancel fail | Transport, timeout, or send failure on cancel. | No cancel. | | TC-E76 | Ambiguous modify fail | Transport, timeout, or send failure on modify. | No modify. | | TC-E77 | Ambiguous batch fail | Whole-batch failure without per-order result. | No batch. | | TC-E78 | Per-order batch reject | Batch response has explicit per-order rejection. | No batch. | TC-E74 through TC-E78 are specified collectively below because they usually require a mock HTTP or WebSocket boundary rather than a live venue. ### Ambiguous outcome failures These cases prove that adapter request failures do not turn into terminal rejection events when the venue outcome is unknown. The pass criteria also define the local prepare-failure carve-out: when a command is known not to have been sent and is attributable to one cancel or modify command, the adapter may emit the matching rejection event. **Pass criteria:** - Submit failures from transport errors, timeouts, WebSocket send failures, retry exhaustion, or response parse failures do not emit `OrderRejected`. - Cancel failures from transport errors, timeouts, WebSocket send failures, retry exhaustion, or whole-request server failures do not emit `OrderCancelRejected`. - Modify failures from transport errors, timeouts, WebSocket send failures, retry exhaustion, or whole-request server failures do not emit `OrderModifyRejected`. - Local cancel prepare failures that prove the command cannot be sent may emit `OrderCancelRejected` when the adapter can attribute the failure to one cancel command. - Local modify prepare failures that prove the command cannot be sent may emit `OrderModifyRejected` when the adapter can attribute the failure to one modify command. - Whole-batch request failures do not emit one rejection per order when the venue did not return per-order results. - Explicit per-order venue rejections still emit the matching rejection event with the venue reason. The order remains in the appropriate in-flight state until a venue update, query result, or reconciliation pass resolves it. ### TC-E70: PostOnly rejection | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, quotes flowing. | | **Action** | ExecTester places post-only order on the wrong side of the book (`test_reject_post_only=True`), causing it to cross the spread. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderRejected`. | | **Pass criteria** | Venue rejects order; `OrderRejected.due_post_only=true`; reason names post-only violation. | | **Skip when** | Adapter does not support post-only flag. | **Considerations:** - The ExecTester's `test_reject_post_only` mode intentionally prices the order to cross. - Some venues may partially fill instead of rejecting; behavior is venue-specific. - Adapters that emit `OrderRejected` for a post-only crossing reject should set `due_post_only=true` so strategies can distinguish this from other venue rejections. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), enable_limit_buys=True, enable_limit_sells=False, use_post_only=True, test_reject_post_only=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .enable_limit_buys(true) .enable_limit_sells(false) .use_post_only(true) .test_reject_post_only(true) .build()? ``` ### TC-E71: ReduceOnly rejection | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Prerequisite** | Adapter connected, no open position for the instrument. | | **Action** | ExecTester opens a market position with `reduce_only=True` via `test_reject_reduce_only=True` and `open_position_on_start_qty`, when no position exists to reduce. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderRejected`. | | **Pass criteria** | Order rejected; `OrderRejected` event with reason indicating reduce-only violation. | | **Skip when** | Adapter does not support reduce-only flag. | **Considerations:** - The `test_reject_reduce_only` flag only applies to the opening market order submitted via `open_position_on_start_qty`. - Verify no prior position exists for the instrument before running this test. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), open_position_on_start_qty=Decimal("0.01"), test_reject_reduce_only=True, enable_limit_buys=False, enable_limit_sells=False, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .open_position_on_start_qty(Decimal::new(1, 2)) .test_reject_reduce_only(true) .enable_limit_buys(false) .enable_limit_sells(false) .build()? ``` ### TC-E72: Unsupported order type | Field | Value | | ------------------ | -------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, order type not in adapter's supported set. | | **Action** | Submit an order type the adapter does not support. | | **Event sequence** | `OrderDenied` (pre-submission rejection by adapter). | | **Pass criteria** | Order denied before reaching venue; `OrderDenied` event with reason. | | **Skip when** | Never (every adapter has unsupported order types to test). | **Considerations:** - `OrderDenied` occurs at the adapter level before the order reaches the venue. - This differs from the normal `OrderRejected` path, which follows a venue rejection. Reconciliation can also synthesize `OrderRejected`; see [Terminal reconciliation provenance](../concepts/execution/policies.md#terminal-reconciliation-provenance). - Test by configuring a stop order type that the adapter does not support. ### TC-E73: Unsupported TIF | Field | Value | | ------------------ | -------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, TIF not in adapter's supported set. | | **Action** | Submit an order with a TIF the adapter does not support. | | **Event sequence** | `OrderDenied` (pre-submission rejection by adapter). | | **Pass criteria** | Order denied before reaching venue; `OrderDenied` event with reason. | | **Skip when** | Never (every adapter has unsupported TIF options to test). | **Considerations:** - Similar to TC-E72 but for time-in-force options. - Test with TIF values from the Nautilus enum that the adapter does not map. --- ## Group 9: Lifecycle and reconciliation Test strategy lifecycle behavior and execution-state recovery on start, stop, and report failure. TC-E88 and TC-E89 run offline as deterministic adapter unit or integration tests rather than through `ExecTester` against a venue. | TC | Name | Description | Skip when | | ------ | --------------------------------- | -------------------------------------------------------- | ------------------------------------------- | | TC-E80 | Open position on start | Open a position immediately when strategy starts. | No market orders. | | TC-E81 | Cancel orders on stop | Cancel all open orders when strategy stops. | Never. | | TC-E82 | Close positions on stop | Close open positions when strategy stops. | No market orders. | | TC-E83 | Unsubscribe on stop | Unsubscribe from data feeds on strategy stop. | No unsub support. | | TC-E84 | Reconcile open orders | Reconcile existing open orders from a prior session. | Never. | | TC-E85 | Reconcile filled orders | Reconcile previously filled orders from a prior session. | Never. | | TC-E86 | Reconcile open long | Reconcile existing open long position. | Never. | | TC-E87 | Reconcile open short | Reconcile existing open short position. | Never. | | TC-E88 | Reconciliation commission failure | A required fill commission cannot be represented. | No fill commission logic. | | TC-E89 | WebSocket commission failure | A private fill commission cannot be represented. | No private fill stream or commission logic. | ### TC-E80: Open position on start | Field | Value | | ------------------ | -------------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, instrument loaded, no existing position. | | **Action** | Strategy starts with `open_position_on_start_qty` set. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled`. | | **Pass criteria** | Position opened on start; market order submitted and filled before limit order maintenance begins. | | **Skip when** | Adapter does not support market orders. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), open_position_on_start_qty=Decimal("0.01"), ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .open_position_on_start_qty(Decimal::new(1, 2)) .build()? ``` ### TC-E81: Cancel orders on stop | Field | Value | | ------------------ | -------------------------------------------------------------- | | **Prerequisite** | Open limit orders from the strategy session. | | **Action** | Stop the strategy with `cancel_orders_on_stop=True` (default). | | **Event sequence** | For each open order: `OrderPendingCancel` -> `OrderCanceled`. | | **Pass criteria** | All strategy-owned open orders canceled on stop. | | **Skip when** | Never. | ### TC-E82: Close positions on stop | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------ | | **Prerequisite** | Open position from the strategy session. | | **Action** | Stop the strategy with `close_positions_on_stop=True` (default). | | **Event sequence** | Closing order: `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled`. | | **Pass criteria** | Positions closed, or only exact sub-precision residuals remain; no open orders remain. | | **Skip when** | Adapter does not support market orders. | ### TC-E83: Unsubscribe on stop | Field | Value | | ------------------ | ---------------------------------------------------------------- | | **Prerequisite** | Active data subscriptions (quotes, trades, book). | | **Action** | Stop the strategy with `can_unsubscribe=True` (default). | | **Event sequence** | Data subscriptions removed. | | **Pass criteria** | No further data events received after stop; clean disconnection. | | **Skip when** | Adapter does not support unsubscribe. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, order_qty=Quantity.from_str("0.01"), can_unsubscribe=True, ) ``` **Rust config:** ```rust ExecTesterConfig::builder() .base(StrategyConfig { strategy_id: Some(strategy_id), ..Default::default() }) .instrument_id(instrument_id) .client_id(client_id) .order_qty(Quantity::from("0.01")) .can_unsubscribe(true) .build()? ``` ### TC-E84: Reconcile open orders | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | One or more open limit orders on the venue from a prior session. | | **Action** | Start the node with `reconciliation=True`. | | **Event sequence** | `OrderStatusReport` generated for each open order. | | **Pass criteria** | Each open order is loaded into the cache with correct `venue_order_id`, status=ACCEPTED, price, quantity, side, and order type. | | **Skip when** | Never. | **Considerations:** - Leave limit orders open from a prior test session (do not cancel on stop). - Configure `external_order_instrument_ids` so strategy registration creates the active claim used to assign reconciled orders. - Verify that the reconciled order count matches the venue-reported count. - Mass-status may include historical terminal orders. Unclaimed ones appear as EXTERNAL. Compare open-order counts against the venue open-order endpoint, not the full mass-status length. - Report `ts_event` values must be Unix nanoseconds even when the venue only has second precision. ### TC-E85: Reconcile filled orders | Field | Value | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | One or more filled orders on the venue from a prior session. | | **Action** | Start the node with `reconciliation=True`. | | **Event sequence** | `FillReport` generated for each historical fill. | | **Pass criteria** | Each filled order is loaded into the cache with correct `venue_order_id`, status=FILLED, fill price, fill quantity, and commission. | | **Skip when** | Never. | **Considerations:** - Requires orders that filled in a prior session. - Verify fill price, quantity, and commission match the venue's reported values. - Some adapters may only report fills within a lookback window. - Historical fill replay can change a live zero commission to the REST fee. That flip is expected when the stream omits fees and reconciliation supplies them. - Report `ts_event` values must be Unix nanoseconds even when the venue only has second precision. ### TC-E86: Reconcile open long position | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------ | | **Prerequisite** | An open long position on the venue from a prior session. | | **Action** | Start the node with `reconciliation=True`. | | **Event sequence** | `PositionStatusReport` generated for the long position. | | **Pass criteria** | Position loaded into cache with correct instrument, side=LONG, quantity, and entry price matching the venue. | | **Skip when** | Never. | **Considerations:** - Open a long position in a prior session and stop the strategy without closing it (`close_positions_on_stop=False`). - Verify the reconciled position quantity and average entry price match the venue. - After reconciliation, the strategy should be able to manage or close this position. ### TC-E87: Reconcile open short position | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | An open short position on the venue from a prior session. | | **Action** | Start the node with `reconciliation=True`. | | **Event sequence** | `PositionStatusReport` generated for the short position. | | **Pass criteria** | Position loaded into cache with correct instrument, side=SHORT, quantity, and entry price matching the venue. | | **Skip when** | Never. | **Considerations:** - Open a short position in a prior session and stop the strategy without closing it (`close_positions_on_stop=False`). - Verify the reconciled position quantity and average entry price match the venue. - After reconciliation, the strategy should be able to manage or close this position. ### TC-E88: Reconciliation commission failure | Field | Value | | ------------------ | --------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | A deterministic fixture produces an out-of-range commission for an owned, confirmed fill. | | **Action** | Exercise direct fill, mass-status, targeted report, and inferred-fill commission paths with the fixture. | | **Event sequence** | Report requests return an error; engine reconciliation logs a hook error and emits no fallback inferred fill. | | **Pass criteria** | The error is observable; startup fails or inferred work defers; a later valid response reconciles exactly once. | | **Skip when** | The adapter does not calculate commission for fill reports. | **Considerations:** - Assert that no zero or generic commission replaces the failed venue calculation. - Exercise the inferred-fill hook as well as direct `FillReport` construction. - Skip the inferred-fill portion when the adapter does not override the shared commission hook. - Assert that hook quantity, price, and liquidity inputs match the emitted inferred fill, including the back-solved price of an incremental residual after a prior fill. - Valid explicit fills may apply, but the residual quantity and dependent terminal transition stay pending. - A position-only synthetic correction without trade evidence is not a commission calculation failure. ### TC-E89: WebSocket commission failure | Field | Value | | ------------------ | -------------------------------------------------------------------------------------------------------------------- | | **Prerequisite** | A private fill fixture produces an out-of-range commission and has a stable venue trade ID. | | **Action** | Deliver the invalid trade, replace its commission with a valid value, then redeliver the same trade ID. | | **Event sequence** | The first delivery emits no fill and changes no fill, terminal, or deduplication state; the replay emits one fill. | | **Pass criteria** | No panic or fallback occurs; the valid replay applies exactly once; REST reconciliation can recover a missed replay. | | **Skip when** | The adapter has no private fill stream or does not calculate commission for private fills. | **Considerations:** - For a venue trade that fills several owned orders, make one report fail and assert that the first delivery emits none of them. This prevents a replay from duplicating reports built before the failure. - Assert that the adapter consumes the deduplication key only after all reports construct and route successfully. - Assert that a failed trade does not confirm or terminalize its order. --- ## Group 10: Options trading Test options-specific execution behavior. Options instruments typically have different constraints from linear derivatives: venues may restrict order types, support alternative pricing modes, or disallow conditional orders. Exact restrictions vary by venue; consult the adapter guide. These tests require a `CryptoOption` instrument. Use an OTM option with reasonable liquidity for fills. | TC | Name | Description | Skip when | | ------- | ----------------------------- | --------------------------------------------------------------------- | ------------------- | | TC-E90 | Limit BUY option | Place a limit buy on an option instrument. | No options support. | | TC-E91 | Limit SELL option | Place a limit sell on an option instrument. | No options support. | | TC-E92 | Limit with alt pricing | Place a limit order with adapter-specific pricing via `order_params`. | No alt pricing. | | TC-E94 | Unsupported order type denied | Submit an order type the adapter rejects for options. | No options support. | | TC-E96 | Conditional order rejected | Submit a stop/conditional order on an option; expect rejection. | No options support. | | TC-E99 | FOK limit option | Place a FOK limit order on an option instrument. | No FOK options. | | TC-E100 | Cancel option order | Cancel an open limit order on an option instrument. | No options support. | | TC-E101 | Reconcile option position | Reconcile an open option position from a prior session. | No options support. | ### TC-E90: Limit BUY option | Field | Value | | ------------------ | --------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, option instrument loaded, quotes flowing. | | **Action** | ExecTester places a limit buy on the option at a passive price. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order accepted by venue with correct instrument, side, price, and quantity. | | **Skip when** | Adapter does not support options trading. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, # CryptoOption instrument order_qty=Quantity.from_str("1"), enable_limit_buys=True, enable_limit_sells=False, tob_offset_ticks=500, ) ``` ### TC-E91: Limit SELL option | Field | Value | | ------------------ | --------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, option instrument loaded, quotes flowing. | | **Action** | ExecTester places a limit sell on the option at a passive price. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order accepted by venue with correct instrument, side, price, and quantity. | | **Skip when** | Adapter does not support options trading. | **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, # CryptoOption instrument order_qty=Quantity.from_str("1"), enable_limit_buys=False, enable_limit_sells=True, tob_offset_ticks=500, ) ``` ### TC-E92: Limit with alternative pricing | Field | Value | | ------------------ | ------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, option instrument loaded. | | **Action** | Place limit order with adapter-specific pricing via `order_params`. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted`. | | **Pass criteria** | Order accepted; venue acknowledges the alternative pricing mode. | | **Skip when** | Adapter does not support alternative pricing modes for options. | **Considerations:** - The `price` field on the order object may be a placeholder when alternative pricing is active. Consult the adapter guide for supported parameter keys. - Example: OKX supports `px_usd` (USD price) and `px_vol` (implied volatility). - Verify in venue responses that the pricing mode is reflected correctly. **Python config:** ```python ExecTesterConfig( instrument_id=instrument_id, # CryptoOption instrument order_qty=Quantity.from_str("1"), enable_limit_buys=True, enable_limit_sells=False, order_params={"px_usd": "100.5"}, # Adapter-specific pricing key ) ``` ### TC-E94: Unsupported order type denied for options | Field | Value | | ------------------ | ----------------------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, option instrument loaded. | | **Action** | Submit an order type the venue does not support for options (e.g. market order). | | **Event sequence** | Adapter-dependent: `OrderDenied` (pre-submission) or `OrderSubmitted` -> `OrderRejected` (post-submission). | | **Pass criteria** | Order does not fill. Denial or rejection reason references the unsupported order type. | | **Skip when** | Adapter does not support options. | **Considerations:** - The exact rejection point varies by adapter. Some adapters deny locally before submitting; others submit and relay the venue rejection. - ExecTester can trigger a market order via `open_position_on_start_qty` on an option instrument. Some unsupported types (e.g. `MarketToLimit`) require manual or programmatic submission. - Test each unsupported type the adapter documents. ### TC-E96: Conditional order rejected for options | Field | Value | | ------------------ | ----------------------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, option instrument loaded. | | **Action** | Submit a conditional order on an option instrument. | | **Event sequence** | Adapter-dependent: `OrderDenied` (pre-submission) or `OrderSubmitted` -> `OrderRejected` (post-submission). | | **Pass criteria** | Order does not fill. Reason references unsupported conditional order type. | | **Skip when** | Adapter does not support options, or adapter supports conditionals for options. | **Considerations:** - Test each conditional type the adapter documents as unsupported for options (e.g. `STOP_MARKET`, `STOP_LIMIT`, `MARKET_IF_TOUCHED`, `LIMIT_IF_TOUCHED`, `TRAILING_STOP_MARKET`). - ExecTester can trigger conditional orders via `enable_stop_buys`/`enable_stop_sells` with `stop_order_type` on an option instrument. ### TC-E99: FOK limit option | Field | Value | | ------------------ | ---------------------------------------------------------------------------------------------- | | **Prerequisite** | Adapter connected, option instrument loaded, sufficient book depth. | | **Action** | Place a limit order with `TimeInForce::Fok` on an option instrument. | | **Event sequence** | `OrderInitialized` -> `OrderSubmitted` -> `OrderAccepted` -> `OrderFilled` or `OrderCanceled`. | | **Pass criteria** | Order fills completely or is canceled. No partial fills. | | **Skip when** | Adapter does not support FOK for options. | **Considerations:** - Some venues use a dedicated order type for options FOK orders (e.g. OKX uses `op_fok`). The adapter handles this mapping transparently. - Use small quantities and aggressive pricing to get a fill for the positive case. ### TC-E100: Cancel option order | Field | Value | | ------------------ | -------------------------------------------------------------- | | **Prerequisite** | Open limit order from TC-E90 or TC-E91. | | **Action** | Cancel the open limit order. | | **Event sequence** | `OrderPendingCancel` -> `OrderCanceled`. | | **Pass criteria** | Order canceled; no longer appears in open orders on the venue. | | **Skip when** | Adapter does not support options. | ### TC-E101: Reconcile option position | Field | Value | | ------------------ | ------------------------------------------------------------------------------------ | | **Prerequisite** | Open option position from a prior session. | | **Action** | Start the node with `reconciliation=True`. | | **Event sequence** | `PositionStatusReport` generated for the option position. | | **Pass criteria** | Position loaded into cache with correct instrument, side, quantity, and entry price. | | **Skip when** | Adapter does not support options. | **Considerations:** - Open an option position in a prior session and stop without closing (`close_positions_on_stop=False`). - Verify the reconciled position matches the venue-reported state. --- ## ExecTester configuration reference Quick reference for every Python `ExecTesterConfig` parameter. Defaults are resolved values after construction; the Rust builder uses equivalent defaults. | Parameter | Type | Default | Affects groups | | ----------------------------------------------- | --------------------- | ---------------------- | -------------- | | `strategy_id` | `StrategyId?` | `None` | All | | `order_id_tag` | `str?` | `None` | All | | `use_hyphens_in_client_order_ids` | `bool` | `True` | All | | `use_uuid_client_order_ids` | `bool` | `False` | All | | `external_order_instrument_ids` | `list[InstrumentId]?` | `None` | 9 | | `instrument_id` | `InstrumentId` | `BTCUSDT-PERP.BINANCE` | All | | `client_id` | `ClientId?` | `None` | All | | `order_qty` | `Quantity` | `0.001` | All | | `order_display_qty` | `Quantity?` | `None` | 2, 7 | | `order_expire_time_delta_mins` | `PositiveInt?` | `None` | 2 | | `order_params` | `dict?` | `None` | 7, 10 | | `subscribe_book` | `bool` | `False` | | | `subscribe_quotes` | `bool` | `True` | | | `subscribe_trades` | `bool` | `True` | | | `book_type` | `BookType` | `L2_MBP` | | | `book_depth` | `PositiveInt?` | `None` | | | `book_interval_ms` | `PositiveInt` | `1000` | | | `book_levels_to_print` | `PositiveInt` | `10` | | | `open_position_on_start_qty` | `Decimal?` | `None` | 1, 9 | | `open_position_on_first_quote` | `bool` | `False` | 1 | | `open_position_time_in_force` | `TimeInForce` | `GTC` | 1 | | `enable_limit_buys` | `bool` | `True` | 2, 4, 5, 6 | | `enable_limit_sells` | `bool` | `True` | 2, 4, 5, 6 | | `enable_stop_buys` | `bool` | `False` | 3, 4 | | `enable_stop_sells` | `bool` | `False` | 3, 4 | | `tob_offset_ticks` | `PositiveInt` | `500` | 2, 4 | | `limit_time_in_force` | `TimeInForce?` | `None` | 2, 6 | | `stop_order_type` | `OrderType` | `STOP_MARKET` | 3 | | `stop_offset_ticks` | `PositiveInt` | `100` | 3 | | `stop_limit_offset_ticks` | `PositiveInt?` | `None` | 3 | | `stop_trigger_type` | `TriggerType` | `DEFAULT` | 3 | | `stop_time_in_force` | `TimeInForce?` | `None` | 3 | | `trailing_offset` | `Decimal?` | `None` | 3 | | `trailing_offset_type` | `TrailingOffsetType` | `BASIS_POINTS` | 3 | | `enable_brackets` | `bool` | `False` | 6 | | `batch_submit_limit_pair` | `bool` | `False` | 2, 5 | | `bracket_entry_order_type` | `OrderType` | `LIMIT` | 6 | | `bracket_offset_ticks` | `PositiveInt` | `500` | 6 | | `modify_orders_to_maintain_tob_offset` | `bool` | `False` | 4 | | `modify_stop_orders_to_maintain_offset` | `bool` | `False` | 4 | | `cancel_replace_orders_to_maintain_tob_offset` | `bool` | `False` | 4 | | `cancel_replace_stop_orders_to_maintain_offset` | `bool` | `False` | 4 | | `trigger_limit_order_maintenance_once` | `bool` | `False` | 4 | | `use_post_only` | `bool` | `False` | 2, 6, 7, 8 | | `limit_aggressive` | `bool` | `False` | 2 | | `use_quote_quantity` | `bool` | `False` | 1, 7 | | `emulation_trigger` | `TriggerType?` | `None` | 2, 3 | | `use_individual_cancels_on_stop` | `bool` | `False` | 5 | | `cancel_orders_on_stop` | `bool` | `True` | 5, 9 | | `close_positions_on_stop` | `bool` | `True` | 9 | | `close_positions_qty_precision` | `int?` | `None` | 9 | | `close_positions_time_in_force` | `TimeInForce?` | `None` | 9 | | `reduce_only_on_stop` | `bool` | `True` | 7, 9 | | `use_batch_cancel_on_stop` | `bool` | `False` | 5 | | `dry_run` | `bool` | `False` | | | `log_data` | `bool` | `True` | | | `test_reject_post_only` | `bool` | `False` | 8 | | `test_reject_reduce_only` | `bool` | `False` | 8 | | `test_modify_rejected` | `bool` | `False` | 4 | | `can_unsubscribe` | `bool` | `True` | 9 | | `clamp_to_instrument_price_range` | `bool` | `False` | 1-8, 10 | | `log_events` | `bool` | `True` | All | | `log_commands` | `bool` | `True` | All | # Test Datasets Source: https://nautilustrader.io/docs/latest/developer_guide/test_datasets/ Target standards for curating, storing, and consuming external datasets used as test fixtures. New datasets should follow these standards. Existing datasets that predate this policy are documented under [legacy datasets](#legacy-datasets). ## Dataset categories **Small data** (< 1 MB) is checked directly into `test_data//` alongside a `metadata.json` file. These files are always available without network access. **Large data** (> 1 MB) is hosted as Parquet in the R2 test-data bucket. A SHA-256 checksum is recorded in `test_data/large/checksums.json`. Before running tests that use large data, prepare the fixtures from the repository root: ```bash cargo run --locked -p nautilus-testkit --bin prepare-test-data ``` This command downloads missing files and verifies every fixture in the tracked checksum manifest. It replaces cached files whose checksums differ, leaves the manifest unchanged, and rejects and removes downloads with mismatched checksums. CI runs this setup after restoring the test-data cache. The `ensure_test_data_exists()` function only checks for a local file. A test that needs a missing fixture fails with a message naming the setup command, without downloading data. Setup and tests both honor `TEST_DATA_ROOT_PATH`. **User-fetched data** is used when a vendor license, entitlement model, or access control does not allow NautilusTrader to redistribute the data through the public repo or the public R2 bucket. In this model, the repo stores only a manifest, fetch instructions, and the transform code. Each user downloads the source data with their own vendor account and converts it locally. Use the user-fetched model when any of the following apply: - The vendor requires each user to hold their own account, API key, or historical-data license. - The license allows internal use but does not clearly allow redistribution of derived fixtures. - The dataset is suitable for examples or opt-in integration tests, but not for default CI. ## Required metadata Every curated dataset that stores or redistributes a concrete artifact must include a `metadata.json` with at minimum: | Field | Description | | -------------- | ------------------------------------------------- | | `file` | Filename of the dataset. | | `sha256` | SHA-256 hash of the file. | | `size_bytes` | File size in bytes. | | `original_url` | Download URL of the original source data. | | `licence` | License terms and any redistribution constraints. | | `added_at` | ISO 8601 timestamp when the dataset was curated. | These fields match the output of `scripts/curate-dataset.sh`. Additional recommended fields for richer provenance: | Field | Description | | --------------- | --------------------------------------------------------- | | `instrument` | Instrument symbol(s) covered. | | `date` | Trading date(s) covered. | | `format` | Storage format (e.g., "Nautilus OrderBookDelta Parquet"). | | `original_file` | Original vendor filename before transformation. | | `parser` | Parser used for transformation (e.g., "itchy 0.3.4"). | User-fetched datasets use the same metadata fields where they apply. They should also include: | Field | Description | | ------------------- | -------------------------------------------------------------------- | | `distribution` | Must be `"user-fetch"`. | | `fetch_method` | How the user acquires the source data (API, web portal, CLI, etc.). | | `fetch_reference` | URL or document reference for the user-facing download flow. | | `auth` | Required credentials or entitlements, if any. | | `transform_version` | Version of the local transform pipeline that builds the final files. | | `redistribution` | Short note describing redistribution limits for the dataset. | | `public_mirror` | Must be `false` for restricted vendor datasets. | For user-fetched datasets without a single committed or mirrored artifact, `file`, `sha256`, and `size_bytes` may be omitted from `metadata.json`. In that case, `target_files` in `manifest.json` is authoritative for the local output files. For user-fetched datasets, `original_url` may point to the vendor download entry point rather than the exact file URL when the exact file is generated per user account or per request. Other metadata fields remain recommended where they apply. In particular, `licence` and `added_at` should still be recorded for user-fetched datasets. ## Storage format New datasets should be stored as **Nautilus Parquet** (not raw vendor formats). This ensures: - Consistent data types across all test datasets. - No vendor format parsing at test time. - Clear derivative-work status for licensing. Use ZSTD compression (level 3) with 1M row groups. User-fetched datasets should also end up as Nautilus Parquet after the local transform step. Raw vendor files should stay outside the repo and outside the public R2 bucket. ## Naming convention ``` ___.parquet ``` Examples: - `itch_AAPL_2019-01-30_deltas.parquet` - `tardis_BTCUSDT_2020-09-01_depth10.parquet` - `histdata_EURUSD.SIM_2020-01_quotes.parquet` ## Curation workflow ### Simple files (single download) Use `scripts/curate-dataset.sh`: ```bash scripts/curate-dataset.sh

is not registered`. | Discover the pool first; analysis cannot infer or register metadata. | | Missing parser capability | Fails before sync. | Use a snapshot-capable combination from [Chain and DEX command support](#chain-and-dex-command-support). | | Non-checksummed address | Fails with `Blockchain address '

' has incorrect checksum`. | Supply an EIP-55 address; factory `getPool` results may need checksum conversion. | | Provider Multicall cap | An `out of gas`, payload, or timeout error aborts final-state hydration. | Lower `multicall_calls_per_rpc_request` or use a provider with higher limits. | | Missing historical state | A first bootstrap cannot read final state at an old `--to-block`. | Use a recent target or an archive-capable provider; see [RPC provider requirements](#rpc-provider-requirements). | | Restricted HyperSync quota | High-activity pools back off, and a full first sync can require thousands of requests. | Lower `--concurrency`, or use `--snapshot-from-rpc` when stored swap history is unnecessary. | | Mid-life `--from-block` | Omitting `Initialize` can leave the profiler without an initial price. | Sync from pool creation when generating a first snapshot. | | No liquidity events | `analyze-pool` errors; `analyze-pools` emits a failure result and continues other pools. | Select a pool with a Mint or Burn at or before the target block. | | Any per-pool failure | Emits `"status": "failure"` and makes `analyze-pool(s)` exit non-zero. | Use the exit code for the overall result and each JSON status for the per-pool result. | Final RPC hydration in `--snapshot-from-rpc` mode is authoritative for checkpoint state. A failed hydration aborts analysis rather than storing a snapshot with stale price state. ## Pool analysis operations ### Discover pools before analysis `analyze-pool(s)` reads pool metadata from the Postgres cache and fails with `Pool

is not registered` when the pool has not been discovered. Run `sync-dex` for the chain and DEX before analysis to populate the `pool` and `token` tables. ### Check command support Use [Chain and DEX command support](#chain-and-dex-command-support) or the command help before starting a sync. `sync-dex` and `analyze-pool(s)` reject unsupported chain and DEX combinations before querying events, rather than returning an empty result. ```bash ./target/debug/nautilus blockchain sync-dex --help ./target/debug/nautilus blockchain analyze-pools --help ``` ### Use checksummed pool addresses Pool addresses must use the EIP-55 checksum. A lowercase address fails with `Blockchain address '