Data CatalogCopy Anchor Link

The data catalog stores NautilusTrader data in Parquet files for backtesting, live trading, and research.

Overview and architectureCopy Anchor Link

ParquetDataCatalog is the Python interface to the Rust catalog and DataFusion query engine. The Rust model and persistence crates define the Arrow schemas for built-in data. Registered custom data supplies its schema and encode/decode handlers at runtime.

Instant timestamps use Timestamp(Nanosecond, Some("UTC")). Durations stay integers. Arrow readers and Nautilus queries preserve the nanoseconds and UTC annotation. SQL readers that map these columns to microsecond-precision TIMESTAMPTZ can truncate sub-microsecond values.

Parquet provides compressed columnar storage and cross-language access. The catalog stores these files under one root without requiring a separate database service. A local path or object-store URI selects the storage backend.

InitializingCopy Anchor Link

Pass a local path or URI as the first constructor argument:

from pathlib import Path

from nautilus_trader.persistence import ParquetDataCatalog


CATALOG_PATH = Path.cwd() / "catalog"
catalog = ParquetDataCatalog(str(CATALOG_PATH))

Filesystem protocols and storage optionsCopy Anchor Link

The catalog accepts the storage protocols supported by its Rust object-store backend.

Supported filesystem protocolsCopy Anchor Link

Option keys are object_store configuration keys for the URI scheme:

• S3, GCS, and Azure keys also take a prefixed form, such as aws_region, google_application_credentials, or azure_storage_sas_token.
• Client options such as timeout apply to each remote scheme. Set allow_http to true to connect over plain http://.
• S3 also accepts the legacy key and secret names.
• An unknown key fails with an error that names it.

Pass credentials and other backend settings through storage_options:

catalog = ParquetDataCatalog(
    "s3://my-bucket/nautilus-data/",
    storage_options={
        "access_key_id": "your-key",
        "secret_access_key": "your-secret",
        "region": "us-east-1",
    },
)

azure_catalog = ParquetDataCatalog(
    "abfs://container@account.dfs.core.windows.net/nautilus-data/",
    storage_options={"account_key": "your-account-key"},
)

After the bucket, container, or host, the base path of a remote URI must not contain spaces or non-ASCII characters. It must also exclude characters that object-store paths encode, such as ~, %, #, ?, ^, or |. The catalog rejects such a URI when it opens, with an error that names it.

Compression and row groupsCopy Anchor Link

DataCatalogConfig sets how a configured catalog reads and writes data files:

from nautilus_trader.config import DataCatalogConfig


catalog = DataCatalogConfig(path="./catalog", compression="snappy", max_row_group_size=65_536)

ParquetDataCatalog takes the same batch_size and max_row_group_size settings, but selects compression by numeric code. DataCatalogConfig accepts the names below, ignoring case:

Construction fails for any other compression name and for a zero batch_size or max_row_group_size. params carries only options for an external catalog backend. The Parquet catalog rejects any params key.

Writing dataCopy Anchor Link

Use the writer for the concrete data type. Instrument definitions and custom data have separate writers.

catalog.write_instruments([instrument])
catalog.write_quote_ticks(quote_ticks)

catalog.write_trade_ticks(
    trade_ticks,
    start=1704067200000000000,
    end=1704153600000000000,
)

catalog.write_bars(bars, skip_disjoint_check=True)

The built-in market-data writers are:

• write_quote_ticks
• write_trade_ticks
• write_order_book_deltas
• write_order_book_depths
• write_bars
• write_mark_price_updates
• write_index_price_updates
• write_option_greeks

Each writer accepts optional start and end overrides as UNIX nanoseconds. The data in one call must have one identity, such as one instrument ID or bar type, and must be ordered by ts_init.

Typed, instrument, and custom-data writes on one instance serialize their native file changes. Do not overlap queries, record writes, or catalog operations with other calls on that instance. A concurrent call can raise RuntimeError if either call needs exclusive access.

File naming and data organizationCopy Anchor Link

The catalog names files from their timestamp range with the pattern {start_timestamp}_{end_timestamp}.parquet. It converts each ISO 8601 timestamp to a filename-safe form by replacing : and . with -.

Built-in data is organized in directories by data type and identifier. For instrument IDs and bar types, the catalog removes / and replaces ^ with _ when creating the URI-safe directory name:

catalog/
├── data/
│   ├── quotes/
│   │   └── EURUSD.SIM/
│   │       └── 2024-01-01T00-00-00-000000000Z_2024-01-01T23-59-59-999999999Z.parquet
│   └── trades/
│       └── BTCUSD.BINANCE/
│           └── 2024-01-01T00-00-00-000000000Z_2024-01-01T23-59-59-999999999Z.parquet

Custom data uses data/custom/<type_name>/ with optional identifier path segments.

Instruments use one directory per concrete instrument class, such as data/currency_pair/ or data/equity/, rather than a shared instruments directory.

Catalog queries, and thus backtests, read only these canonical directories. Queries do not read legacy layouts such as data/trade_tick/, data/quote_tick/, and the Python-written data/custom_<snake_case>/.

To upgrade a catalog that still uses a legacy layout, convert it with nautilus catalog migrate-parquet.

Reading dataCopy Anchor Link

Use a typed query when the expected return type is known. start and end are UNIX nanoseconds:

quotes = catalog.query_quote_ticks(
    identifiers=["EUR/USD.SIM"],
    start=1704067200000000000,
    end=1704153600000000000,
)

trades = catalog.query_trade_ticks(
    identifiers=["BTC/USD.BINANCE"],
    start=1704067200000000000,
    end=1704153600000000000,
)

With the streaming feature enabled, register catalogs through LiveNodeConfig.catalogs using DataCatalogConfig entries. Historical requests read available catalog intervals and request missing intervals from the data client.

Quote and trade requests with an explicit params["schema"] bypass catalog reads and write-back. Source schemas can change sampling or trade IDs, but catalog keys do not distinguish them.

Set params={"update_catalog": True} to write successful source responses back to the catalog. Use "catalog_name" to select a registered catalog explicitly.

Write-back requires a client that reports completion explicitly, such as Databento or a Python adapter. Other clients return data without updating the catalog and log why write-back is skipped.

Write-back applies these coverage rules:

• Failed and intermediate responses skip write-back. Requests split across multiple source envelopes also skip write-back, including parent-symbol book-delta requests that return multiple payloads.
• Requests with a record limit skip write-back. Decoded row counts cannot prove whether the source exhausted that limit.
• Batches with instruments or bar types outside the request skip write-back because their source coverage intervals are unknown.
• Successful empty responses can extend known-empty coverage when both bounds are available and the backend supports it.

Catalog write failures are logged and do not discard the returned data.

BacktestDataConfig: backtest dataCopy Anchor Link

BacktestDataConfig defines the catalog data that a BacktestNode loads for one run.

Core parametersCopy Anchor Link

• data_type is a NautilusDataType value: QuoteTick, TradeTick, Bar, OrderBookDelta, OrderBookDepth, MarkPriceUpdate, IndexPriceUpdate, FundingRateUpdate, InstrumentStatus, OptionGreeks, InstrumentClose, or Instrument. Instrument loads each instrument class the catalog holds for the selected identifiers.
• catalog is a required DataCatalogConfig that supplies the source for instrument and replay data loading.
• One of instrument_id, instrument_ids, or bar_types is required.
• start_time and end_time are optional UNIX nanosecond bounds.
• filter_expr is an optional DataFusion SQL predicate.
• For bars, bar_spec combines with the instrument ID to select an EXTERNAL bar type. Explicit bar_types can select internal, external, or composite bars.
• optimize_file_loading registers whole directories when possible.
• batch_deltas (default True) replays OrderBookDelta data as OrderBookDeltas batches closed by F_LAST. See order book delta replay.

Migrate flat catalog settingsCopy Anchor Link

Replace the flat catalog arguments and property access with catalog=DataCatalogConfig(...) and config.catalog.<field>:

If catalog_fs_rust_storage_options was supplied, migrate that entire map. Otherwise, migrate catalog_fs_storage_options. Do not merge the maps. Use native object-store option keys, as described in filesystem protocols and storage options.

A catalog defaults to Parquet and the file protocol. Set batch_size, compression, and max_row_group_size as typed catalog arguments; params is reserved for external backend options. Catalog batch_size controls catalog batches; run-level chunk_size controls replay chunks. For replay, name and read_only do not affect loading; compression and max_row_group_size apply only to catalog writes. Storage-option readback exposes keys through config.catalog.fs_rust_storage_option_keys, and representations redact the values.

Basic usage examplesCopy Anchor Link

from nautilus_trader.config import BacktestDataConfig
from nautilus_trader.config import DataCatalogConfig
from nautilus_trader.model import BarAggregation
from nautilus_trader.model import BarSpecification
from nautilus_trader.model import InstrumentId
from nautilus_trader.model import NautilusDataType
from nautilus_trader.model import PriceType

quote_data = BacktestDataConfig(
    data_type=NautilusDataType.QuoteTick,
    catalog=DataCatalogConfig(path="/path/to/catalog"),
    instrument_id=InstrumentId.from_str("EUR/USD.SIM"),
    start_time=1704067200000000000,
    end_time=1704153600000000000,
)

trade_data = BacktestDataConfig(
    data_type=NautilusDataType.TradeTick,
    catalog=DataCatalogConfig(path="/path/to/catalog"),
    instrument_ids=[
        InstrumentId.from_str("BTC/USD.BINANCE"),
        InstrumentId.from_str("ETH/USD.BINANCE"),
    ],
)

bar_data = BacktestDataConfig(
    data_type=NautilusDataType.Bar,
    catalog=DataCatalogConfig(path="/path/to/catalog"),
    instrument_id=InstrumentId.from_str("AAPL.NASDAQ"),
    bar_spec=BarSpecification(5, BarAggregation.MINUTE, PriceType.LAST),
)

This bar config selects AAPL.NASDAQ-5-MINUTE-LAST-EXTERNAL.

Cloud storage and filteringCopy Anchor Link

book_data = BacktestDataConfig(
    data_type=NautilusDataType.OrderBookDelta,
    catalog=DataCatalogConfig(
        path="my-bucket/nautilus-data",
        fs_protocol="s3",
        fs_rust_storage_options={
            "access_key_id": "your-access-key",
            "secret_access_key": "your-secret-key",
            "region": "us-east-1",
        },
    ),
    instrument_id=InstrumentId.from_str("BTC/USD.COINBASE"),
    filter_expr="ts_init >= 1704067200000000000",
)

Integration with BacktestRunConfigCopy Anchor Link

Pass the data configurations to BacktestRunConfig:

from decimal import Decimal

from nautilus_trader.config import BacktestDataConfig
from nautilus_trader.config import BacktestRunConfig
from nautilus_trader.config import BacktestVenueConfig
from nautilus_trader.config import DataCatalogConfig
from nautilus_trader.execution import MakerTakerFeeModel
from nautilus_trader.model import AccountType
from nautilus_trader.model import BookType
from nautilus_trader.model import InstrumentId
from nautilus_trader.model import NautilusDataType
from nautilus_trader.model import OmsType

data_configs = [
    BacktestDataConfig(
        data_type=NautilusDataType.QuoteTick,
        catalog=DataCatalogConfig(path="/path/to/catalog"),
        instrument_id=InstrumentId.from_str("EUR/USD.SIM"),
    ),
]

run_config = BacktestRunConfig(
    venues=[
        BacktestVenueConfig(
            name="SIM",
            oms_type=OmsType.HEDGING,
            account_type=AccountType.MARGIN,
            book_type=BookType.L1_MBP,
            starting_balances=["1_000_000 USD"],
            fee_model=MakerTakerFeeModel(
                maker_rate=Decimal("0"),
                taker_rate=Decimal("0"),
            ),
        ),
    ],
    data=data_configs,
    start=1704067200000000000,
    end=1704153600000000000,
)

Data loading processCopy Anchor Link

When a backtest runs, the BacktestNode processes each BacktestDataConfig:

1. Create a ParquetDataCatalog from the configuration.
2. Load the required instrument definitions while building the engine.
3. Build and run a DataFusion query from the configuration fields.
4. Sort merged data by ts_init and add it to the backtest engine.

Direct catalog accessCopy Anchor Link

• Use ParquetDataCatalog to query or write a catalog directly.
• Use BacktestDataConfig when a BacktestNode should load catalog data for a run.
• Use LiveNodeConfig.catalogs to register catalogs for historical requests. This does not preload data as BacktestDataConfig does. The separate streaming field configures Feather writing only.

Querying and filteringCopy Anchor Link

The generic query selects a data family through NautilusDataType, which maps to catalog directories such as quotes, trades, or bars. Use it when you need the files or optimize_file_loading controls:

catalog.query(
    data_type=NautilusDataType.QuoteTick,
    identifiers=["EUR/USD.SIM"],
    start=1704067200000000000,
    end=1704153600000000000,
    where_clause="ts_event <= ts_init",
    files=None,
)

All of these query methods use UNIX nanosecond time bounds and accept a DataFusion SQL where_clause:

• query provides the generic interface.
• Typed methods such as query_quote_ticks, query_trade_ticks, and query_bars return the concrete model type.
• query_custom_data resolves custom decoders through the runtime registry.

Catalog operationsCopy Anchor Link

Catalog operations rename, consolidate, or delete data files. Their type selectors are:

• NautilusDataType: Selects a data family. NautilusDataType.Instrument covers each instrument class.
• NautilusRecordType: Selects a record family.
• NautilusInstrumentType: Targets one instrument class.

delete_data_range(...) takes a NautilusDataType alone, because only data families support ranged deletes.

Reset file namesCopy Anchor Link

Reset Parquet file names to match their content timestamps so filename-based filtering stays accurate. reset_all_file_names() processes the entire catalog. reset_data_file_names(...) targets a data path.

For data types partitioned by instrument:

• Supply an instrument ID to target that instrument's directory.
• Without an ID, the operation recursively reads the type directory and renames each file within its own instrument directory. It checks each directory's intervals before it renames any file.

catalog.reset_all_file_names()
catalog.reset_data_file_names(NautilusDataType.QuoteTick, "EUR/USD.SIM")
catalog.reset_data_file_names(NautilusDataType.TradeTick, "BTC/USD.BINANCE")

Recover from overlapping file namesCopy Anchor Link

Overlapping file names block writes to the affected directory. A rejected coverage extension never renames files, so overlap points to an earlier rename, a manual move, or a concurrent writer.

When the file contents are still disjoint, recover the directory with a filename reset:

1. Stop all writers to the catalog.
2. Back up the affected directory. The reset renames files one at a time, so an I/O failure partway through can leave some files renamed. On object stores that rename by copying then deleting, a failure can also leave a file under both names.
3. Run reset_data_file_names(...) for the affected path. For custom data types, use reset_all_file_names() instead, which covers each leaf directory including custom layouts.
4. If the reset reports that another file holds a new name, rename that file. Use an unused interval name outside the data range. Then run the reset again. Repeat until it succeeds.
5. Confirm each renamed file matches its content range, then write a later disjoint interval to confirm the directory accepts writes again.

The reset processes one directory at a time. Before renaming any file in a directory, it reads each file's ts_init range. It fails when content ranges overlap or when a file's new name is the current name of another file. On failure, it leaves each file name in that directory unchanged. Directories reset before the failing one keep their new names.

Use filename reset only for filename-only damage. When file contents overlap, a filename reset cannot reconcile the data, so the reset fails. Rebuild the affected range from source through a separately validated process.

Consolidate catalogCopy Anchor Link

Combine small Parquet files to reduce file count and query overhead. With no bounds, consolidate_catalog() processes each leaf data directory in the catalog. consolidate_data(...) operates on one directory. Supply an instrument ID for data types partitioned by instrument.

catalog.consolidate_catalog()

catalog.consolidate_catalog(
    start=1704067200000000000,
    end=1704153600000000000,
    ensure_contiguous_files=True,
)

catalog.consolidate_data(
    NautilusDataType.QuoteTick,
    identifier="EUR/USD.SIM",
    start=1704067200000000000,
    end=1706745600000000000,
)

Consolidate catalog by periodCopy Anchor Link

Split data files into fixed periods. Durations and time bounds use nanoseconds. Both methods accept optional bounds. Supply an identifier to the data-type method for data partitioned by instrument.

• Catalog-wide method: Processes quotes, trades, order book deltas, order book depths, bars, index prices, mark prices, instrument closes, and registered custom types. It logs a warning and skips other types.
• Data-type method: Rejects instrument and record selectors, which have no period-typed rewrite. Use consolidate_data(...) for those.

DAY_NS = 86_400_000_000_000
HOUR_NS = 3_600_000_000_000

catalog.consolidate_catalog_by_period(period_nanos=DAY_NS)

catalog.consolidate_catalog_by_period(
    period_nanos=HOUR_NS,
    start=1704067200000000000,
    end=1704153600000000000,
)

catalog.consolidate_data_by_period(
    data_type=NautilusDataType.QuoteTick,
    identifier="EUR/USD.SIM",
    period_nanos=HOUR_NS,
)

catalog.consolidate_data_by_period(
    data_type=NautilusDataType.TradeTick,
    identifier="EUR/USD.SIM",
    period_nanos=HOUR_NS,
    start=1704067200000000000,
    end=1706745600000000000,
)

Delete data rangeCopy Anchor Link

Delete data within a time range, optionally limited to one data type and instrument. Supply an instrument ID for data partitioned by instrument. The time bounds work as follows:

• Omitting start extends the range to the beginning.
• Omitting end extends the range to the end.
• For delete_data_range(...), omitting both bounds removes all matching data.

delete_data_range(...) supports quotes, trades, bars, order book deltas, order book depth, and registered custom types. Pass NautilusDataType.OrderBookDepth for order book depth and NautilusDataType.Custom("MarketTickPython") for a custom data type.

• delete_catalog_range(...) continues after unsupported directories, logs a warning, and leaves their data unchanged.
• Use delete_data_range(...) to confirm that the requested type is supported. NautilusDataType.Instrument is rejected because instrument definitions do not support ranged deletion.

catalog.delete_catalog_range(
    start=1704067200000000000,
    end=1704153600000000000,
)

catalog.delete_catalog_range(end=1704067200000000000)

catalog.delete_data_range(
    data_type=NautilusDataType.QuoteTick,
    identifier="BTC/USD.BINANCE",
)

catalog.delete_data_range(
    data_type=NautilusDataType.TradeTick,
    identifier="EUR/USD.SIM",
    start=1704067200000000000,
    end=1706745600000000000,
)

Feather streaming and conversionCopy Anchor Link

The runtime can stage records in local Feather files and promote them into a catalog with StreamingConfig(writer_path=..., catalog=DataCatalogConfig(...)). Staged records become available to catalog queries after promotion succeeds. A staging flush and a catalog commit are separate steps.

Streaming defaults to promotion on close, no interval-based promotion, and retention of committed Feather sources. With a positive promotion_interval_ms:

• Live streaming uses wall-clock scheduling.
• Backtest streaming checks the supplied clock during writes and flushes.

See stream data into a Parquet catalog for defaults, configuration, query visibility, and recovery.

StreamingFeatherWriter stays available for direct staging. Its completed sessions can be converted manually with ParquetDataCatalog.convert_stream_to_data().