Polars

Polars is a fast DataFrame library implemented in Rust with a Python API. It processes large datasets efficiently and is well suited to time-series work. Since QuestDB 10.0 the Python and Rust clients both speak Polars natively over QWP, in both directions.

QWP results arrive as Arrow record batches, which is Polars' own memory layout, so a query becomes a DataFrame with no row-by-row conversion. Ingestion sends the frame column by column over the same connection pool. No ConnectorX or PGWire driver is involved.

Prerequisites​

  • QuestDB 10.0 or later, running and accessible. See the quick start.
  • For Python: version 3.10 or later, plus the client and Polars.
  • For Rust: questdb-rs with the Polars crate features enabled.
python3 -m pip install -U questdb polars pyarrow

Python​

Query into a DataFrame​

db.query() streams Arrow batches from the server. to_polars() materializes the whole result:

import questdb

with questdb.connect("ws::addr=localhost:9000;") as db:
    with db.query(
        "SELECT timestamp, symbol, price, amount FROM trades "
        "WHERE timestamp IN '$now-1h..$now'"
    ) as result:
        df = result.to_polars()

print(df.head())

Bind values with $1..$N placeholders rather than interpolating them into the SQL string:

df = db.query(
    "SELECT * FROM trades WHERE symbol = $1 AND price > $2",
    ["ETH-USDT", 2615.0],
).to_polars()

Use to_polars() by default. It requires pyarrow. If you need a pyarrow-free installation, pass the result to pl.DataFrame instead:

import polars as pl

with db.query("SELECT * FROM trades LIMIT 1000") as result:
    df = pl.DataFrame(result)

This still materializes the complete result and can be slower for SYMBOL-heavy queries.

Stream large results​

For results that do not fit comfortably in memory, iterate batch by batch with iter_polars():

import questdb

with questdb.connect("ws::addr=localhost:9000;") as db:
    with db.query("SELECT price, amount FROM trades") as result:
        notional = sum(
            (chunk["price"] * chunk["amount"]).sum()
            for chunk in result.iter_polars()
        )

print(notional)

A result is single-use and must stay on the thread that created it. Use a with block, or call close(), so the connection returns to the pool.

Ingest a DataFrame​

db.dataframe() accepts Polars DataFrame and LazyFrame alongside pandas and pyarrow inputs. Each call publishes the frame in batches and blocks until the server acknowledges the last one:

import polars as pl
import questdb

df = pl.DataFrame({
    "symbol": ["ETH-USDT", "BTC-USDT"],
    "price": [2615.54, 65432.10],
    "amount": [0.00044, 0.00120],
    "timestamp": [1735689600000000000, 1735689601000000000],
}).with_columns(pl.col("timestamp").cast(pl.Datetime("ns", "UTC")))

with questdb.connect("ws::addr=localhost:9000;") as db:
    db.dataframe(df, table_name="trades", symbols=["symbol"], at="timestamp")

symbols takes a list of column names to store as SYMBOL, and at names the designated timestamp column. See DataFrame ingestion for batching, null handling, and the full parameter set.

Rust​

Crate features​

Polars support is gated behind optional crate features:

Cargo.toml
[dependencies]
questdb-rs = { version = "7", features = ["polars"] }

The polars feature enables both polars-ingress and polars-egress. Enable only the one you need if your application goes in a single direction. See crate features for the full list.

Query into a DataFrame​

Borrow a reader, prepare the SQL, bind values, and collect the cursor into a frame:

use questdb::QuestDb;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let db = QuestDb::connect("ws::addr=localhost:9000;")?;
    let mut reader = db.borrow_reader()?;

    let dataframe = reader
        .prepare("SELECT timestamp, symbol, price FROM trades WHERE price > $1")
        .bind_f64(2615.0)
        .execute()?
        .fetch_all_polars()?;

    println!("{} rows", dataframe.height());
    Ok(())
}

fetch_all_polars() materializes the complete result. For large results, prefer next_polars() or iter_polars(), which yield one frame per batch.

The Cursor returned by execute() borrows the reader, so keep the whole chain in one statement as above. Splitting it across a block that also owns the reader fails to compile.

note

The client depends on polars with default features off, and only within a version range. To use the returned frame in your own code, add polars as a direct dependency at a version inside that range, otherwise its DataFrame is a different type. Check the range in the client's Cargo.toml. Formatting a frame with {} also needs one of the polars fmt features, which the client does not enable.

Ingest a DataFrame​

flush_polars_dataframe() borrows a direct sender from the pool, publishes a commit boundary, waits for the requested ACK, and returns the connection:

use questdb::ingress::{
    column_sender::ArrowColumnOverride,
    polars::PolarsIngestOptions,
    AckLevel,
    ColumnName,
};

let overrides: [ArrowColumnOverride<'_>; 0] = [];
let options = PolarsIngestOptions::new()
    .max_rows(50_000)
    .timestamp_column(ColumnName::new("timestamp")?)
    .overrides(&overrides)
    .ack_level(AckLevel::Ok);

db.flush_polars_dataframe("trades", &dataframe, &options)?;

Omitting timestamp_column asks the server to assign timestamps, and max_rows(0) uses the default batch size. The call checkpoints the frame and automatically retries the uncommitted tail after a transient failover. Replay is at-least-once, so use deduplication when duplicates would be harmful. See Arrow and Polars ingestion for the Arrow equivalents.

Legacy: ConnectorX over PGWire​

warning

Since QuestDB 10.0 the recommended way to move data between Polars and QuestDB is the native client support shown above. ConnectorX is documented here for legacy reasons only: it goes through PGWire, it reads but cannot ingest, and it needs a workaround to avoid PostgreSQL features QuestDB does not implement.

ConnectorX is a Rust library for fast data transfer between Python and various databases. Its PostgreSQL connector works against QuestDB's PGWire endpoint, which lets pl.read_database_uri() read a query into a Polars DataFrame.

pip install polars pyarrow connectorx
import polars as pl

QUESTDB_URI = "redshift://admin:quest@localhost:8812/qdb"
QUERY = "SELECT * FROM tables() LIMIT 5;"

df = pl.read_database_uri(query=QUERY, uri=QUESTDB_URI)
print("Received DataFrame:")
print(df)
caution

The URI uses the redshift scheme, not postgresql. By default the PostgreSQL connector uses features QuestDB does not support; the Redshift scheme makes ConnectorX avoid them.

See also​